Search

We study the Lagrangian Index Policy (LIP) for restless multi-armed bandits with long-run average reward. In particular, we compare the performance of LIP with the performance of the Whittle Index Policy (WIP), both heuristic policies known to be asymptotically optimal under certain natural conditions. Even though in most cases their performances are very similar, in the cases when WIP shows bad performance, LIP continues to perform very well. We then propose reinforcement learning algorithms, both tabular and NN-based, to obtain online learning schemes for LIP in the model-free setting. The proposed reinforcement learning schemes for LIP requires significantly less memory than the analogous scheme for WIP. We calculate analytically the Lagrangian index for the restart model, which describes the optimal web crawling and the minimization of the weighted age of information. We also give a new proof of asymptotic optimality in case of homogeneous bandits as the number of arms goes to infinity, based on exchangeability and de Finetti's theorem.

Rapid adoption of machine learning (ML) technologies has led to a surge in power consumption across diverse systems, from tiny IoT devices to massive datacenter clusters. Benchmarking the energy efficiency of these systems is crucial for optimization, but presents novel challenges due to the variety of hardware platforms, workload characteristics, and system-level interactions. This paper introduces MLPerf Power, a comprehensive benchmarking methodology with capabilities to evaluate the energy efficiency of ML systems at power levels ranging from microwatts to megawatts. Developed by a consortium of industry professionals from more than 20 organizations, MLPerf Power establishes rules and best practices to ensure comparability across diverse architectures. We use representative workloads from the MLPerf benchmark suite to collect 1,841 reproducible measurements from 60 systems across the entire range of ML deployment scales. Our analysis reveals trade-offs between performance, complexity, and energy efficiency across this wide range of systems, providing actionable insights for designing optimized ML solutions from the smallest edge devices to the largest cloud infrastructures. This work emphasizes the importance of energy efficiency as a key metric in the evaluation and comparison of the ML system, laying the foundation for future research in this critical area. We discuss the implications for developing sustainable AI solutions and standardizing energy efficiency benchmarking for ML systems., Comment: 14 pages, 11 figures, 1 table

Recent models suggest approximately half of all accreting supermassive black holes (SMBHs; $M_{\rm BH}$ $\gtrsim$ 10$^{5}$ M$_{\odot}$) are expected to undergo intense growth phases behind Compton-thick ($N_{\rm H}$ $>$ 1.5 $\times$ 10$^{24}$ cm$^{-2}$) veils of obscuring gas. However, despite being a viable source for the seeding of SMBHs, there are currently no examples known of a Compton-thick accreting intermediate mass black hole (IMBH; $M_{\rm BH}$ $\sim$ 10$^{2}$ $-$ 10$^{5}$ M$_{\odot}$). We present a detailed X-ray spectral analysis of IC 750 $-$ the only AGN to-date with a precise megamaser-based intermediate mass $<$ 10$^{5}$ M$_{\odot}$. We find the equivalent width of neutral 6.4 keV Fe K$\alpha$ to be 1.9$^{+2.2}_{-1.0}$ keV via phenomenological modelling of the co-added 177 ks Chandra spectrum. Such large equivalent widths are seldom produced by processes other than fluorescence from dense obscuration. We fit three physically-motivated X-ray spectral models to infer a range of possible intrinsic 2$-$10 keV luminosity posteriors that encompass the systematic uncertainties associated with a choice of model. Despite a wide range of predicted intrinsic 2$-$10 keV luminosities between $\sim$ 10$^{41}$ and 10$^{43}$ erg s$^{-1}$, all three models agree that IC 750 has a Compton-thick line-of-sight column density to $>$ 99\% confidence. Compton-thick obscuration is well-documented to impinge substantial bias on the pursuit of SMBH AGN. Our results thus provide the first indication that Compton-thick obscuration should also be properly considered to uncover and understand the IMBH population in an unbiased manner., Comment: Accepted for publication in ApJ. 15 pages, 6 figures

Compact star-forming galaxies were dominant galaxy types in the early Universe. Blueberry galaxies (BBs) represent their local analogues being very compact and having intensive star formation. Motivated by high X-ray emission recently found in other analogical dwarf galaxies, called Green Peas, we probe into the X-ray properties of BBs to determine if their X-ray emission is consistent with the empirical laws for star-forming galaxies. We performed the first X-ray observations of a small sample of BBs with the XMM-Newton satellite. Spectral analysis for detected sources and upper limits measured via Bayesian-based analysis for very low-count measurements were used to determine the X-ray properties of our galaxy sample. Clear detection was obtained only for 2 sources, with one source exhibiting an enhanced X-ray luminosity to the scaling relations. For the remaining 5 sources, only an upper limit was constrained, suggesting BBs to be rather underluminous as a whole. Our analysis shows that the large scatter cannot be easily explained by the stochasticity effects. While the bright source is above and inconsistent at almost the 99% confidence level, the upper limits of the two sources are below the expected distribution. These results indicate that the empirical relations between the star formation rate, metallicity, and X-ray luminosity might not hold for BBs with uniquely high specific star formation rates. One possible explanation could be that the BBs may not be old enough to have a significant X-ray binary population. The high luminosity of the only bright source can be then caused by an additional X-ray source, such as a hidden active galactic nucleus or more extreme ultraluminous X-ray sources., Comment: 10 pages, 4 + 1 (appendix) figures, accepted for publication in Astronomy & Astrophysics on 9.8.2024

Historical documents frequently suffer from damage and inconsistencies, including missing or illegible text resulting from issues such as holes, ink problems, and storage damage. These missing portions or gaps are referred to as lacunae. In this study, we employ transformer-based optical character recognition (OCR) models trained on synthetic data containing lacunae in a supervised manner. We demonstrate their effectiveness in detecting and restoring lacunae, achieving a success rate of 65%, compared to a base model lacking knowledge of lacunae, which achieves only 5% restoration. Additionally, we investigate the mechanistic properties of the model, such as the log probability of transcription, which can identify lacunae and other errors (e.g., mistranscriptions due to complex writing or ink issues) in line images without directly inspecting the image. This capability could be valuable for scholars seeking to distinguish images containing lacunae or errors from clean ones. Although we explore the potential of attention mechanisms in flagging lacunae and transcription errors, our findings suggest it is not a significant factor. Our work highlights a promising direction in utilizing transformer-based OCR models for restoring or analyzing damaged historical documents., Comment: Accepted to ICDAR 2024 Workshop on Computational Paleography

Memorization in language models is typically treated as a homogenous phenomenon, neglecting the specifics of the memorized data. We instead model memorization as the effect of a set of complex factors that describe each sample and relate it to the model and corpus. To build intuition around these factors, we break memorization down into a taxonomy: recitation of highly duplicated sequences, reconstruction of inherently predictable sequences, and recollection of sequences that are neither. We demonstrate the usefulness of our taxonomy by using it to construct a predictive model for memorization. By analyzing dependencies and inspecting the weights of the predictive model, we find that different factors influence the likelihood of memorization differently depending on the taxonomic category.

The Whittle index policy is a heuristic that has shown remarkably good performance (with guaranteed asymptotic optimality) when applied to the class of problems known as Restless Multi-Armed Bandit Problems (RMABPs). In this paper we present QWI and QWINN, two reinforcement learning algorithms, respectively tabular and deep, to learn the Whittle index for the total discounted criterion. The key feature is the use of two time-scales, a faster one to update the state-action Q -values, and a relatively slower one to update the Whittle indices. In our main theoretical result we show that QWI, which is a tabular implementation, converges to the real Whittle indices. We then present QWINN, an adaptation of QWI algorithm using neural networks to compute the Q -values on the faster time-scale, which is able to extrapolate information from one state to another and scales naturally to large state-space environments. For QWINN, we show that all local minima of the Bellman error are locally stable equilibria, which is the first result of its kind for DQN-based schemes. Numerical computations show that QWI and QWINN converge faster than the standard Q -learning algorithm, neural-network based approximate Q-learning and other state of the art algorithms., Comment: ACM Transactions on Modeling and Performance Evaluation of Computing Systems, 2024

Risk sensitive decision making finds important applications in current day use cases. Existing risk measures consider a single or finite collection of random variables, which do not account for the asymptotic behaviour of underlying systems. Conditional Value at Risk (CVaR) is the most commonly used risk measure, and has been extensively utilized for modelling rare events in finite horizon scenarios. Naive extension of existing risk criteria to asymptotic regimes faces fundamental challenges, where basic assumptions of existing risk measures fail. We present a complete simulation based approach for sequentially computing Asymptotic CVaR (ACVaR), a risk measure we define on limiting empirical averages of markovian rewards. Large deviations theory, density estimation, and two-time scale stochastic approximation are utilized to define a 'tilted' probability kernel on the underlying state space to facilitate ACVaR simulation. Our algorithm enjoys theoretical guarantees, and we numerically evaluate its performance over a variety of test cases., Comment: 9 pages, 5 figures

Dwarf highly star-forming galaxies (SFGs) dominated the early Universe and are considered the main driver of its reionization. However, direct observations of these distant galaxies are mainly confined to rest-frame ultraviolet and visible light, limiting our understanding of their complete properties. Therefore, it is still paramount to study their local analogs, the green pea (GP) and blueberry (BB) galaxies. This work aims to expand our knowledge of BBs by identifying a new sample that is closer and in the southern sky. In addition to the already known BBs, this new sample will allow for a statistically significant study of their properties probed by visible and infrared (IR) light. By utilizing the HECATE catalog, which provides photometry and characterization of galaxies, along with data from Pan-STARSS and SDSS, this study selects and analyzes a new sample of BBs. We employed spectral energy distribution fitting to derive homogeneous measurements of star-formation rates and stellar masses. Additionally, we measured emission-line fluxes, including $\rm HeII~\lambda 4686$, through spectral fitting. Through this work, we identified 48 BBs, of which 40 were first recognized as such, with the nearest at 19~Mpc. 14 of the BBs are in the south sky. The BBs tend to be extremely IR red in both WISE $W1-W2$ and $W2-W3$ colors, distinguishing them from typical SFGs. Dwarf SFGs with higher specific star-formation rates tend to have redder IR colors. Blueberry galaxies stand out as the most intensely star-forming sources in the local Universe among dwarf galaxies. They exhibit unique characteristics, such as being intrinsically bluer in visible light, redder in the infrared, and less massive. They also have higher specific star-formation rates, equivalent widths, lower metallicities, and the most strongly ionized interstellar medium compared to typical SFGs and GPs., Comment: 18 pages, 11 + 4 (appendix) figures. Accepted for publication in Astronomy & Astrophysics on 2024-05-02. Updated to match the published version

Green Peas (GPs) are young, compact, star-forming dwarf galaxies, and local (z~0.3) analogues of the early galaxies (z>6) considered to be mainly responsible for the reionisation of the Universe. Recent X-ray observations of GPs have detected high excess emission which cannot be accounted for by star formation alone, and implies presence of an active galactic nucleus (AGN). We employ radio observations to study the radio properties of GPs, build their radio spectral energy distributions, and verify the presence of AGNs. We performed new radio observations of three GPs with the Karl G. Jansky Very Large Array (JVLA) in the L, C, and X bands (1.4, 6 and 10 GHz resp.), supplemented by the data from archival observations and large radio surveys. We also analysed the archival radio data for a larger sample of GPs and blueberry (BBs) galaxies, which are lower-mass and lower-redshift analogues of the GPs. To understand the significance of the radio observations, we assess the detectability of these sources, and compare the detected radio luminosities with the expectations from theoretical and empirical relations. Two of the three targeted GPs were strongly detected ($>10\sigma$) in the JVLA observations and their fluxes are consistent with star formation, while the third source was undetected. Although a large fraction (~75%) of the sources from the larger archival sample of GPs and BBs could be detected with archival surveys, only a small number (<40%) are detected and their radio luminosity is significantly lower than the expectation from empirical relations. Our results show that the majority of the dwarf galaxy sample is highly underluminous. Especially towards the lower end of galaxy mass and star formation rate (SFR), the radio luminosity-SFR relation deviates from the empirical relations, suggesting that the relations established for larger galaxies may not hold towards the low-mass end., Comment: 9 pages, 4 figures, accepted for publication to A&A

We address the problem of user association in a dense millimeter wave (mmWave) network, in which each arriving user brings a file containing a random number of packets and each time slot is divided into multiple mini-slots. This problem is an instance of the restless multi-armed bandit problem, and is provably hard to solve. Using a technique introduced by Whittle, we relax the hard per-stage constraint that each arriving user must be associated with exactly one mmWave base station (mBS) to a long-term constraint and then use the Lagrangian multiplier technique to convert the problem into an unconstrained problem. This decouples the process governing the system into separate Markov Decision Processes at different mBSs. We prove that the problem is Whittle indexable, present a scheme for computing the Whittle indices of different mBSs, and propose an association scheme under which, each arriving user is associated with the mBS with the smallest value of the Whittle index. Using extensive simulations, we show that the proposed Whittle index based scheme outperforms several user association schemes proposed in prior work in terms of various performance metrics such as average cost, delay, throughput, and Jain's fairness index., Comment: 16 pages, 7 figures

Timing vulnerabilities in processors have emerged as a potent threat. As processors are the foundation of any computing system, identifying these flaws is imperative. Recently fuzzing techniques, traditionally used for detecting software vulnerabilities, have shown promising results for uncovering vulnerabilities in large-scale hardware designs, such as processors. Researchers have adapted black-box or grey-box fuzzing to detect timing vulnerabilities in processors. However, they cannot identify the locations or root causes of these timing vulnerabilities, nor do they provide coverage feedback to enable the designer's confidence in the processor's security. To address the deficiencies of the existing fuzzers, we present WhisperFuzz--the first white-box fuzzer with static analysis--aiming to detect and locate timing vulnerabilities in processors and evaluate the coverage of microarchitectural timing behaviors. WhisperFuzz uses the fundamental nature of processors' timing behaviors, microarchitectural state transitions, to localize timing vulnerabilities. WhisperFuzz automatically extracts microarchitectural state transitions from a processor design at the register-transfer level (RTL) and instruments the design to monitor the state transitions as coverage. Moreover, WhisperFuzz measures the time a design-under-test (DUT) takes to process tests, identifying any minor, abnormal variations that may hint at a timing vulnerability. WhisperFuzz detects 12 new timing vulnerabilities across advanced open-sourced RISC-V processors: BOOM, Rocket Core, and CVA6. Eight of these violate the zero latency requirements of the Zkt extension and are considered serious security vulnerabilities. Moreover, WhisperFuzz also pinpoints the locations of the new and the existing vulnerabilities., Comment: Accepted to USENIX Sec'24

We derive a concentration bound of the type `for all $n \geq n_0$ for some $n_0$' for TD(0) with linear function approximation. We work with online TD learning with samples from a single sample path of the underlying Markov chain. This makes our analysis significantly different from offline TD learning or TD learning with access to independent samples from the stationary distribution of the Markov chain. We treat TD(0) as a contractive stochastic approximation algorithm, with both martingale and Markov noises. Markov noise is handled using the Poisson equation and the lack of almost sure guarantees on boundedness of iterates is handled using the concept of relaxed concentration inequalities., Comment: Submitted to Stochastic Systems

Oberman gave a stochastic control formulation of the problem of estimating the convex envelope of a non-convex function. Based on this, we develop a reinforcement learning scheme to approximate the convex envelope, using a variant of Q-learning for controlled optimal stopping. It shows very promising results on a standard library of test problems.

In this paper, we propose a reinforcement learning algorithm to solve a multi-agent Markov decision process (MMDP). The goal, inspired by Blackwell's Approachability Theorem, is to lower the time average cost of each agent to below a pre-specified agent-specific bound. For the MMDP, we assume the state dynamics to be controlled by the joint actions of agents, but the per-stage costs to only depend on the individual agent's actions. We combine the Q-learning algorithm for a weighted combination of the costs of each agent, obtained by a gossip algorithm with the Metropolis-Hastings or Multiplicative Weights formalisms to modulate the averaging matrix of the gossip. We use multiple timescales in our algorithm and prove that under mild conditions, it approximately achieves the desired bounds for each of the agents. We also demonstrate the empirical performance of this algorithm in the more general setting of MMDPs having jointly controlled per-stage costs.

The Internet of Things (IoT) is emerging as a critical technology to connect resource-constrained devices such as sensors and actuators as well as appliances to the Internet. In this paper, we propose a novel methodology for node cardinality estimation in wireless networks such as the IoT and Radio-Frequency IDentification (RFID) systems, which uses the privileged feature distillation (PFD) technique and works using a neural network with a teacher-student model. The teacher is trained using both privileged and regular features, and the student is trained with predictions from the teacher and regular features. We propose node cardinality estimation algorithms based on the PFD technique for homogeneous as well as heterogeneous wireless networks. We show via extensive simulations that the proposed PFD based algorithms for homogeneous as well as heterogeneous networks achieve much lower mean squared errors in the computed node cardinality estimates than state-of-the-art protocols proposed in prior work, while taking the same number of time slots for executing the node cardinality estimation process as the latter protocols., Comment: 15 pages, 17 figures, journal paper

This article recalls the recent work on a linear programming formulation of infinite horizon risk-sensitive control via its equivalence with a single controller game, using a classic work of Vrieze. This is then applied to a constrained risk-sensitive control problem with a risk-sensitive cost and risk-sensitive constraint. This facilitates a Lagrange multiplier based resolution thereof. In the process, this leads to an unconstrained linear program and its dual, parametrized by a parameter that is a surrogate for Lagrange multiplier. This also opens up the possibility of a primal - dual type numerical scheme wherein the linear program is a subroutine within the subgradient ascent based update rule for the Lagrange multiplier. This equivalent unconstrained risk-sensitive control formulation does not seem obvious without the linear programming equivalents as intermediaries. We also discuss briefly other related algorithmic possibilities for future research., Comment: under review with a journal

In this article we prove under suitable assumptions that the marginals of any solution to a relaxed controlled martingale problem on a Polish space $E$ can be mimicked by a Markovian solution of a Markov-relaxed controlled martingale problem. We also show how such `Markov mimics' can be obtained by relative entropy minimisation. We provide many examples where the above results can be applied.

Large Language Models (LLMs) have a privacy concern because they memorize training data (including personally identifiable information (PII) like emails and phone numbers) and leak it during inference. A company can train an LLM on its domain-customized data which can potentially also include their users' PII. In order to comply with privacy laws such as the "right to be forgotten", the data points of users that are most vulnerable to extraction could be deleted. We find that once the most vulnerable points are deleted, a new set of points become vulnerable to extraction. So far, little attention has been given to understanding memorization for fine-tuned models. In this work, we also show that not only do fine-tuned models leak their training data but they also leak the pre-training data (and PII) memorized during the pre-training phase. The property of new data points becoming vulnerable to extraction after unlearning and leakage of pre-training data through fine-tuned models can pose significant privacy and legal concerns for companies that use LLMs to offer services. We hope this work will start an interdisciplinary discussion within AI and law communities regarding the need for policies to tackle these issues., Comment: 5 pages, 8 figures, accepted to the first GenLaw workshop at ICML'23, Hawai'i

Two time scale stochastic approximation algorithms emulate singularly perturbed deterministic differential equations in a certain limiting sense, i.e., the interpolated iterates on each time scale approach certain differential equations in the large time limit when viewed on the `algorithmic time scale' defined by the corresponding step sizes viewed as time steps. Their fluctuations around these deterministic limits, after suitable scaling, can be shown to converge to a Gauss-Markov process in law for each time scale. This turns out to be a linear diffusion for the faster iterates and an ordinary differential equation for the slower iterates., Comment: 40 pages

The Atacama Large Millimetre/submillimetre Array (ALMA) is the world's most advanced radio interferometric facility, producing science data with an average rate of about 1 TB per day. After a process of calibration, imaging and quality assurance, the scientific data are stored in the ALMA Science Archive (ASA), along with the corresponding raw data, making the ASA an invaluable resource for original astronomical research. Due to their complexity, each ALMA data set has the potential for scientific results that go well beyond the ideas behind the original proposal that led to each observation. For this reason, the European ALMA Regional Centre initiated the High-Level Data Products initiative to develop science-oriented data products derived from data sets publicly available in the ASA, that go beyond the formal ALMA deliverables. The first instance of this initiative is the creation of a catalogue of submillimetre (submm) detections of Sloan Digital Sky Survey (SDSS) quasars from the SDSS Data Release 14 that lie in the aggregate ALMA footprint observed since ALMA Cycle 0. The ALMA fluxes are extracted in an automatic fashion, using the ALMA Data Mining Toolkit. All extractions above a signal-to-noise cut of 3.5 are considered, they have been visually inspected and the reliable detections are presented in a catalogue of 376 entries, corresponding to 275 unique quasars. Interesting targets found in the process, i.e. lensed or jetted quasars as well as quasars with nearby submm counterparts are highlighted, to facilitate further studies or potential follow up observations.