Search

We introduce the notion of a move graph, that is, a directed graph whose vertex set is a $\mathbb Z$-module $\mathbb Z_n^m$, and whose arc set is uniquely determined by the action $M\!:\!\mathbb Z_n^m\to \mathbb Z_n^m$ where $M$ is an $m\times m$ matrix with integer entries. We study the manner in which properties of move graphs differ when one varies the choice of cyclic group $\mathbb Z_n$. Our principal focus is on a special family of such graphs, which we refer to as ``sub-add move graphs.''

Evaluating policies using off-policy data is crucial for applying reinforcement learning to real-world problems such as healthcare and autonomous driving. Previous methods for off-policy evaluation (OPE) generally suffer from high variance or irreducible bias, leading to unacceptably high prediction errors. In this work, we introduce STAR, a framework for OPE that encompasses a broad range of estimators -- which include existing OPE methods as special cases -- that achieve lower mean squared prediction errors. STAR leverages state abstraction to distill complex, potentially continuous problems into compact, discrete models which we call abstract reward processes (ARPs). Predictions from ARPs estimated from off-policy data are provably consistent (asymptotically correct). Rather than proposing a specific estimator, we present a new framework for OPE and empirically demonstrate that estimators within STAR outperform existing methods. The best STAR estimator outperforms baselines in all twelve cases studied, and even the median STAR estimator surpasses the baselines in seven out of the twelve cases., Comment: Accepted at the Thirty-eighth Annual Conference on Neural Information Processing Systems (NeurIPS 2024)

One of the most fascinating aspects of quantum networks is their capability to distribute entanglement as a nonlocal communication resource. In a first step, this requires network-ready devices that can generate and store entangled states. Another crucial step, however, is to develop measurement techniques that allow for entanglement detection. Demonstrations for different platforms suffer from being either not complete, or destructive, or local. Here we demonstrate a complete and nondestructive measurement scheme that always projects any initial state of two spatially separated network nodes onto a maximally entangled state. Each node consists of an atom trapped inside an optical resonator from which two photons are successively reflected. Polarisation measurements on the photons discriminate between the four maximally entangled states. Remarkably, such states are not destroyed by our measurement. In the future, our technique might serve to probe the decay of entanglement and to stabilise it against dephasing via repeated measurements.

Novel reinforcement learning algorithms, or improvements on existing ones, are commonly justified by evaluating their performance on benchmark environments and are compared to an ever-changing set of standard algorithms. However, despite numerous calls for improvements, experimental practices continue to produce misleading or unsupported claims. One reason for the ongoing substandard practices is that conducting rigorous benchmarking experiments requires substantial computational time. This work investigates the sources of increased computation costs in rigorous experiment designs. We show that conducting rigorous performance benchmarks will likely have computational costs that are often prohibitive. As a result, we argue for using an additional experimentation paradigm to overcome the limitations of benchmarking., Comment: 19 pages, 13 figures, The Forty-first International Conference on Machine Learning (ICML 2024)

We present ICU-Sepsis, an environment that can be used in benchmarks for evaluating reinforcement learning (RL) algorithms. Sepsis management is a complex task that has been an important topic in applied RL research in recent years. Therefore, MDPs that model sepsis management can serve as part of a benchmark to evaluate RL algorithms on a challenging real-world problem. However, creating usable MDPs that simulate sepsis care in the ICU remains a challenge due to the complexities involved in acquiring and processing patient data. ICU-Sepsis is a lightweight environment that models personalized care of sepsis patients in the ICU. The environment is a tabular MDP that is widely compatible and is challenging even for state-of-the-art RL algorithms, making it a valuable tool for benchmarking their performance. However, we emphasize that while ICU-Sepsis provides a standardized environment for evaluating RL algorithms, it should not be used to draw conclusions that guide medical practice., Comment: Reinforcement Learning Conference 2024

Entanglement has evolved from an enigmatic concept of quantum physics to a key ingredient of quantum technology. It explains correlations between measurement outcomes that contradict classical physics, and has been widely explored with small sets of individual qubits. Multi-partite entangled states build up in gate-based quantum-computing protocols, and $\unicode{x2013}$ from a broader perspective $\unicode{x2013}$ were proposed as the main resource for measurement-based quantum-information processing. The latter requires the ex-ante generation of a multi-qubit entangled state described by a graph. Small graph states such as Bell or linear cluster states have been produced with photons, but the proposed quantum computing and quantum networking applications require fusion of such states into larger and more powerful states in a programmable fashion. Here we achieve this goal by employing an optical resonator containing two individually addressable atoms. Ring and tree graph states with up to eight qubits, with the names reflecting the entanglement topology, are efficiently fused from the photonic states emitted by the individual atoms. The fusion process itself employs a cavity-assisted gate between the two atoms. Our technique is in principle scalable to even larger numbers of qubits, and is the decisive step towards, for instance, a memory-less quantum repeater in a future quantum internet., Comment: 18 pages, 5 figures, 2 tables

In this paper, we introduce a fresh perspective on the challenges of credit assignment and policy evaluation. First, we delve into the nuances of eligibility traces and explore instances where their updates may result in unexpected credit assignment to preceding states. From this investigation emerges the concept of a novel value function, which we refer to as the \emph{bidirectional value function}. Unlike traditional state value functions, bidirectional value functions account for both future expected returns (rewards anticipated from the current state onward) and past expected returns (cumulative rewards from the episode's start to the present). We derive principled update equations to learn this value function and, through experimentation, demonstrate its efficacy in enhancing the process of policy evaluation. In particular, our results indicate that the proposed learning approach can, in certain challenging contexts, perform policy evaluation more rapidly than TD($\lambda$) -- a method that learns forward value functions, $v^\pi$, \emph{directly}. Overall, our findings present a new perspective on eligibility traces and potential advantages associated with the novel value function it inspires, especially for policy evaluation., Comment: Accepted in The 38th Annual AAAI Conference on Artificial Intelligence

Designing reward functions for efficiently guiding reinforcement learning (RL) agents toward specific behaviors is a complex task. This is challenging since it requires the identification of reward structures that are not sparse and that avoid inadvertently inducing undesirable behaviors. Naively modifying the reward structure to offer denser and more frequent feedback can lead to unintended outcomes and promote behaviors that are not aligned with the designer's intended goal. Although potential-based reward shaping is often suggested as a remedy, we systematically investigate settings where deploying it often significantly impairs performance. To address these issues, we introduce a new framework that uses a bi-level objective to learn \emph{behavior alignment reward functions}. These functions integrate auxiliary rewards reflecting a designer's heuristics and domain knowledge with the environment's primary rewards. Our approach automatically determines the most effective way to blend these types of feedback, thereby enhancing robustness against heuristic reward misspecification. Remarkably, it can also adapt an agent's policy optimization process to mitigate suboptimalities resulting from limitations and biases inherent in the underlying RL algorithms. We evaluate our method's efficacy on a diverse set of tasks, from small-scale experiments to high-dimensional control challenges. We investigate heuristic auxiliary rewards of varying quality -- some of which are beneficial and others detrimental to the learning process. Our results show that our framework offers a robust and principled way to integrate designer-specified heuristics. It not only addresses key shortcomings of existing approaches but also consistently leads to high-performing solutions, even when given misaligned or poorly-specified auxiliary reward functions., Comment: (Spotlight) Thirty-seventh Conference on Neural Information Processing Systems (NeurIPS 2023)

Representation learning is increasingly employed to generate representations that are predictive across multiple downstream tasks. The development of representation learning algorithms that provide strong fairness guarantees is thus important because it can prevent unfairness towards disadvantaged groups for all downstream prediction tasks. To prevent unfairness towards disadvantaged groups in all downstream tasks, it is crucial to provide representation learning algorithms that provide fairness guarantees. In this paper, we formally define the problem of learning representations that are fair with high confidence. We then introduce the Fair Representation learning with high-confidence Guarantees (FRG) framework, which provides high-confidence guarantees for limiting unfairness across all downstream models and tasks, with user-defined upper bounds. After proving that FRG ensures fairness for all downstream models and tasks with high probability, we present empirical evaluations that demonstrate FRG's effectiveness at upper bounding unfairness for multiple downstream models and tasks.

Strong coupling of molecules to vacuum fields has been widely reported to lead to modified chemical properties such as reaction rates. However, some recent attempts to reproduce infrared strong coupling results have not been successful, suggesting that factors other than strong coupling may sometimes be involved. Here we re-examine the first of these vacuum-modified chemistry experiments in which changes to a molecular photoisomerisation process in the UV-vis spectral range were attributed to strong coupling of the molecules to visible light. We observed significant variations in photoisomerisation rates consistent with the original work; however, we found no evidence that these changes need to be attributed to strong coupling. Instead, we suggest that the photoisomerisation rates involved are most strongly influenced by the absorption of ultraviolet radiation in the cavity. Our results indicate that care must be taken to rule out non-polaritonic effects before invoking strong coupling to explain any changes of chemical properties arising in cavity-based experiments., Comment: 33 pages, 17 figures

Coagent networks for reinforcement learning (RL) [Thomas and Barto, 2011] provide a powerful and flexible framework for deriving principled learning rules for arbitrary stochastic neural networks. The coagent framework offers an alternative to backpropagation-based deep learning (BDL) that overcomes some of backpropagation's main limitations. For example, coagent networks can compute different parts of the network \emph{asynchronously} (at different rates or at different times), can incorporate non-differentiable components that cannot be used with backpropagation, and can explore at levels higher than their action spaces (that is, they can be designed as hierarchical networks for exploration and/or temporal abstraction). However, the coagent framework is not just an alternative to BDL; the two approaches can be blended: BDL can be combined with coagent learning rules to create architectures with the advantages of both approaches. This work generalizes the coagent theory and learning rules provided by previous works; this generalization provides more flexibility for network architecture design within the coagent framework. This work also studies one of the chief disadvantages of coagent networks: high variance updates for networks that have many coagents and do not use backpropagation. We show that a coagent algorithm with a policy network that does not use backpropagation can scale to a challenging RL domain with a high-dimensional state and action space (the MuJoCo Ant environment), learning reasonable (although not state-of-the-art) policies. These contributions motivate and provide a more general theoretical foundation for future work that studies coagent networks.

Monitoring drivers' mental workload facilitates initiating and maintaining safe interactions with in-vehicle information systems, and thus delivers adaptive human machine interaction with reduced impact on the primary task of driving. In this paper, we tackle the problem of workload estimation from driving performance data. First, we present a novel on-road study for collecting subjective workload data via a modified peripheral detection task in naturalistic settings. Key environmental factors that induce a high mental workload are identified via video analysis, e.g. junctions and behaviour of vehicle in front. Second, a supervised learning framework using state-of-the-art time series classifiers (e.g. convolutional neural network and transform techniques) is introduced to profile drivers based on the average workload they experience during a journey. A Bayesian filtering approach is then proposed for sequentially estimating, in (near) real-time, the driver's instantaneous workload. This computationally efficient and flexible method can be easily personalised to a driver (e.g. incorporate their inferred average workload profile), adapted to driving/environmental contexts (e.g. road type) and extended with data streams from new sources. The efficacy of the presented profiling and instantaneous workload estimation approaches are demonstrated using the on-road study data, showing $F_{1}$ scores of up to 92% and 81%, respectively., Comment: Accepted for IEEE Transactions on Intelligent Vehicles

In this work, we consider the off-policy policy evaluation problem for contextual bandits and finite horizon reinforcement learning in the nonstationary setting. Reusing old data is critical for policy evaluation, but existing estimators that reuse old data introduce large bias such that we can not obtain a valid confidence interval. Inspired from a related field called survey sampling, we introduce a variant of the doubly robust (DR) estimator, called the regression-assisted DR estimator, that can incorporate the past data without introducing a large bias. The estimator unifies several existing off-policy policy evaluation methods and improves on them with the use of auxiliary information and a regression approach. We prove that the new estimator is asymptotically unbiased, and provide a consistent variance estimator to a construct a large sample confidence interval. Finally, we empirically show that the new estimator improves estimation for the current and future policy values, and provides a tight and valid interval estimation in several nonstationary recommendation environments., Comment: AISTATS 2023

We propose a first-order method for convex optimization, where instead of being restricted to the gradient from a single parameter, gradients from multiple parameters can be used during each step of gradient descent. This setup is particularly useful when a few processors are available that can be used in parallel for optimization. Our method uses gradients from multiple parameters in synergy to update these parameters together towards the optima. While doing so, it is ensured that the computational and memory complexity is of the same order as that of gradient descent. Empirical results demonstrate that even using gradients from as low as \textit{two} parameters, our method can often obtain significant acceleration and provide robustness to hyper-parameter settings. We remark that the primary goal of this work is less theoretical, and is instead aimed at exploring the understudied case of using multiple gradients during each step of optimization., Comment: Accepted at OPT workshop @ Thirty-sixth Conference on Neural Information Processing Systems (NeurIPS 2022)

Methods for sequential decision-making are often built upon a foundational assumption that the underlying decision process is stationary. This limits the application of such methods because real-world problems are often subject to changes due to external factors (passive non-stationarity), changes induced by interactions with the system itself (active non-stationarity), or both (hybrid non-stationarity). In this work, we take the first steps towards the fundamental challenge of on-policy and off-policy evaluation amidst structured changes due to active, passive, or hybrid non-stationarity. Towards this goal, we make a higher-order stationarity assumption such that non-stationarity results in changes over time, but the way changes happen is fixed. We propose, OPEN, an algorithm that uses a double application of counterfactual reasoning and a novel importance-weighted instrument-variable regression to obtain both a lower bias and a lower variance estimate of the structure in the changes of a policy's past performances. Finally, we show promising results on how OPEN can be used to predict future performances for several domains inspired by real-world applications that exhibit non-stationarity., Comment: Accepted at Thirty-sixth Conference on Neural Information Processing Systems (NeurIPS 2022)

In many domains, the exploration process of reinforcement learning will be too costly as it requires trying out suboptimal policies, resulting in a need for off-policy evaluation, in which a target policy is evaluated based on data collected from a known behaviour policy. In this context, importance sampling estimators provide estimates for the expected return by weighting the trajectory based on the probability ratio of the target policy and the behaviour policy. Unfortunately, such estimators have a high variance and therefore a large mean squared error. This paper proposes state-based importance sampling estimators which reduce the variance by dropping certain states from the computation of the importance weight. To illustrate their applicability, we demonstrate state-based variants of ordinary importance sampling, weighted importance sampling, per-decision importance sampling, incremental importance sampling, doubly robust off-policy evaluation, and stationary density ratio estimation. Experiments in four domains show that state-based methods consistently yield reduced variance and improved accuracy compared to their traditional counterparts.

Molecular strong coupling offers exciting prospects in physics, chemistry and materials science. Whilst attention has been focused on developing realistic models for the molecular systems, the important role played by the entire photonic mode structure of the optical cavities has been less explored. We show that the effectiveness of molecular strong coupling may be critically dependent on cavity finesse. Specifically we only see emission associated with a dispersive lower polariton for cavities with sufficient finesse. By developing an analytical model of cavity photoluminescence in a multimode structure we clarify the role of finite-finesse in polariton formation, and show that lowering the finesse reduces the extent of the mixing of light and matter in polariton states. We suggest that the detailed nature of the photonic modes supported by a cavity will be as important in developing a coherent framework for molecular strong coupling as the inclusion of realistic molecular models.

Recent research has shown that seemingly fair machine learning models, when used to inform decisions that have an impact on peoples' lives or well-being (e.g., applications involving education, employment, and lending), can inadvertently increase social inequality in the long term. This is because prior fairness-aware algorithms only consider static fairness constraints, such as equal opportunity or demographic parity. However, enforcing constraints of this type may result in models that have negative long-term impact on disadvantaged individuals and communities. We introduce ELF (Enforcing Long-term Fairness), the first classification algorithm that provides high-confidence fairness guarantees in terms of long-term, or delayed, impact. We prove that the probability that ELF returns an unfair solution is less than a user-specified tolerance and that (under mild assumptions), given sufficient training data, ELF is able to find and return a fair solution if one exists. We show experimentally that our algorithm can successfully mitigate long-term unfairness., Comment: 24 pages, 5 figures

Model-based reinforcement learning promises to learn an optimal policy from fewer interactions with the environment compared to model-free reinforcement learning by learning an intermediate model of the environment in order to predict future interactions. When predicting a sequence of interactions, the rollout length, which limits the prediction horizon, is a critical hyperparameter as accuracy of the predictions diminishes in the regions that are further away from real experience. As a result, with a longer rollout length, an overall worse policy is learned in the long run. Thus, the hyperparameter provides a trade-off between quality and efficiency. In this work, we frame the problem of tuning the rollout length as a meta-level sequential decision-making problem that optimizes the final policy learned by model-based reinforcement learning given a fixed budget of environment interactions by adapting the hyperparameter dynamically based on feedback from the learning process, such as accuracy of the model and the remaining budget of interactions. We use model-free deep reinforcement learning to solve the meta-level decision problem and demonstrate that our approach outperforms common heuristic baselines on two well-known reinforcement learning environments.

Entanglement is a powerful concept with an enormous potential for scientific and technological advances. A central focus in modern research is to extend the generation and control of entangled states from few to many qubits, and protect them against decoherence. Optical photons play a prominent role as these qubit carriers are naturally robust and easy to manipulate. However, the most successful technique to date for creating photonic entanglement is inherently probabilistic and therefore subject to severe scalability limitations. Here we avoid these by implementing a deterministic protocol with a single memory atom in a cavity. We interleave controlled single-photon emissions with tailored atomic qubit rotations to efficiently grow Greenberger-Horne-Zeilinger states of up to 14 photons and linear cluster states of up to 12 photons with a fidelity lower bounded by 76(6)% and 56(4)%, respectively. Thanks to a source-to-detection efficiency of 43.18(7)% per photon we measure these large states about once every minute, orders of magnitude faster than in any previous experiment. In the future, this rate could be increased even further, the scheme could be extended to two atoms in a cavity, or several sources could be quantum mechanically coupled, to generate higher-dimensional cluster states. Overcoming the limitations encountered by probabilistic schemes for photonic entanglement generation, our results may offer a way towards scalable measurement-based quantum computation and communication.