Search

Diffusion policies have achieved superior performance in imitation learning and offline reinforcement learning (RL) due to their rich expressiveness. However, the vanilla diffusion training procedure requires samples from target distribution, which is impossible in online RL since we cannot sample from the optimal policy, making training diffusion policies highly non-trivial in online RL. Backpropagating policy gradient through the diffusion process incurs huge computational costs and instability, thus being expensive and impractical. To enable efficient diffusion policy training for online RL, we propose Soft Diffusion Actor-Critic (SDAC), exploiting the viewpoint of diffusion models as noise-perturbed energy-based models. The proposed SDAC relies solely on the state-action value function as the energy functions to train diffusion policies, bypassing sampling from the optimal policy while maintaining lightweight computations. We conducted comprehensive comparisons on MuJoCo benchmarks. The empirical results show that SDAC outperforms all recent diffusion-policy online RLs on most tasks, and improves more than 120% over soft actor-critic on complex locomotion tasks such as Humanoid and Ant., Comment: 19 pages, 4 figures

Femtosecond laser light can transfer spin angular momentum between magnetic subspecies that exhibit hybridized valence bands within an alloy or compound, and represents the fastest route for manipulating the magnetization of a material. To date, ultrafast spin transfer has predominantly been explained in terms of the initial and final states available for laser excitation. Here, by comparing the measured and calculated dynamics across the entire $M$-edges of two very similar Heusler compounds, $Co_2MnGa$ and $Co_2MnGe$ as well as a sample of elemental Co, we find that simply accounting for the initial and final electron states available for laser excitation cannot alone explain the experimental observations. The influence of spin lifetimes must also be included, due to the shifting of the Fermi level upon replacing Ga with Ge, or the presence of crystalline disorder. This explains why the ordered $L2_1$ phase of $Co_2MnGa$ demonstrates strong laser-induced magnetic signal enhancements across the entire Co-edge, while similar enhancements were not observed in partially disordered $Co_2MnGe$. Although intra-site spin-transfers were expected in the minority channel in pure Co due to the presence of many more available states in the minority channel above the Fermi level, no such signal was observed due to very short few-femtosecond spin lifetimes in a metal. Finally, we identify key regions in the magnetic asymmetry where a transiently enhanced signal could be misinterpreted as a light-induced spin-transfer signature.

In the relentless pursuit of quantum computational advantage, we present a significant advancement with the development of Zuchongzhi 3.0. This superconducting quantum computer prototype, comprising 105 qubits, achieves high operational fidelities, with single-qubit gates, two-qubit gates, and readout fidelity at 99.90%, 99.62% and 99.18%, respectively. Our experiments with an 83-qubit, 32-cycle random circuit sampling on Zuchongzhi 3.0 highlight its superior performance, achieving one million samples in just a few hundred seconds. This task is estimated to be infeasible on the most powerful classical supercomputers, Frontier, which would require approximately $6.4\times 10^9$ years to replicate the task. This leap in processing power places the classical simulation cost six orders of magnitude beyond Google's SYC-67 and SYC-70 experiments [Nature 634, 328(2024)], firmly establishing a new benchmark in quantum computational advantage. Our work not only advances the frontiers of quantum computing but also lays the groundwork for a new era where quantum processors play an essential role in tackling sophisticated real-world challenges.

To address deviations from expected performance in stochastic systems, we propose a risk-sensitive control synthesis method to minimize certain risk measures over the limiting stationary distribution. Specifically, we extend Worst-case Conditional Value-at-Risk (W-CVaR) optimization for Linear Time-invariant (LTI) systems to handle nonzero-mean noise and affine controllers, using only the first and second moments of noise, which enhances robustness against model uncertainty. Highlighting the strong coupling between the linear and bias terms of the controller, we reformulate the synthesis problem as a Bilinear Matrix Inequality (BMI), and propose an alternating optimization algorithm with guaranteed convergence. Finally, we demonstrate the numerical performance of our approach in two representative settings, which shows that the proposed algorithm successfully synthesizes risk-sensitive controllers that outperform the na\"ive LQR baseline., Comment: 8 pages, 4 figures, 2 illustrations

Off-policy evaluation (OPE) is one of the most fundamental problems in reinforcement learning (RL) to estimate the expected long-term payoff of a given target policy with only experiences from another behavior policy that is potentially unknown. The distribution correction estimation (DICE) family of estimators have advanced the state of the art in OPE by breaking the curse of horizon. However, the major bottleneck of applying DICE estimators lies in the difficulty of solving the saddle-point optimization involved, especially with neural network implementations. In this paper, we tackle this challenge by establishing a linear representation of value function and stationary distribution correction ratio, i.e., primal and dual variables in the DICE framework, using the spectral decomposition of the transition operator. Such primal-dual representation not only bypasses the non-convex non-concave optimization in vanilla DICE, therefore enabling an computational efficient algorithm, but also paves the way for more efficient utilization of historical data. We highlight that our algorithm, SpectralDICE, is the first to leverage the linear representation of primal-dual variables that is both computation and sample efficient, the performance of which is supported by a rigorous theoretical sample complexity guarantee and a thorough empirical evaluation on various benchmarks., Comment: 29 pages, 5 figures

Network Markov Decision Processes (MDPs), a popular model for multi-agent control, pose a significant challenge to efficient learning due to the exponential growth of the global state-action space with the number of agents. In this work, utilizing the exponential decay property of network dynamics, we first derive scalable spectral local representations for network MDPs, which induces a network linear subspace for the local $Q$-function of each agent. Building on these local spectral representations, we design a scalable algorithmic framework for continuous state-action network MDPs, and provide end-to-end guarantees for the convergence of our algorithm. Empirically, we validate the effectiveness of our scalable representation-based approach on two benchmark problems, and demonstrate the advantages of our approach over generic function approximation approaches to representing the local $Q$-functions., Comment: Updated title, corrected an issue with an author's name

In Bayesian optimization, a black-box function is maximized via the use of a surrogate model. We apply distributed Thompson sampling, using a Gaussian process as a surrogate model, to approach the multi-agent Bayesian optimization problem. In our distributed Thompson sampling implementation, each agent receives sampled points from neighbors, where the communication network is encoded in a graph; each agent utilizes their own Gaussian process to model the objective function. We demonstrate theoretical bounds on Bayesian average regret and Bayesian simple regret, where the bound depends on the structure of the communication graph. Unlike in batch Bayesian optimization, this bound is applicable in cases where the communication graph amongst agents is constrained. When compared to sequential single-agent Thompson sampling, our bound guarantees faster convergence with respect to time as long as the communication graph is connected. We confirm the efficacy of our algorithm with numerical simulations on traditional optimization test functions, demonstrating the significance of graph connectivity on improving regret convergence., Comment: To appear in IEEE Control Systems Letters (L-CSS). 11 pages

Lifelong learning in artificial intelligence (AI) aims to mimic the biological brain's ability to continuously learn and retain knowledge, yet it faces challenges such as catastrophic forgetting. Recent neuroscience research suggests that neural activity in biological systems undergoes representational drift, where neural responses evolve over time, even with consistent inputs and tasks. We hypothesize that representational drift can alleviate catastrophic forgetting in AI during new task acquisition. To test this, we introduce DriftNet, a network designed to constantly explore various local minima in the loss landscape while dynamically retrieving relevant tasks. This approach ensures efficient integration of new information and preserves existing knowledge. Experimental studies in image classification and natural language processing demonstrate that DriftNet outperforms existing models in lifelong learning. Importantly, DriftNet is scalable in handling a sequence of tasks such as sentiment analysis and question answering using large language models (LLMs) with billions of parameters on a single Nvidia A100 GPU. DriftNet efficiently updates LLMs using only new data, avoiding the need for full dataset retraining. Tested on GPT-2 and RoBERTa, DriftNet is a robust, cost-effective solution for lifelong learning in LLMs. This study not only advances AI systems to emulate biological learning, but also provides insights into the adaptive mechanisms of biological neural systems, deepening our understanding of lifelong learning in nature.

In stochastic systems, risk-sensitive control balances performance with resilience to less likely events. Although existing methods rely on finite-horizon risk criteria, this paper introduces \textit{limiting-risk criteria} that capture long-term cumulative risks through probabilistic limiting theorems. Extending the Linear Quadratic Regulation (LQR) framework, we incorporate constraints on these limiting-risk criteria derived from the asymptotic behavior of cumulative costs, accounting for extreme deviations. Using tailored Functional Central Limit Theorems (FCLT), we demonstrate that the time-correlated terms in the limiting-risk criteria converge under strong ergodicity, and establish conditions for convergence in non-stationary settings while characterizing the distribution and providing explicit formulations for the limiting variance of the risk functional. The FCLT is developed by applying ergodic theory for Markov chains and obtaining \textit{uniform ergodicity} of the controlled process. For quadratic risk functionals on linear dynamics, in addition to internal stability, the uniform ergodicity requires the (possibly heavy-tailed) dynamic noise to have a finite fourth moment. This offers a clear path to quantifying long-term uncertainty. We also propose a primal-dual constrained policy optimization method that optimizes the average performance while ensuring limiting-risk constraints are satisfied. Our framework offers a practical, theoretically guaranteed approach for long-term risk-sensitive control, backed by convergence guarantees and validations through simulations.

The growing demand for AI training data has transformed data annotation into a global industry, but traditional approaches relying on human annotators are often time-consuming, labor-intensive, and prone to inconsistent quality. We propose the Model-in-the-Loop (MILO) framework, which integrates AI/ML models into the annotation process. Our research introduces a collaborative paradigm that leverages the strengths of both professional human annotators and large language models (LLMs). By employing LLMs as pre-annotation and real-time assistants, and judges on annotator responses, MILO enables effective interaction patterns between human annotators and LLMs. Three empirical studies on multimodal data annotation demonstrate MILO's efficacy in reducing handling time, improving data quality, and enhancing annotator experiences. We also introduce quality rubrics for flexible evaluation and fine-grained feedback on open-ended annotations. The MILO framework has implications for accelerating AI/ML development, reducing reliance on human annotation alone, and promoting better alignment between human and machine values.

Interactive preference learning systems infer human preferences by presenting queries as pairs of options and collecting binary choices. Although binary choices are simple and widely used, they provide limited information about preference strength. To address this, we leverage human response times, which are inversely related to preference strength, as an additional signal. We propose a computationally efficient method that combines choices and response times to estimate human utility functions, grounded in the EZ diffusion model from psychology. Theoretical and empirical analyses show that for queries with strong preferences, response times complement choices by providing extra information about preference strength, leading to significantly improved utility estimation. We incorporate this estimator into preference-based linear bandits for fixed-budget best-arm identification. Simulations on three real-world datasets demonstrate that using response times significantly accelerates preference learning compared to choice-only approaches. Additional materials, such as code, slides, and talk video, are available at https://shenlirobot.github.io/pages/NeurIPS24.html, Comment: NeurIPS 2024 (Oral) camera ready

In this study, we present MedS-Bench, a comprehensive benchmark designed to evaluate the performance of large language models (LLMs) in clinical contexts. Unlike existing benchmarks that focus on multiple-choice question answering, MedS-Bench spans 11 high-level clinical tasks, including clinical report summarization, treatment recommendations, diagnosis, named entity recognition, and medical concept explanation, among others. We evaluated six leading LLMs, e.g., MEDITRON, Mistral, InternLM 2, Llama 3, GPT-4, and Claude-3.5 using few-shot prompting, and found that even the most sophisticated models struggle with these complex tasks. To address these limitations, we developed MedS-Ins, a large-scale instruction tuning dataset for medicine. MedS-Ins comprises 58 medically oriented language corpora, totaling 13.5 million samples across 122 tasks. To demonstrate the dataset's utility, we conducted a proof-of-concept experiment by performing instruction tuning on a lightweight, open-source medical language model. The resulting model, MMedIns-Llama 3, significantly outperformed existing models across nearly all clinical tasks. To promote further advancements in the application of LLMs to clinical challenges, we have made the MedS-Ins dataset fully accessible and invite the research community to contribute to its expansion.Additionally, we have launched a dynamic leaderboard for MedS-Bench, which we plan to regularly update the test set to track progress and enhance the adaptation of general LLMs to the medical domain. Leaderboard: https://henrychur.github.io/MedS-Bench/. Github: https://github.com/MAGIC-AI4Med/MedS-Ins.

This paper is concerned with a class of linear-quadratic stochastic large-population problems with partial information, where the individual agent only has access to a noisy observation process related to the state. The dynamics of each agent follows a linear stochastic differential equation driven by individual noise, and all agents are coupled together via the control average term. Using the mean-field game approach and the backward separation principle with a state decomposition technique, the decentralized optimal control can be obtained in the open-loop form through a forward-backward stochastic differential equation with the conditional expectation. The optimal filtering equation is also provided. By the decoupling method, the decentralized optimal control can also be further presented as the feedback of state filtering via the Riccati equation. The explicit solution of the control average limit is given, and the consistency condition system is discussed. Moreover, the related $\varepsilon$-Nash equilibrium property is verified. To illustrate the good performance of theoretical results, an example in finance is studied.

We present an all-fiber, fully open Fabry-Perot interferometer (FPI) cavity that is suitable for fluidic measurement applications. Fabrication of the FPI involves the alignment and bonding of three optical fiber sections using either ceramic glue or low-temperature melting glass. The fabrication procedure allows the protection of the cleaved optical fiber end faces, which serve as the two mirrors of the FPI, from damage, thus ensuring the high visibility of the FPI sensor. The FPI's complete openness permits the analyte of interest fluids to flow smoothly into the cavity and interact directly with the light, obviating the need for additional assistance. The fabrication experiment demonstrates that the fabrication procedure can readily achieve a visibility of over 20 dB. Refractive index testing indicates that the sensor exhibits a sensitivity of over 1116 nm/RIU within the range of 1.334-1.375. A comparison of temperature investigations indicates that the fully open cavity FPI fabricated by bonding with low-temperature melting glass exhibits relatively lower temperature immunity than that bonded with ceramic glue. Both have a relatively low temperature fluctuation within the temperature range of 40{\deg}C-100{\deg}C, with less than 3 nm and 4.5 nm in the over 60{\deg}C changes, respectively. Our proposed fully open FPI is an economical, robust, and simple-to-fabricate structure with the potential for mass production. This renders it an appealing option for practical optofluidics applications., Comment: 6 pages, 10 figures

Electron-positron pair production is investigated in the multiphoton regime under polarized fields. Partial wave analysis can be applied to reveal the momentum spectral structure and identify the positions of valleys/peaks of the multiphoton rings by combining with $CP$ conservation. Moreover, considering the spin states of created pairs in partial wave analysis, the particle orbital angular momentum associated with the valleys/peaks can also be identified from multiphoton rings. It is found that the results for angular distribution by this approximate analytical treatment and by the Dirac-Heisenberg-Wigner formalism are consistent with each other. The present results are helpful for understanding the angular momentum information in momentum spectra, and they also have an implication for revealing the vortex structure near the valleys., Comment: 17 pages, 4 figures, 2 tables

Topological materials are of great interest because they can support metallic edge or surface states that are robust against perturbations, with the potential for technological applications. Here we experimentally explore the light-induced non-equilibrium properties of two distinct topological phases in NaCd4As3: a topological crystalline insulator (TCI) phase and a topological insulator (TI) phase. This material has surface states that are protected by mirror symmetry in the TCI phase at room temperature, while it undergoes a structural phase transition to a TI phase below 200 K. After exciting the TI phase by an ultrafast laser pulse, we observe a leading band edge shift of >150 meV, that slowly builds up and reaches a maximum after ~0.6 ps, and that persists for ~8 ps. The slow rise time of the excited electron population and electron temperature suggests that the electronic and structural orders are strongly coupled in this TI phase. It also suggests that the directly excited electronic states and the probed electronic states are weakly coupled. Both couplings are likely due to a partial relaxation of the lattice distortion, which is known to be associated with the TI phase. In contrast, no distinct excited state is observed in the TCI phase immediately or after photoexcitation, which we attribute to the low density of states and phase space available near the Fermi level. Our results show how ultrafast laser excitation can reveal the distinct excited states and interactions in phase-rich topological materials., Comment: 21 pages, 9 figures, added theoretical insight in the discussion section and modified abstract, corrected typos and rephrased sentences, results and figures unchanged, added a co-author involved in sample preparation

$TbMn_{6}Sn_{6}$ is a ferrimagnetic material which exhibits a highly unusual phase transition near room temperature where spins remain collinear while the total magnetic moment rotates from out-of-plane to in-plane. The mechanisms underlying this phenomenon have been studied in the quasi-static limit and the reorientation has been attributed to the competing anisotropies of Tb and Mn, whose magnetic moments have very different temperature dependencies. In this work, we present the first measurement of the spin-reorientation transition in $TbMn_{6}Sn_{6}$. By probing very small signals with the transverse magneto-optical Kerr effect (TMOKE) at the Mn M-edge, we show that the re-orientation timescale spans from 12 ps to 24 ps, depending on the laser excitation fluence. We then verify these data with a simple model of spin precession with a temperature-dependent magnetocrystalline anisotropy field to show that the spin reorientation timescale is consistent with the reorientation being driven by very large anisotropies energies on approximately $\approx$ meV scales. Promisingly, the model predicts a possibility of 180o reorientation of the out-of-plane moment over a range of excitation fluences. This could facilitate optically controlled magnetization switching between very stable ground states, which could have useful applications in spintronics or data storage.

We propose a new Sauter-like field model with combinatorial multiple potentials consisting of a deep slow-varying and some shallow fast-varying potentials. The dynamically assisted Sauter-Schwinger effect on the pair production is found by using the computational quantum field theory. The enhanced pair production is found to be significant at about one order increasing for multiple potentials rather than single potential. In case of dominated by Schwinger mechanism, the obvious time effect leads to electrons concentrating at the two edges of the potential, meanwhile, the momentum locates at the zero nearby. In contrary, however, for the multiphoton processes, the pair generation makes the electrons distributing outside the potential and the momentum appearing multiple peaks far away from zero and evenly evolving toward a step-like structure. An interesting finding is that the particles of pair produced in the alternating potential has a quasi-monoenergetic structure compared to the oscillating potential well or/and potential barrier, which is helpful to achieve the high quality positron source., Comment: 32 pages, 19 figures

Many multi-dimensional (M-D) graph signals appear in the real world, such as digital images, sensor network measurements and temperature records from weather observation stations. It is a key challenge to design a transform method for processing these graph M-D signals in the linear canonical transform domain. This paper proposes the two-dimensional graph linear canonical transform based on the central discrete dilated Hermite function (2-D CDDHFs-GLCT) and the two-dimensional graph linear canonical transform based on chirp multiplication-chirp convolution-chirp multiplication decomposition (2-D CM-CC-CM-GLCT). Then, extending 2-D CDDHFs-GLCT and 2-D CM-CC-CM-GLCT to M-D CDDHFs-GLCT and M-D CM-CC-CM-GLCT. In terms of the computational complexity, additivity and reversibility, M-D CDDHFs-GLCT and M-D CM-CC-CM-GLCT are compared. Theoretical analysis shows that the computational complexity of M-D CM-CC-CM-GLCT algorithm is obviously reduced. Simulation results indicate that M-D CM-CC-CM-GLCT achieves comparable additivity to M-D CDDHFs-GLCT, while M-D CM-CC-CM-GLCT exhibits better reversibility. Finally, M-D GLCT is applied to data compression to show its application advantages. The experimental results reflect the superiority of M-D GLCT in the algorithm design and implementation of data compression., Comment: arXiv admin note: text overlap with arXiv:2407.17513

The graph linear canonical transform (GLCT) is presented as an extension of the graph Fourier transform (GFT) and the graph fractional Fourier transform (GFrFT), offering more flexibility as an effective tool for graph signal processing. In this paper, we introduce a GLCT based on chirp multiplication-chirp convolution-chirp multiplication decomposition (CM-CC-CM-GLCT), which irrelevant to sampling periods and without oversampling operation. Various properties and special cases of the CM-CC-CM-GLCT are derived and discussed. In terms of computational complexity, additivity, and reversibility, we compare the CM-CC-CM-GLCT and the GLCT based on the central discrete dilated Hermite function (CDDHFs-GLCT). Theoretical analysis demonstrates that the computational complexity of the CM-CC-CM-GLCT is significantly reduced. Simulation results indicate that the CM-CC-CM-GLCT achieves similar additivity to the CDDHFs-GLCT. Notably, the CM-CC-CM-GLCT exhibits better reversibility.

While there are numerous works in multi-agent reinforcement learning (MARL), most of them focus on designing algorithms and proving convergence to a Nash equilibrium (NE) or other equilibrium such as coarse correlated equilibrium. However, NEs can be non-unique and their performance varies drastically. Thus, it is important to design algorithms that converge to Nash equilibrium with better rewards or social welfare. In contrast, classical game theory literature has extensively studied equilibrium selection for multi-agent learning in normal-form games, demonstrating that decentralized learning algorithms can asymptotically converge to potential-maximizing or Pareto-optimal NEs. These insights motivate this paper to investigate equilibrium selection in the MARL setting. We focus on the stochastic game model, leveraging classical equilibrium selection results from normal-form games to propose a unified framework for equilibrium selection in stochastic games. The proposed framework is highly modular and can extend various learning rules and their corresponding equilibrium selection results from normal-form games to the stochastic game setting.

Any-to-any singing voice conversion (SVC) is an interesting audio editing technique, aiming to convert the singing voice of one singer into that of another, given only a few seconds of singing data. However, during the conversion process, the issue of timbre leakage is inevitable: the converted singing voice still sounds like the original singer's voice. To tackle this, we propose a latent diffusion model for SVC (LDM-SVC) in this work, which attempts to perform SVC in the latent space using an LDM. We pretrain a variational autoencoder structure using the noted open-source So-VITS-SVC project based on the VITS framework, which is then used for the LDM training. Besides, we propose a singer guidance training method based on classifier-free guidance to further suppress the timbre of the original singer. Experimental results show the superiority of the proposed method over previous works in both subjective and objective evaluations of timbre similarity., Comment: Accepted by Interspeech 2024

This survey explores the geometric perspective on policy optimization within the realm of feedback control systems, emphasizing the intrinsic relationship between control design and optimization. By adopting a geometric viewpoint, we aim to provide a nuanced understanding of how various ``complete parameterization'' -- referring to the policy parameters together with its Riemannian geometry -- of control design problems, influence stability and performance of local search algorithms. The paper is structured to address key themes such as policy parameterization, the topology and geometry of stabilizing policies, and their implications for various (non-convex) dynamic performance measures. We focus on a few iconic control design problems, including the Linear Quadratic Regulator (LQR), Linear Quadratic Gaussian (LQG) control, and $\mathcal{H}_\infty$ control. In particular, we first discuss the topology and Riemannian geometry of stabilizing policies, distinguishing between their static and dynamic realizations. Expanding on this geometric perspective, we then explore structural properties of the aforementioned performance measures and their interplay with the geometry of stabilizing policies in presence of policy constraints; along the way, we address issues such as spurious stationary points, symmetries of dynamic feedback policies, and (non-)smoothness of the corresponding performance measures. We conclude the survey with algorithmic implications of policy optimization in feedback design.