Search

We propose a novel parameter-efficient training (PET) method for large language models that adapts models to downstream tasks by optimizing a small subset of the existing model parameters. Unlike prior methods, this subset is not fixed in location but rather which parameters are modified evolves over the course of training. This dynamic parameter selection can yield good performance with many fewer parameters than extant methods. Our method enables a seamless scaling of the subset size across an arbitrary proportion of the total model size, while popular PET approaches like prompt tuning and LoRA cover only a small part of this spectrum. We match or outperform prompt tuning and LoRA in most cases on a variety of NLP tasks (MT, QA, GSM8K, SuperGLUE) for a given parameter budget across different model families and sizes., Comment: NeurIPS 2024 Workshop on Adaptive Foundation Models

The Nash equilibrium (NE) is fundamental game-theoretic concept for characterizing stability in static strategic form games. However, at times, NE fails to capture outcomes in dynamic settings, where players' actions evolve over time in response to one another. In such cases, game dynamics fail to converge to an NE, instead exhibiting cyclic or oscillatory patterns. To address this, we introduce the concept of an 'equilibrium cycle' (EC). Unlike NE, which defines a fixed point of mutual best responses, an EC is a set-valued solution concept designed to capture the asymptotic or long-term behavior of dynamic interactions, even when a traditional best response does not exist. The EC identifies a minimal rectangular set of action profiles that collectively capture oscillatory game dynamics, effectively generalizing the notion of stability beyond static equilibria. An EC satisfies three important properties: \textit{stability} against external deviations (ensuring robustness), \textit{unrest} with respect to internal deviations (driving oscillation), and \textit{minimality} (defining the solution's tightness). This set-valued outcome generalizes the minimal curb set to discontinuous games, where best responses may not exist. In finite games, the EC also relates to sink strongly connected components (SCCs) of the best response graph.

Josephson element-based parametric amplifiers (JPAs) typically require rf pump power that is several orders of magnitude stronger than the maximum signal power they can handle. The low power efficiency and strong pump leakage towards signal circuitry could be critical concerns in application. In this work, we discuss how to optimize the pump coupling scheme for a three-wave mixing JPA by employing microwave filtering techniques, with the goal of maximizing the pump power efficiency and minimize pump leakage without sacrificing other properties of interest. We implement the corresponding filter design in a SNAIL-based JPA and demonstrate more than three orders of magnitude improvement in both power efficiency and pump leakage suppression compared to a similar device with regular capacitive coupling, while maintaining state-of-the-art dynamic range and near-quantum-limited noise performance. Furthermore, we show experimentally that the filter-coupled JPA is more robust against noise input from the pump port, exhibiting no significant change in added noise performance with up to 4 K of effective noise temperature at the pump port.

Large language models (LLMs) have demonstrated significant success in natural language processing (NLP) tasks and have shown promising results in other domains such as protein sequence generation. However, there remain salient differences between LLMs used for NLP, which effectively handle multiple tasks and are available in small sizes, and protein language models that are often specialized for specific tasks and only exist in larger sizes. In this work, we introduce two small protein language models, based on Llama-3-8B and Phi-3-mini, that are capable of both uncontrollable and controllable protein generation. For the uncontrollable generation task, our best model achieves an average pLDDT score of 69.75, demonstrating robust performance in generating viable protein structures. For the controllable generation task, in which the model generates proteins according to properties specified in the prompt, we achieve a remarkable average TM-Score of 0.84, indicating high structural similarity to target proteins. We chose 10 properties, including six classes of enzymes, to extend the capabilities of prior protein language models. Our approach utilizes the Low-Rank Adaptor (LoRA) technique, reducing trainable parameters to just 4% of the original model size, lowering computational requirements. By using a subset of the UniRef50 dataset and small models, we reduced the overall training time by 70% without compromising performance. Notably, Phi-3-mini reduced trainable parameters by 60%, decreasing training cost by 30% compared to Llama 3. Consequently, Phi-3 achieved a comparable TM-Score of 0.81, demonstrating that smaller models can match the performance of larger ones, like Llama 3. We also demonstrate the deployment of our models on the energy efficient ET-SoC-1 chip, significantly improving the TPS/W by a factor of 3.

In this work, we present a novel Koopman spectrum-based reachability verification method for nonlinear systems. Contrary to conventional methods that focus on characterizing all potential states of a dynamical system over a presupposed time span, our approach seeks to verify the reachability by assessing the non-emptiness of estimated time-to-reach intervals without engaging in the explicit computation of reachable set. Based on the spectral analysis of the Koopman operator, we reformulate the problem of verifying existence of reachable trajectories into the problem of determining feasible time-to-reach bounds required for system reachability. By solving linear programming (LP) problems, our algorithm can effectively estimate all potential time intervals during which a dynamical system can enter (and exit) target sets from given initial sets over an unbounded time horizon. Finally, we demonstrate our method in challenging settings, such as verifying the reachability between non-convex or even disconnected sets, as well as backward reachability and multiple entries into target sets. Additionally, we validate its applicability in addressing real-world challenges and scalability to high-dimensional systems through case studies in verifying the reachability of the cart-pole and multi-agent consensus systems., Comment: This work has been submitted to the IEEE for possible publication

Demand forecasting faces challenges induced by Peak Events (PEs) corresponding to special periods such as promotions and holidays. Peak events create significant spikes in demand followed by demand ramp down periods. Neural networks like MQCNN and MQT overreact to demand peaks by carrying over the elevated PE demand into subsequent Post-Peak-Event (PPE) periods, resulting in significantly over-biased forecasts. To tackle this challenge, we introduce a neural forecasting model called Split Peak Attention DEcomposition, SPADE. This model reduces the impact of PEs on subsequent forecasts by modeling forecasting as consisting of two separate tasks: one for PEs; and the other for the rest. Its architecture then uses masked convolution filters and a specialized Peak Attention module. We show SPADE's performance on a worldwide retail dataset with hundreds of millions of products. Our results reveal a reduction in PPE degradation by 4.5% and an improvement in PE accuracy by 3.9%, relative to current production models.

Advanced Air Mobility (AAM) operations are expected to transform air transportation while challenging current air traffic management practices. By introducing a novel market-based mechanism, we address the problem of on-demand allocation of capacity-constrained airspace to AAM vehicles with heterogeneous and private valuations. We model airspace and air infrastructure as a collection of contiguous regions with constraints on the number of vehicles that simultaneously enter, stay, or exit each region. Vehicles request access to the airspace with trajectories spanning multiple regions at different times. We use the graph structure of our airspace model to formulate the allocation problem as a path allocation problem on a time-extended graph. To ensure the cost information of AAM vehicles remains private, we introduce a novel mechanism that allocates each vehicle a budget of "air-credits" and anonymously charges prices for traversing the edges of the time-extended graph. We seek to compute a competitive equilibrium that ensures that: (i) capacity constraints are satisfied, (ii) a strictly positive resource price implies that the sector capacity is fully utilized, and (iii) the allocation is integral and optimal for each AAM vehicle given current prices, without requiring access to individual vehicle utilities. However, a competitive equilibrium with integral allocations may not always exist. We provide sufficient conditions for the existence and computation of a fractional-competitive equilibrium, where allocations can be fractional. Building on these theoretical insights, we propose a distributed, iterative, two-step algorithm that: 1) computes a fractional competitive equilibrium, and 2) derives an integral allocation from this equilibrium. We validate the effectiveness of our approach in allocating trajectories for two emerging urban air mobility services: drone delivery and air taxis., Comment: 31 pages, 7 figures, 3 tables

We present numerical results for the effects of influence by high-amplitude periodic pulse series on a network of nonlocally coupled Hindmarsh-Rose neurons with 2D geometry of the topology. We consider the case when the pulse amplitude is larger than the amplitude of oscillations in the autonomous network for a wide range of pulse frequencies. An initial regime in the network is a spiral wave chimera. We show that the effects of external influence strongly depend on a balance between the pulse frequency and frequencies of the spectral peaks of the autonomous network. Except for the destructive role of the pulses, when they lead to loss of stability of the initial regime, we have also revealed a constructive role. We have found for the first time the emergence of a new type of multi-front spiral waves, when the wavefront represents a set of several close fronts, and the wave dynamics are significantly different from common spiral waves: neurons oscillate independently to the wave rotation, the rotation velocity is in many times less than for the common spiral wave, etc. We have also discovered several types of cluster spatiotemporal structures induced by the pulses.

Incremental stability of dynamical systems ensures the convergence of trajectories from different initial conditions towards each other rather than a fixed trajectory or equilibrium point. Here, we introduce and characterize a novel class of incremental Lyapunov functions, an incremental stability notion known as Incremental Input-to-State practical Stability ({\delta}-ISpS). Using Gaussian Process, we learn the unknown dynamics of a class of control systems. We then present a backstepping control design scheme that provides state-feedback controllers that render the partially unknown control system {\delta}-ISpS. To show the effectiveness of the proposed controller, we implement it in two case studies., Comment: 15 page, 4 figures

Parenting brings emotional and physical challenges, from balancing work, childcare, and finances to coping with exhaustion and limited personal time. Yet, one in three parents never seek support. AI systems potentially offer stigma-free, accessible, and affordable solutions. Yet, user adoption often fails due to issues with explainability and reliability. To see if these issues could be solved using a co-design approach, we developed and tested NurtureBot, a wellbeing support assistant for new parents. 32 parents co-designed the system through Asynchronous Remote Communities method, identifying the key challenge as achieving a "successful chat". Aspart of co-design, parents role-played as NurturBot, rewriting its dialogues to improve user understanding, control, and outcomes. The refined prototype evaluated by 32 initial and 46 new parents, showed improved user experience and usability, with final CUQ score of 91.3/100, demonstrating successful interaction patterns. Our process revealed useful interaction design lessons for effective AI parenting support.

We study a monolayer of metal balls under periodic chiral driving in the horizontal plane. Energy dissipation occurs in this system via (i) inelastic collisions and (ii) frictional interaction with the substrate. We show that below a density-dependent critical drive, the system phase separates into a fluid phase coexisting with a solid phase. Unlike ordinary coexistence, however, the system does not obey the lever rule, as the fluid-phase density depends on the overall particle density. Additionally, the pressure is discontinuous across the fluid-solid interface, accompanied by a chiral edge current at the interface.

It is well known that descents and excedances are equidistributed in the symmetric group. We show that the descent and excedance enumerators, summed over permutations with a fixed first letter are identical when we perform a simple change of the first letter. We generalize this to type B and other colored permutation groups. We are led to defining descents and excedances through linear orders. With respect to a particular order, when the number of colors is even, we get a result that generalizes the type B results. Lastly, we get a type B counterpart of Conger's result which refines the well known Carlitz identity., Comment: 14 pages

Advancements in Large Language Models (LLMs) are revolutionizing the development of autonomous agentic systems by enabling dynamic, context-aware task decomposition and automated tool selection. These sophisticated systems possess significant automation potential across various industries, managing complex tasks, interacting with external systems to enhance knowledge, and executing actions independently. This paper presents three primary contributions to advance this field: - Advanced Agentic Framework: A system that handles multi-hop queries, generates and executes task graphs, selects appropriate tools, and adapts to real-time changes. - Novel Evaluation Metrics: Introduction of Node F1 Score, Structural Similarity Index (SSI), and Tool F1 Score to comprehensively assess agentic systems. - Specialized Dataset: Development of an AsyncHow-based dataset for analyzing agent behavior across different task complexities. Our findings reveal that asynchronous and dynamic task graph decomposition significantly enhances system responsiveness and scalability, particularly for complex, multi-step tasks. Detailed analysis shows that structural and node-level metrics are crucial for sequential tasks, while tool-related metrics are more important for parallel tasks. Specifically, the Structural Similarity Index (SSI) is the most significant predictor of performance in sequential tasks, and the Tool F1 Score is essential for parallel tasks. These insights highlight the need for balanced evaluation methods that capture both structural and operational dimensions of agentic systems. Additionally, our evaluation framework, validated through empirical analysis and statistical testing, provides valuable insights for improving the adaptability and reliability of agentic systems in dynamic environments., Comment: 38th Conference on Neural Information Processing Systems (NeurIPS 2024), NeurIPS 2024 Workshop on Open-World Agents

Distributed time series data presents a challenge for federated learning, as clients often possess different feature sets and have misaligned time steps. Existing federated time series models are limited by the assumption of perfect temporal or feature alignment across clients. In this paper, we propose FedTDD, a novel federated time series diffusion model that jointly learns a synthesizer across clients. At the core of FedTDD is a novel data distillation and aggregation framework that reconciles the differences between clients by imputing the misaligned timesteps and features. In contrast to traditional federated learning, FedTDD learns the correlation across clients' time series through the exchange of local synthetic outputs instead of model parameters. A coordinator iteratively improves a global distiller network by leveraging shared knowledge from clients through the exchange of synthetic data. As the distiller becomes more refined over time, it subsequently enhances the quality of the clients' local feature estimates, allowing each client to then improve its local imputations for missing data using the latest, more accurate distiller. Experimental results on five datasets demonstrate FedTDD's effectiveness compared to centralized training, and the effectiveness of sharing synthetic outputs to transfer knowledge of local time series. Notably, FedTDD achieves 79.4% and 62.8% improvement over local training in Context-FID and Correlational scores.

Mathematical models are established tools to assist in outbreak response. They help characterise complex patterns in disease spread, simulate control options to assist public health authorities in decision-making, and longer-term operational and financial planning. In the context of vaccine-preventable diseases (VPDs), vaccines are one of the most-cost effective outbreak response interventions, with the potential to avert significant morbidity and mortality through timely delivery. Models can contribute to the design of vaccine response by investigating the importance of timeliness, identifying high-risk areas, prioritising the use of limited vaccine supply, highlighting surveillance gaps and reporting, and determining the short- and long-term benefits. In this review, we examine how models have been used to inform vaccine response for 10 VPDs, and provide additional insights into the challenges of outbreak response modelling, such as data gaps, key vaccine-specific considerations, and communication between modellers and stakeholders. We illustrate that while models are key to policy-oriented outbreak vaccine response, they can only be as good as the surveillance data that inform them., Comment: 19 pages including references and abstract, 1 figure

Deriving inference from heterogeneous inputs (such as images, text, and audio) is an important skill for humans to perform day-to-day tasks. A similar ability is desirable for the development of advanced Artificial Intelligence (AI) systems. While state-of-the-art models are rapidly closing the gap with human-level performance on diverse computer vision and NLP tasks separately, they struggle to solve tasks that require joint reasoning over visual and textual modalities. Inspired by GLUE (Wang et. al., 2018)- a multitask benchmark for natural language understanding, we propose VL-GLUE in this paper. VL-GLUE consists of over 100k samples spanned across seven different tasks, which at their core require visuo-linguistic reasoning. Moreover, our benchmark comprises of diverse image types (from synthetically rendered figures, and day-to-day scenes to charts and complex diagrams) and includes a broad variety of domain-specific text (from cooking, politics, and sports to high-school curricula), demonstrating the need for multi-modal understanding in the real-world. We show that this benchmark is quite challenging for existing large-scale vision-language models and encourage development of systems that possess robust visuo-linguistic reasoning capabilities., Comment: 18 pages, 7 figures

Paints and coatings undergo a variety of physical and chemical changes under environmental exposures. Accurate prediction of these changes is important for the applications of coatings. This work presents a novel approach to modeling accelerated weathering of coating by combining concurrent physical and chemical processes. The model integrates key factors influencing coating degradation and employs a multi-scale framework to capture macro-scale physical changes and micro-level chemical transformations. The chemical component/model simulates photo-degradation reactions using kinetic equations, while the physical component uses Monte Carlo simulations where repeated random events develop surface erosion. The surface topography and chemistry of coating is generated statistically through the physical model. The variations in surface topography and chemistry of coating are correlated with the chemical changes from the chemical model, resulting in estimation of both physical and chemical changes in the coating during real accelerated weathering time. Results demonstrate accurate predictions of chemistry changes, surface degradation profiles, roughness, gloss loss, and relative fracture toughness, which are validated successfully with the experimentally available data. This integrated approach provides insight into coating failure mechanisms, enabling accurate service life prediction, and serve as a tool for formulating durable coatings and optimizing testing protocols.