Search

The ability of large language models (LLMs) to mimic human-like intelligence has led to a surge in LLM-based autonomous agents. Though recent LLMs seem capable of planning and reasoning given user instructions, their effectiveness in applying these capabilities for autonomous task solving remains underexplored. This is especially true in enterprise settings, where automated agents hold the promise of a high impact. To fill this gap, we propose WorkArena++, a novel benchmark consisting of 682 tasks corresponding to realistic workflows routinely performed by knowledge workers. WorkArena++ is designed to evaluate the planning, problem-solving, logical/arithmetic reasoning, retrieval, and contextual understanding abilities of web agents. Our empirical studies across state-of-the-art LLMs and vision-language models (VLMs), as well as human workers, reveal several challenges for such models to serve as useful assistants in the workplace. In addition to the benchmark, we provide a mechanism to effortlessly generate thousands of ground-truth observation/action traces, which can be used for fine-tuning existing models. Overall, we expect this work to serve as a useful resource to help the community progress toward capable autonomous agents. The benchmark can be found at https://github.com/ServiceNow/WorkArena/tree/workarena-plus-plus.

We study the use of large language model-based agents for interacting with software via web browsers. Unlike prior work, we focus on measuring the agents' ability to perform tasks that span the typical daily work of knowledge workers utilizing enterprise software systems. To this end, we propose WorkArena, a remote-hosted benchmark of 33 tasks based on the widely-used ServiceNow platform. We also introduce BrowserGym, an environment for the design and evaluation of such agents, offering a rich set of actions as well as multimodal observations. Our empirical evaluation reveals that while current agents show promise on WorkArena, there remains a considerable gap towards achieving full task automation. Notably, our analysis uncovers a significant performance disparity between open and closed-source LLMs, highlighting a critical area for future exploration and development in the field., Comment: 21 pages, 11 figures, preprint

The field of transfer learning is undergoing a significant shift with the introduction of large pretrained models which have demonstrated strong adaptability to a variety of downstream tasks. However, the high computational and memory requirements to finetune or use these models can be a hindrance to their widespread use. In this study, we present a solution to this issue by proposing a simple yet effective way to trade computational efficiency for asymptotic performance which we define as the performance a learning algorithm achieves as compute tends to infinity. Specifically, we argue that zero-shot structured pruning of pretrained models allows them to increase compute efficiency with minimal reduction in performance. We evaluate our method on the Nevis'22 continual learning benchmark that offers a diverse set of transfer scenarios. Our results show that pruning convolutional filters of pretrained models can lead to more than 20% performance improvement in low computational regimes.

A shared goal of several machine learning communities like continual learning, meta-learning and transfer learning, is to design algorithms and models that efficiently and robustly adapt to unseen tasks. An even more ambitious goal is to build models that never stop adapting, and that become increasingly more efficient through time by suitably transferring the accrued knowledge. Beyond the study of the actual learning algorithm and model architecture, there are several hurdles towards our quest to build such models, such as the choice of learning protocol, metric of success and data needed to validate research hypotheses. In this work, we introduce the Never-Ending VIsual-classification Stream (NEVIS'22), a benchmark consisting of a stream of over 100 visual classification tasks, sorted chronologically and extracted from papers sampled uniformly from computer vision proceedings spanning the last three decades. The resulting stream reflects what the research community thought was meaningful at any point in time, and it serves as an ideal test bed to assess how well models can adapt to new tasks, and do so better and more efficiently as time goes by. Despite being limited to classification, the resulting stream has a rich diversity of tasks from OCR, to texture analysis, scene recognition, and so forth. The diversity is also reflected in the wide range of dataset sizes, spanning over four orders of magnitude. Overall, NEVIS'22 poses an unprecedented challenge for current sequential learning approaches due to the scale and diversity of tasks, yet with a low entry barrier as it is limited to a single modality and well understood supervised learning problems. Moreover, we provide a reference implementation including strong baselines and an evaluation protocol to compare methods in terms of their trade-off between accuracy and compute.

Autonomous surface vehicles (ASV) allow the investigation of coastal areas, ports and harbors as well as harsh and dangerous environments such as the arctic regions. Despite receiving increasing attention, the hydrodynamic analysis of ASV performance subject to variable operational parameters is little investigated. In this context, this paper presents a multi-fidelity (MF) hydrodynamic analysis of an ASV, namely the Shallow Water Autonomous Multipurpose Platform (SWAMP), at surveying speed in calm water and subject to variable payload and location of the center of mass, accounting for the variety of equipment that the vehicle can carry. The analysis is conducted in deep water, which is the condition mostly encountered by the ASV during surveys of coastal and harbors areas. Quantities of interest are the resistance, the vehicle attitude, and the wave generated in the region between the catamaran hulls. These are assessed using a Reynolds Averaged Navier Stokes Equation (RANSE) code and a linear potential flow (PF) solver. The objective is to accurately assess the quantities of interest, along with identifying the limitation of PF analysis in the current context. Finally, a multi-fidelity Gaussian Process (MF-GP) model is obtained combining RANSE and PF solutions. The latter also include variable grid refinement and coupling between hydrodynamic loads and rigid body equations of motion. The surrogate model is iteratively refined using an active learning approach. Numerical results show that the MF-GP is effective in producing response surfaces of the SWAMP performance with a limited computational cost. It is highlighted how the SWAMP performance is significantly affected not only by the payload, but also by the location of the center of mass. The latter can be therefore properly calibrated to minimize the resistance and allow for longer-range operations.

Continual learning (CL) enables the development of models and agents that learn from a sequence of tasks while addressing the limitations of standard deep learning approaches, such as catastrophic forgetting. In this work, we investigate the factors that contribute to the performance differences between task-agnostic CL and multi-task (MTL) agents. We pose two hypotheses: (1) task-agnostic methods might provide advantages in settings with limited data, computation, or high dimensionality, and (2) faster adaptation may be particularly beneficial in continual learning settings, helping to mitigate the effects of catastrophic forgetting. To investigate these hypotheses, we introduce a replay-based recurrent reinforcement learning (3RL) methodology for task-agnostic CL agents. We assess 3RL on a synthetic task and the Meta-World benchmark, which includes 50 unique manipulation tasks. Our results demonstrate that 3RL outperforms baseline methods and can even surpass its multi-task equivalent in challenging settings with high dimensionality. We also show that the recurrent task-agnostic agent consistently outperforms or matches the performance of its transformer-based counterpart. These findings provide insights into the advantages of task-agnostic CL over task-aware MTL approaches and highlight the potential of task-agnostic methods in resource-constrained, high-dimensional, and multi-task environments.

Modularity is a compelling solution to continual learning (CL), the problem of modeling sequences of related tasks. Learning and then composing modules to solve different tasks provides an abstraction to address the principal challenges of CL including catastrophic forgetting, backward and forward transfer across tasks, and sub-linear model growth. We introduce local module composition (LMC), an approach to modular CL where each module is provided a local structural component that estimates a module's relevance to the input. Dynamic module composition is performed layer-wise based on local relevance scores. We demonstrate that agnosticity to task identities (IDs) arises from (local) structural learning that is module-specific as opposed to the task- and/or model-specific as in previous works, making LMC applicable to more CL settings compared to previous works. In addition, LMC also tracks statistics about the input distribution and adds new modules when outlier samples are detected. In the first set of experiments, LMC performs favorably compared to existing methods on the recent Continual Transfer-learning Benchmark without requiring task identities. In another study, we show that the locality of structural learning allows LMC to interpolate to related but unseen tasks (OOD), as well as to compose modular networks trained independently on different task sequences into a third modular network without any fine-tuning. Finally, in search for limitations of LMC we study it on more challenging sequences of 30 and 100 tasks, demonstrating that local module selection becomes much more challenging in presence of a large number of candidate modules. In this setting best performing LMC spawns much fewer modules compared to an oracle based baseline, however, it reaches a lower overall accuracy. The codebase is available under https://github.com/oleksost/LMC.

Finding neural network weights that generalize well from small datasets is difficult. A promising approach is to learn a weight initialization such that a small number of weight changes results in low generalization error. We show that this form of meta-learning can be improved by letting the learning algorithm decide which weights to change, i.e., by learning where to learn. We find that patterned sparsity emerges from this process, with the pattern of sparsity varying on a problem-by-problem basis. This selective sparsity results in better generalization and less interference in a range of few-shot and continual learning problems. Moreover, we find that sparse learning also emerges in a more expressive model where learning rates are meta-learned. Our results shed light on an ongoing debate on whether meta-learning can discover adaptable features and suggest that learning by sparse gradient descent is a powerful inductive bias for meta-learning systems., Comment: Published at NeurIPS 2021

The field of Continual Learning (CL) seeks to develop algorithms that accumulate knowledge and skills over time through interaction with non-stationary environments. In practice, a plethora of evaluation procedures (settings) and algorithmic solutions (methods) exist, each with their own potentially disjoint set of assumptions. This variety makes measuring progress in CL difficult. We propose a taxonomy of settings, where each setting is described as a set of assumptions. A tree-shaped hierarchy emerges from this view, where more general settings become the parents of those with more restrictive assumptions. This makes it possible to use inheritance to share and reuse research, as developing a method for a given setting also makes it directly applicable onto any of its children. We instantiate this idea as a publicly available software framework called Sequoia, which features a wide variety of settings from both the Continual Supervised Learning (CSL) and Continual Reinforcement Learning (CRL) domains. Sequoia also includes a growing suite of methods which are easy to extend and customize, in addition to more specialized methods from external libraries. We hope that this new paradigm and its first implementation can help unify and accelerate research in CL. You can help us grow the tree by visiting www.github.com/lebrice/Sequoia.

Classical machine learning algorithms often assume that the data are drawn i.i.d. from a stationary probability distribution. Recently, continual learning emerged as a rapidly growing area of machine learning where this assumption is relaxed, i.e. where the data distribution is non-stationary and changes over time. This paper represents the state of data distribution by a context variable $c$. A drift in $c$ leads to a data distribution drift. A context drift may change the target distribution, the input distribution, or both. Moreover, distribution drifts might be abrupt or gradual. In continual learning, context drifts may interfere with the learning process and erase previously learned knowledge; thus, continual learning algorithms must include specialized mechanisms to deal with such drifts. In this paper, we aim to identify and categorize different types of context drifts and potential assumptions about them, to better characterize various continual-learning scenarios. Moreover, we propose to use the distribution drift framework to provide more precise definitions of several terms commonly used in the continual learning field.

Explainability for machine learning models has gained considerable attention within the research community given the importance of deploying more reliable machine-learning systems. In computer vision applications, generative counterfactual methods indicate how to perturb a model's input to change its prediction, providing details about the model's decision-making. Current methods tend to generate trivial counterfactuals about a model's decisions, as they often suggest to exaggerate or remove the presence of the attribute being classified. For the machine learning practitioner, these types of counterfactuals offer little value, since they provide no new information about undesired model or data biases. In this work, we identify the problem of trivial counterfactual generation and we propose DiVE to alleviate it. DiVE learns a perturbation in a disentangled latent space that is constrained using a diversity-enforcing loss to uncover multiple valuable explanations about the model's prediction. Further, we introduce a mechanism to prevent the model from producing trivial explanations. Experiments on CelebA and Synbols demonstrate that our model improves the success rate of producing high-quality valuable explanations when compared to previous state-of-the-art methods. Code is available at https://github.com/ElementAI/beyond-trivial-explanations., Comment: ICCV 2021

In the last few years, we have witnessed a renewed and fast-growing interest in continual learning with deep neural networks with the shared objective of making current AI systems more adaptive, efficient and autonomous. However, despite the significant and undoubted progress of the field in addressing the issue of catastrophic forgetting, benchmarking different continual learning approaches is a difficult task by itself. In fact, given the proliferation of different settings, training and evaluation protocols, metrics and nomenclature, it is often tricky to properly characterize a continual learning algorithm, relate it to other solutions and gauge its real-world applicability. The first Continual Learning in Computer Vision challenge held at CVPR in 2020 has been one of the first opportunities to evaluate different continual learning algorithms on a common hardware with a large set of shared evaluation metrics and 3 different settings based on the realistic CORe50 video benchmark. In this paper, we report the main results of the competition, which counted more than 79 teams registered, 11 finalists and 2300$ in prizes. We also summarize the winning approaches, current challenges and future research directions., Comment: Pre-print v1: 12 pages, 3 figures, 8 tables

Progress in the field of machine learning has been fueled by the introduction of benchmark datasets pushing the limits of existing algorithms. Enabling the design of datasets to test specific properties and failure modes of learning algorithms is thus a problem of high interest, as it has a direct impact on innovation in the field. In this sense, we introduce Synbols -- Synthetic Symbols -- a tool for rapidly generating new datasets with a rich composition of latent features rendered in low resolution images. Synbols leverages the large amount of symbols available in the Unicode standard and the wide range of artistic font provided by the open font community. Our tool's high-level interface provides a language for rapidly generating new distributions on the latent features, including various types of textures and occlusions. To showcase the versatility of Synbols, we use it to dissect the limitations and flaws in standard learning algorithms in various learning setups including supervised learning, active learning, out of distribution generalization, unsupervised representation learning, and object counting.

Continual learning studies agents that learn from streams of tasks without forgetting previous ones while adapting to new ones. Two recent continual-learning scenarios have opened new avenues of research. In meta-continual learning, the model is pre-trained to minimize catastrophic forgetting of previous tasks. In continual-meta learning, the aim is to train agents for faster remembering of previous tasks through adaptation. In their original formulations, both methods have limitations. We stand on their shoulders to propose a more general scenario, OSAKA, where an agent must quickly solve new (out-of-distribution) tasks, while also requiring fast remembering. We show that current continual learning, meta-learning, meta-continual learning, and continual-meta learning techniques fail in this new scenario. We propose Continual-MAML, an online extension of the popular MAML algorithm as a strong baseline for this scenario. We empirically show that Continual-MAML is better suited to the new scenario than the aforementioned methodologies, as well as standard continual learning and meta-learning approaches.

We introduce and study the problem of Online Continual Compression, where one attempts to simultaneously learn to compress and store a representative dataset from a non i.i.d data stream, while only observing each sample once. A naive application of auto-encoders in this setting encounters a major challenge: representations derived from earlier encoder states must be usable by later decoder states. We show how to use discrete auto-encoders to effectively address this challenge and introduce Adaptive Quantization Modules (AQM) to control variation in the compression ability of the module at any given stage of learning. This enables selecting an appropriate compression for incoming samples, while taking into account overall memory constraints and current progress of the learned compression. Unlike previous methods, our approach does not require any pretraining, even on challenging datasets. We show that using AQM to replace standard episodic memory in continual learning settings leads to significant gains on continual learning benchmarks. Furthermore we demonstrate this approach with larger images, LiDAR, and reinforcement learning environments.

Continual learning, the setting where a learning agent is faced with a never ending stream of data, continues to be a great challenge for modern machine learning systems. In particular the online or "single-pass through the data" setting has gained attention recently as a natural setting that is difficult to tackle. Methods based on replay, either generative or from a stored memory, have been shown to be effective approaches for continual learning, matching or exceeding the state of the art in a number of standard benchmarks. These approaches typically rely on randomly selecting samples from the replay memory or from a generative model, which is suboptimal. In this work, we consider a controlled sampling of memories for replay. We retrieve the samples which are most interfered, i.e. whose prediction will be most negatively impacted by the foreseen parameters update. We show a formulation for this sampling criterion in both the generative replay and the experience replay setting, producing consistent gains in performance and greatly reduced forgetting. We release an implementation of our method at https://github.com/optimass/Maximally_Interfered_Retrieval.

A large, world-wide community of physicists is working to realise an exceptional physics program of energy-frontier, electron-positron collisions with the International Linear Collider (ILC). This program will begin with a central focus on high-precision and model-independent measurements of the Higgs boson couplings. This method of searching for new physics beyond the Standard Model is orthogonal to and complements the LHC physics program. The ILC at 250 GeV will also search for direct new physics in exotic Higgs decays and in pair-production of weakly interacting particles. Polarised electron and positron beams add unique opportunities to the physics reach. The ILC can be upgraded to higher energy, enabling precision studies of the top quark and measurement of the top Yukawa coupling and the Higgs self-coupling. The key accelerator technology, superconducting radio-frequency cavities, has matured. Optimised collider and detector designs, and associated physics analyses, were presented in the ILC Technical Design Report, signed by 2400 scientists. There is a strong interest in Japan to host this international effort. A detailed review of the many aspects of the project is nearing a conclusion in Japan. Now the Japanese government is preparing for a decision on the next phase of international negotiations, that could lead to a project start within a few years. The potential timeline of the ILC project includes an initial phase of about 4 years to obtain international agreements, complete engineering design and prepare construction, and form the requisite international collaboration, followed by a construction phase of 9 years.

The International Linear Collider (ILC) being proposed in Japan is an electron-positron linear collider with an initial energy of 250 GeV. The ILC accelerator is based on the technology of superconducting radio-frequency cavities. This technology has reached a mature stage in the European XFEL project and is now widely used. The ILC will start by measuring the Higgs properties, providing high-precision and model-independent determinations of its parameters. The ILC at 250 GeV will also search for direct new physics in exotic Higgs decays and in pair-production of weakly interacting particles. The use of polarised electron and positron beams opens new capabilities and scenarios that add to the physics reach. The ILC can be upgraded to higher energy, enabling precision studies of the top quark and measurement of the top Yukawa coupling and the Higgs self-coupling. The international -- including European -- interest for the project is very strong. Europe has participated in the ILC project since its early conception and plays a major role in its present development covering most of its scientific and technological aspects: physics studies, accelerator and detectors. The potential for a wide participation of European groups and laboratories is thus high, including important opportunities for European industry. Following decades of technical development, R&D, and design optimisation, the project is ready for construction and the European particle physics community, technological centers and industry are prepared to participate in this challenging endeavour.

Silicon Photomultipliers are potentially ideal detectors for Quantum Optics and Quantum Information studies based on mesoscopic states of light. However, their non-idealities hampered their use so far. An optimal mode of operation has been developed and it is presented here, proving that this class of sensors can actually be exploited for the characterization of both classical and quantum properties of light., Comment: 15 pages, 14 figures

Detector stochastic deviations from an ideal response can hamper the measurement of quantum properties of light especially in the mesoscopic regime where photon-number resolution is required. We demonstrate that, by a proper analysis of the output signal, nonclassicality of twin-beam states can be detected and exploited with commercial and cost effective silicon-based photon-number-resolving detectors.

Generating high-quality text with sufficient diversity is essential for a wide range of Natural Language Generation (NLG) tasks. Maximum-Likelihood (MLE) models trained with teacher forcing have consistently been reported as weak baselines, where poor performance is attributed to exposure bias (Bengio et al., 2015; Ranzato et al., 2015); at inference time, the model is fed its own prediction instead of a ground-truth token, which can lead to accumulating errors and poor samples. This line of reasoning has led to an outbreak of adversarial based approaches for NLG, on the account that GANs do not suffer from exposure bias. In this work, we make several surprising observations which contradict common beliefs. First, we revisit the canonical evaluation framework for NLG, and point out fundamental flaws with quality-only evaluation: we show that one can outperform such metrics using a simple, well-known temperature parameter to artificially reduce the entropy of the model's conditional distributions. Second, we leverage the control over the quality / diversity trade-off given by this parameter to evaluate models over the whole quality-diversity spectrum and find MLE models constantly outperform the proposed GAN variants over the whole quality-diversity space. Our results have several implications: 1) The impact of exposure bias on sample quality is less severe than previously thought, 2) temperature tuning provides a better quality / diversity trade-off than adversarial training while being easier to train, easier to cross-validate, and less computationally expensive. Code to reproduce the experiments is available at github.com/pclucas14/GansFallingShort

In this paper we solve the discrete time mean-variance hedging problem when asset returns follow a multivariate autoregressive hidden Markov model. Time dependent volatility and serial dependence are well established properties of financial time series and our model covers both. To illustrate the relevance of our proposed methodology, we first compare the proposed model with the well-known hidden Markov model via likelihood ratio tests and a novel goodness-of-fit test on the S\&P 500 daily returns. Secondly, we present out-of-sample hedging results on S\&P 500 vanilla options as well as a trading strategy based on theoretical prices, which we compare to simpler models including the classical Black-Scholes delta-hedging approach.

The definition of Good Experimental Methodologies (GEMs) in robotics is a topic of widespread interest due also to the increasing employment of robots in everyday civilian life. The present work contributes to the ongoing discussion on GEMs for Unmanned Surface Vehicles (USVs). It focuses on the definition of GEMs and provides specific guidelines for path-following experiments. Statistically designed experiments (DoE) offer a valid basis for developing an empirical model of the system being investigated. A two-step adaptive experimental procedure for evaluating path-following performance and based on DoE, is tested on the simulator of the Charlie USV. The paper argues the necessity of performing extensive simulations prior to the execution of field trials.

The after-pulsing probability in Silicon Photomulti- pliers and its time constant are obtained measuring the mean number of photo-electrons in a variable time window following a light pulse. The method, experimentally simple and statistically robust due to the use of the Central Limit Theorem, has been applied to an HAMAMATSU MPPC S10362-11-100C.

Measurements of the power spectrum of the seeing in the range 0.001-1 Hz have been performed in order to understand the criticity of the transits' method for solar diameter monitoring., Comment: 3 pages, 3 figures, proc. of the Fourth French-Chinese meeting on Solar Physics Understanding Solar Activity: Advances and Challenges, 15 - 18 November, 2011 Nice, France

We present a description of the operation of a multi-pixel detector in the presence of non-negligible dark-count and cross-talk effects. We apply the model to devise self-consistent calibration strategies to be performed on the very light under investigation.

Hybridization of pixel detector systems has to satisfy tight requirements: high yield, long term reliability, mechanical stability, thermal compliance and robustness have to go together with low passive mass added to the system, radiation hardness, flexibility in the technology end eventually low cost. The current technologies for the interconnection of the front-end chips and the sensor are reviewed and compared, together with the solutions for the interface to the far-end electronics., Comment: 7 pages, no figure, presented at PIXEL2000, International Workshop on semiconductor pixel detectors for particles and X rays, 5-8 June 2000, Genova, Italy. Proceedings to be published in Nuclear Instruments and Methods in Physics Research

The e+e- linear collider physics programme sets highly demanding requirements on the accurate determination of charged particle trajectories close to their production point. A new generation of Vertex Trackers, based on different technologies of high resolution silicon sensors, is being developed to provide the needed performances. These developments are based on the experience with the LEP/SLC vertex detectors and on the results of the R&D programs for the LHC trackers and also define a further program of R&D specific to the linear collider applications. In this paper the present status of the conceptual tracker design, silicon detector R&D and physics studies is discussed., Comment: 17 pages, 15 figures, Invited Talk at the Workshop on the development of future linear electron-positron colliders for particle physics studies and for research using free electon lasers; Lund 23-26 September 1999, to appear in the Proceedings