Search

What computational structures emerge in transformers trained on next-token prediction? In this work, we provide evidence that transformers implement constrained Bayesian belief updating -- a parallelized version of partial Bayesian inference shaped by architectural constraints. To do this, we integrate the model-agnostic theory of optimal prediction with mechanistic interpretability to analyze transformers trained on a tractable family of hidden Markov models that generate rich geometric patterns in neural activations. We find that attention heads carry out an algorithm with a natural interpretation in the probability simplex, and create representations with distinctive geometric structure. We show how both the algorithmic behavior and the underlying geometry of these representations can be theoretically predicted in detail -- including the attention pattern, OV-vectors, and embedding vectors -- by modifying the equations for optimal future token predictions to account for the architectural constraints of attention. Our approach provides a principled lens on how gradient descent resolves the tension between optimal prediction and architectural design.

Exploiting quantum effects of mechanical motion, such as backaction evading measurements or squeezing, requires preparation of the oscillator in a high-purity state. The largest state purities in optomechanics to date have relied on cryogenic cooling, combined with coupling to electromagnetic resonators driven with a coherent radiation field. In this work, we cool the mega-hertz-frequency librational mode of an optically levitated silica nanoparticle from room temperature to its quantum ground state. Cooling is realized by coherent scattering into a Fabry-Perot cavity. We use sideband thermometry to infer a phonon population of 0.04 quanta under optimal conditions, corresponding to a state purity of 92%. The purity reached by our room-temperature experiment exceeds the performance offered by mechanically clamped oscillators in a cryogenic environment. Our work establishes a platform for high-purity quantum optomechanics at room temperature.

Carbon dioxide levels below the soil surface are an important measurement relating to plant health, especially for plants such as perennial grasses in northern climates where ice encasement can occur over winter. In such cases, the CO$_2$ levels can build up and become toxic. This is likely a significant contributor to turfgrass death over winter; however, there is an insufficient amount of data regarding this phenomenon in large part due to the lack of effective sensors. Many off the shelf CO$_2$ sensors exist, but they are not sufficiently hardened for in ground deployment over winter. As a result, the only options currently available are very costly automated gas samplers or manual sampling at intervals with laboratory testing -- a process that results in a limited number of data points and is labor intensive. To combat this problem we have taken an established NDIR CO$_2$ sensor and hardened it for use in winter and ice encased environments to allow for continuous automated sampling of subsurface CO$_2$ levels to better understand ice encasement damage in perennial grass systems.

This is an index of the contributions by the Giant Radio Array for Neutrino Detection (GRAND) Collaboration to the 10th International Workshop on Acoustic and Radio EeV Neutrino Detection Activities (ARENA 2024, University of Chicago, June 11-14, 2024). The contributions include an overview of GRAND in its present and future incarnations, methods of radio-detection that are being developed for them, and ongoing joint work between the GRAND and BEACON experiments., Comment: Note: To access the list of contributions, please follow the "HTML" link that can be found on the arXiv page

The operation of upcoming ultra-high-energy cosmic-ray, gamma-ray, and neutrino radio-detection experiments, like the Giant Radio Array for Neutrino Detection (GRAND), poses significant computational challenges involving the production of numerous simulations of particle showers and their detection, and a high data throughput. GRANDlib is an open-source software tool designed to meet these challenges. Its primary goal is to perform end-to-end simulations of the detector operation, from the interaction of ultra-high-energy particles, through -- by interfacing with external air-shower simulations -- the ensuing particle shower development and its radio emission, to its detection by antenna arrays and its processing by data-acquisition systems. Additionally, GRANDlib manages the visualization, storage, and retrieval of experimental and simulated data. We present an overview of GRANDlib to serve as the basis of future GRAND analyses., Comment: 11 pages, 9 figures, plus appendices; Matches published version

Reduced-rank approach has been used for decades in robust linear estimation of both deterministic and random vector of parameters in linear model y=Hx+\sqrt{epsilon}n. In practical settings, estimation is frequently performed under incomplete or inexact model knowledge, which in the stochastic case significantly increases mean-square-error (MSE) of an estimate obtained by the linear minimum mean-square-error (MMSE) estimator, which is MSE-optimal among linear estimators in the theoretical case of perfect model knowledge. However, the improved performance of reduced-rank estimators over MMSE estimator in estimation under incomplete or inexact model knowledge has been established to date only by means of numerical simulations and arguments indicating that the reduced-rank approach may provide improved performance over MMSE estimator in certain settings. In this paper we focus on the high signal-to-noise ratio (SNR) case, which has not been previously considered as a natural area of application of reduced-rank estimators. We first show explicit sufficient conditions under which familiar reduced-rank MMSE and truncated SVD estimators achieve lower MSE than MMSE estimator if singular values of array response matrix H are perturbed. We then extend these results to the case of a generic perturbation of array response matrix H, and demonstrate why MMSE estimator frequently attains higher MSE than reduced-rank MMSE and truncated SVD estimators if H is ill-conditioned. The main results of this paper are verified in numerical simulations.

The Extreme Universe Space Observatory on a Super Pressure Balloon 2 (EUSO-SPB2) flew on May 13$^{\text{th}}$ and 14$^{\text{th}}$ of 2023. Consisting of two novel optical telescopes, the payload utilized next-generation instrumentation for the observations of extensive air showers from near space. One instrument, the fluorescence telescope (FT) searched for Ultra-High Energy Cosmic Rays (UHECRs) by recording the atmosphere below the balloon in the near-UV with a 1~$\mu$s time resolution using 108 multi-anode photomultiplier tubes with a total of 6,912 channels. Validated by pre-flight measurements during a field campaign, the energy threshold was estimated around 2~EeV with an expected event rate of approximately 1 event per 10 hours of observation. Based on the limited time afloat, the expected number of UHECR observations throughout the flight is between 0 and 2. Consistent with this expectation, no UHECR candidate events have been found. The majority of events appear to be detector artifacts that were not rejected properly due to a shortened commissioning phase. Despite the earlier-than-expected termination of the flight, data were recorded which provide insights into the detectors stability in the near-space environment as well as the diffuse ultraviolet emissivity of the atmosphere, both of which are impactful to future experiments.

The Seidel matrix of a tournament on $n$ players is an $n\times n$ skew-symmetric matrix with entries in $\{0, 1, -1\}$ that encapsulates the outcomes of the games in the given tournament. It is known that the determinant of an $n\times n$ Seidel matrix is $0$ if $n$ is odd, and is an odd perfect square if $n$ is even. This leads to the study of the set \[ \mathcal{D}(n)= \{ \sqrt{\det S}: \mbox{ $S$ is an $n\times n$ Seidel matrix}\}. \] This paper studies various questions about $\mathcal{D}(n)$. It is shown that $\mathcal{D}(n)$ is a proper subset of $\mathcal{D}(n+2)$ for every positive even integer, and every odd integer in the interval $[1, 1+n^2/2]$ is in $\mathcal{D}(n)$ for $n$ even. The expected value and variance of $\det S$ over the $n\times n$ Seidel matrices chosen uniformly at random is determined, and upper bounds on $\max \mathcal{D}(n)$ are given, and related to the Hadamard conjecture. Finally, it is shown that for infinitely many $n$, $\mathcal{D}(n)$ contains a gap (that is, there are odd integers $k<\ell

Text embeddings are essential for many tasks, such as document retrieval, clustering, and semantic similarity assessment. In this paper, we study how to contrastively train text embedding models in a compute-optimal fashion, given a suite of pre-trained decoder-only language models. Our innovation is an algorithm that produces optimal configurations of model sizes, data quantities, and fine-tuning methods for text-embedding models at different computational budget levels. The resulting recipe, which we obtain through extensive experiments, can be used by practitioners to make informed design choices for their embedding models. Specifically, our findings suggest that full fine-tuning and low-rank adaptation fine-tuning produce optimal models at lower and higher computational budgets respectively., Comment: NeurIPS 2024

With the increasing emphasis on machine learning and artificial intelligence to drive knowledge discovery in the agricultural sciences, spatial internet of things (IoT) technologies have become increasingly important for collecting real-time, high resolution data for these models. However, managing large fleets of devices while maintaining high data quality remains an ongoing challenge as scientists iterate from prototype to mature end-to-end applications. Here, we provide a set of case studies using the framework of technology readiness levels for an open source spatial IoT system. The spatial IoT systems underwent 3 major and 14 minor system versions, had over 2,727 devices manufactured both in academic and commercial contexts, and are either in active or planned deployment across four continents. Our results show the evolution of a generalizable, open source spatial IoT system designed for agricultural scientists, and provide a model for academic researchers to overcome the challenges that exist in going from one-off prototypes to thousands of internet-connected devices., Comment: 20 pages, 5 figures, 1 table

Real-world applications of AI Planning often require a highly expressive modeling language to accurately capture important intricacies of target systems. Hybrid systems are ubiquitous in the real-world, and PDDL+ is the standardized modeling language for capturing such systems as planning domains. PDDL+ enables accurate encoding of mixed discrete-continuous system dynamics, exogenous activity, and many other interesting features exhibited in realistic scenarios. However, the uptake in usage of PDDL+ has been slow and apprehensive, largely due to a general shortage of PDDL+ planning software, and rigid limitations of the few existing planners. To overcome this chasm, we present Nyx, a novel PDDL+ planner built to emphasize lightness, simplicity, and, most importantly, adaptability. The planner is designed to be effortlessly customizable to expand its capabilities well beyond the scope of PDDL+. As a result, Nyx can be tailored to virtually any potential real-world application requiring some form of AI Planning, paving the way for wider adoption of planning methods for solving real-world problems.

Deep equilibrium (DEQ) models are widely recognized as a memory efficient alternative to standard neural networks, achieving state-of-the-art performance in language modeling and computer vision tasks. These models solve a fixed point equation instead of explicitly computing the output, which sets them apart from standard neural networks. However, existing DEQ models often lack formal guarantees of the existence and uniqueness of the fixed point, and the convergence of the numerical scheme used for computing the fixed point is not formally established. As a result, DEQ models are potentially unstable in practice. To address these drawbacks, we introduce a novel class of DEQ models called positive concave deep equilibrium (pcDEQ) models. Our approach, which is based on nonlinear Perron-Frobenius theory, enforces nonnegative weights and activation functions that are concave on the positive orthant. By imposing these constraints, we can easily ensure the existence and uniqueness of the fixed point without relying on additional complex assumptions commonly found in the DEQ literature, such as those based on monotone operator theory in convex analysis. Furthermore, the fixed point can be computed with the standard fixed point algorithm, and we provide theoretical guarantees of its geometric convergence, which, in particular, simplifies the training process. Experiments demonstrate the competitiveness of our pcDEQ models against other implicit models.

The Extreme Universe Space Observatory on a Super Pressure Balloon 1 (EUSO-SPB1) was launched in 2017 April from Wanaka, New Zealand. The plan of this mission of opportunity on a NASA super pressure balloon test flight was to circle the southern hemisphere. The primary scientific goal was to make the first observations of ultra-high-energy cosmic-ray extensive air showers (EASs) by looking down on the atmosphere with an ultraviolet (UV) fluorescence telescope from suborbital altitude (33~km). After 12~days and 4~hours aloft, the flight was terminated prematurely in the Pacific Ocean. Before the flight, the instrument was tested extensively in the West Desert of Utah, USA, with UV point sources and lasers. The test results indicated that the instrument had sensitivity to EASs of approximately 3 EeV. Simulations of the telescope system, telescope on time, and realized flight trajectory predicted an observation of about 1 event assuming clear sky conditions. The effects of high clouds were estimated to reduce this value by approximately a factor of 2. A manual search and a machine-learning-based search did not find any EAS signals in these data. Here we review the EUSO-SPB1 instrument and flight and the EAS search., Comment: 18 pages, 19 figures

This is a collection of papers presented by the JEM-EUSO Collaboration at the 38th International Cosmic Ray Conference (Nagoya, Japan, July 26-August 3, 2023)

X (formerly Twitter) has evolved into a contemporary agora, offering a platform for individuals to express opinions and viewpoints on current events. The majority of the topics discussed on Twitter are directly related to ongoing events, making it an important source for monitoring public discourse. However, linking tweets to specific news presents a significant challenge due to their concise and informal nature. Previous approaches, including topic models, graph-based models, and supervised classifiers, have fallen short in effectively capturing the unique characteristics of tweets and articles. Inspired by the success of the CLIP model in computer vision, which employs contrastive learning to model similarities between images and captions, this paper introduces a contrastive learning approach for training a representation space where linked articles and tweets exhibit proximity. We present our contrastive learning approach, CATBERT (Contrastive Articles Tweets BERT), leveraging pre-trained BERT models. The model is trained and tested on a dataset containing manually labeled English and Polish tweets and articles related to the Russian-Ukrainian war. We evaluate CATBERT's performance against traditional approaches like LDA, and the novel method based on OpenAI embeddings, which has not been previously applied to this task. Our findings indicate that CATBERT demonstrates superior performance in associating tweets with relevant news articles. Furthermore, we demonstrate the performance of the models when applied to finding the main topic -- represented by an article -- of the whole cascade of tweets. In this new task, we report the performance of the different models in dependence on the cascade size.

In this study, we explore the use of Large Language Models (LLMs) to counteract hate speech. We conducted the first real-life A/B test assessing the effectiveness of LLM-generated counter-speech. During the experiment, we posted 753 automatically generated responses aimed at reducing user engagement under tweets that contained hate speech toward Ukrainian refugees in Poland. Our work shows that interventions with LLM-generated responses significantly decrease user engagement, particularly for original tweets with at least ten views, reducing it by over 20%. This paper outlines the design of our automatic moderation system, proposes a simple metric for measuring user engagement and details the methodology of conducting such an experiment. We discuss the ethical considerations and challenges in deploying generative AI for discourse moderation.

JEM--EUSO is an international program for the development of space-based Ultra-High Energy Cosmic Ray observatories. The program consists of a series of missions which are either under development or in the data analysis phase. All instruments are based on a wide-field-of-view telescope, which operates in the near-UV range, designed to detect the fluorescence light emitted by extensive air showers in the atmosphere. We describe the simulation software ESAFin the framework of the JEM--EUSO program and explain the physical assumptions used. We present here the implementation of the JEM--EUSO, POEMMA, K--EUSO, TUS, Mini--EUSO, EUSO--SPB1 and EUSO--TA configurations in ESAF. For the first time ESAF simulation outputs are compared with experimental data.

The second generation Extreme Universe Space Observatory on a Super-Pressure Balloon (EUSO-SPB2) mission is a stratospheric balloon mission developed within the Joint Exploratory Missions for Extreme Universe Space Observatory (JEM-EUSO) program. The Fluorescence Telescope (FT) is one of the two separate Schmidt telescopes of EUSO-SPB2, which aims at measuring the fluorescence emission of extensive air showers from cosmic rays above the energy of 1 EeV, looking downwards onto the atmosphere from the float altitude of 33 km. The FT measures photons with a time resolution of 1.05 $\mu$s in two different modes: single photon counting (PC) and charge integration (KI). In this paper, we describe the latter and report on the measurements of its characteristics. We also present a new trigger based on this channel, the so-called KI trigger, which allows to measure additional types of events, namely very short and intense light pulses. We report on the tests of this trigger mode in the laboratory and at the TurLab facility, and its implementation in the EUSO-SPB2 mission.

Mini-EUSO is a wide Field-of-View (FoV, 44$^{\circ}$) telescope currently in operation from a nadia-facing UV-transparent window in the Russian Zvezda module on the International Space Station (ISS). It is the first detector of the JEM-EUSO program deployed on the ISS, launched in August 2019. The main goal of Mini-EUSO is to measure the UV emissions from the ground and atmosphere, using an orbital platform. Mini-EUSO is mainly sensitive in the 290-430 nm bandwidth. Light is focused by a system of two Fresnel lenses of 25 cm diameter each on the Photo- Detector-Module (PDM), which consists of an array of 36 Multi-Anode Photomultiplier Tubes (MAPMTs), for a total of 2304 pixels working in photon counting mode, in three different time resolutions of 2.5 ${\mu}$s, 320 ${\mu}$s, 40.96 ms operation in parallel. In the longest time scale, the data is continuously acquired to monitor the UV emission of the Earth. It is best suited for the observation of ground sources and therefore has been used for the observational campaigns of the Mini-EUSO. In this contribution, we present the assembled UV flasher, the operation of the field campaign and the analysis of the obtained data. The result is compared with the overall efficiency computed from the expectations which takes into account the atmospheric attenuation and the parameterization of different effects such as the optics efficiency, the MAPMT detection efficiency, BG3 filter transmittance and the transparency of the ISS window.

Mini-EUSO is a small, near-UV telescope observing the Earth and its atmosphere from the International Space Station. The time resolution of 2.5 microseconds and the instantaneous ground coverage of about $320\times 320$ km$^2$ allows it to detect some Transient Luminous Events, including Elves. Elves, with their almost circular shape and a radius expanding in time form cone-like structures in space-time, which are usually easy to be recognised by the eye, but not simple to filter out from the myriad of other events, many of them not yet categorised. In this work, we present a fast and efficient approach for detecting Elves in the data using a 3D CNN-based one-class classifier., Comment: ICRC 2023 Proceedings

In this work, we introduce a framework for cross-lingual speech synthesis, which involves an upstream Voice Conversion (VC) model and a downstream Text-To-Speech (TTS) model. The proposed framework consists of 4 stages. In the first two stages, we use a VC model to convert utterances in the target locale to the voice of the target speaker. In the third stage, the converted data is combined with the linguistic features and durations from recordings in the target language, which are then used to train a single-speaker acoustic model. Finally, the last stage entails the training of a locale-independent vocoder. Our evaluations show that the proposed paradigm outperforms state-of-the-art approaches which are based on training a large multilingual TTS model. In addition, our experiments demonstrate the robustness of our approach with different model architectures, languages, speakers and amounts of data. Moreover, our solution is especially beneficial in low-resource settings., Comment: Accepted at ICONIP 2023

The ability to engineer cavity-mediated interactions has emerged as a powerful tool for the generation of non-local correlations and the investigation of non-equilibrium phenomena in many-body systems. Levitated optomechanical systems have recently entered the multi-particle regime, with promise for using arrays of massive strongly coupled oscillators for exploring complex interacting systems and sensing. Here, by combining advances in multi-particle optical levitation and cavity-based quantum control, we demonstrate, for the first time, programmable cavity-mediated interactions between nanoparticles in vacuum. The interaction is mediated by photons scattered by spatially separated particles in a cavity, resulting in strong coupling ($G_\text{zz}/\Omega_\text{z} = 0.238\pm0.005$) that does not decay with distance within the cavity mode volume. We investigate the scaling of the interaction strength with cavity detuning and inter-particle separation, and demonstrate the tunability of interactions between different mechanical modes. Our work paves the way towards exploring many-body effects in nanoparticle arrays with programmable cavity-mediated interactions, generating entanglement of motion, and using interacting particle arrays for optomechanical sensing., Comment: 7 pages, 4 figures, 4 pages in Supplementary, 3 figures in Supplementary

Signal processing based on stimulated Brillouin scattering (SBS) is limited by the narrow linewidth of the optoacoustic response, which confines many Brillouin applications to continuous wave signals or optical pulses longer than several nanoseconds. In this work, we experimentally demonstrate Brillouin interactions at the 150 ps time scale and a delay for a record 15 ns which corresponds to a delay of 100 pulse widths. This breakthrough experimental result was enabled by the high local gain of the chalcogenide waveguides as the optoacoustic interaction length reduces with pulse width. We successfully transfer 150ps-long pulses to traveling acoustic waves within a Brillouin-based memory setup. The information encoded in the optical pulses is stored for 15 ns in the acoustic field. We show the retrieval of eight amplitude levels, multiple consecutive pulses and low distortion in pulse shape. The extension of Brillouin-based storage to the ultra-short pulse regime is an important step for the realisation of practical Brillouin-based delay lines and other optical processing applications., Comment: 6 pages, 6 figures

The Giant Radio Array for Neutrino Detection (GRAND) is an envisioned observatory of ultra-high-energy particles of cosmic origin, with energies in excess of 100 PeV. GRAND uses large surface arrays of autonomous radio-detection units to look for the radio emission from extensive air showers that are triggered by the interaction of ultra-high-energy cosmic rays, gamma rays, and neutrinos in the atmosphere or underground. In particular, for ultra-high-energy neutrinos, the future final phase of GRAND aims to be sensitive enough to discover them in spite of their plausibly tiny flux. Presently, three prototype GRAND radio arrays are in operation: GRANDProto300, in China, GRAND@Auger, in Argentina, and GRAND@Nancay, in France. Their goals are to field-test the design of the radio-detection units, understand the radio background to which they are exposed, and develop tools for diagnostic, data gathering, and data analysis. This list of contributions to the 38th International Cosmic Ray Conference (ICRC 2023) presents an overview of GRAND, in its present and future incarnations, and a look at the first data collected by GRANDProto13, the first phase of GRANDProto300., Comment: Note: To access the list of contributions, please follow the "HTML" link that can be found on the arXiv page (v2: fixed name spelling of one author)

The vast amounts of data to be collected by the Giant Radio Array for Neutrino Detection (GRAND) and its prototype - GRANDProto300 - require the use of a data format very efficient in terms of i/o speed and compression. At the same time, the data should be easily accessible, without the knowledge of the intricacies of the format, both for bulk processing and for detailed event-by-event analysis and reconstruction. We present the format and the structure prepared for GRAND data, the concept of the data-processing chain, and data-oriented and analysis-oriented interfaces written in Python., Comment: Proceedings of the 38th International Cosmic Ray Conference (ICRC2023)