Search

Relational reasoning refers to the ability to infer and understand the relations between multiple entities. In humans, this ability underpins many higher cognitive functions, such as problem solving and decision-making, and has been reliably linked to fluid intelligence. Despite machine learning models making impressive advances across various domains, such as natural language processing and vision, the extent to which such models can perform relational reasoning tasks remains unclear. Here we study the importance of positional encoding (PE) for relational reasoning in the Transformer, and find that a learnable PE outperforms all other commonly-used PEs (e.g., absolute, relative, rotary, etc.). Moreover, we find that when using a PE with a learnable parameter, the choice of initialization greatly influences the learned representations and its downstream generalization performance. Specifically, we find that a learned PE initialized from a small-norm distribution can 1) uncover ground-truth position information, 2) generalize in the presence of noisy inputs, and 3) produce behavioral patterns that are consistent with human performance. Our results shed light on the importance of learning high-performing and robust PEs during relational reasoning tasks, which will prove useful for tasks in which ground truth positions are not provided or not known.

The advent of the Transformer has led to the development of large language models (LLM), which appear to demonstrate human-like capabilities. To assess the generality of this class of models and a variety of other base neural network architectures to multimodal domains, we evaluated and compared their capacity for multimodal generalization. We introduce a multimodal question-answer benchmark to evaluate three specific types of out-of-distribution (OOD) generalization performance: distractor generalization (generalization in the presence of distractors), systematic compositional generalization (generalization to new task permutations), and productive compositional generalization (generalization to more complex tasks structures). We found that across model architectures (e.g., RNNs, Transformers, Perceivers, etc.), models with multiple attention layers, or models that leveraged cross-attention mechanisms between input domains, fared better. Our positive results demonstrate that for multimodal distractor and systematic generalization, either cross-modal attention or models with deeper attention layers are key architectural features required to integrate multimodal inputs. On the other hand, neither of these architectural features led to productive generalization, suggesting fundamental limitations of existing architectures for specific types of multimodal generalization. These results demonstrate the strengths and limitations of specific architectural components underlying modern neural models for multimodal reasoning. Finally, we provide Generic COG (gCOG), a configurable benchmark with several multimodal generalization splits, for future studies to explore., Comment: ICLR 2024

During the public Kaggle competition "IceCube -- Neutrinos in Deep Ice", thousands of reconstruction algorithms were created and submitted, aiming to estimate the direction of neutrino events recorded by the IceCube detector. Here we describe in detail the three ultimate best, award-winning solutions. The data handling, architecture, and training process of each of these machine learning models is laid out, followed up by an in-depth comparison of the performance on the kaggle datatset. We show that on cascade events in IceCube above 10 TeV, the best kaggle solution is able to achieve an angular resolution of better than 5 degrees, and for tracks correspondingly better than 0.5 degrees. These performance measures compare favourably to the current state-of-the-art in the field.

Couplings between topological edge channels open electronic phases possessing nontrivial eigenmodes far beyond the noninteracting-edge picture. However, inelastic scatterings mask the eigenmodes' inherent features, often preventing us from identifying the phases, as is the case for the quintessential Landau-level filling factor v = 2/3 edge composed of the counter-propagating v = 1/3 and 1 (1/3-1) channels. Here, we study the coherent-incoherent crossover of the 1/3-1 channels by tuning the channel length in-situ using a new device architecture comprising a junction of v = 1/3 and 1 systems, the particle-hole conjugate of the 2/3 edge. We successfully observed the concurrence of the fluctuating electrical conductance and the quantized thermal conductance in the crossover regime, the definitive hallmark of the eigenmodes in the disorder-dominated edge phase left experimentally unverified.

Humans have a remarkable ability to rapidly generalize to new tasks that is difficult to reproduce in artificial learning systems. Compositionality has been proposed as a key mechanism supporting generalization in humans, but evidence of its neural implementation and impact on behavior is still scarce. Here we study the computational properties associated with compositional generalization in both humans and artificial neural networks (ANNs) on a highly compositional task. First, we identified behavioral signatures of compositional generalization in humans, along with their neural correlates using whole-cortex functional magnetic resonance imaging (fMRI) data. Next, we designed pretraining paradigms aided by a procedure we term {\em primitives pretraining} to endow compositional task elements into ANNs. We found that ANNs with this prior knowledge had greater correspondence with human behavior and neural compositional signatures. Importantly, primitives pretraining induced abstract internal representations, excellent zero-shot generalization, and sample-efficient learning. Moreover, it gave rise to a hierarchy of abstract representations that matched human fMRI data, where sensory rule abstractions emerged in early sensory areas, and motor rule abstractions emerged in later motor areas. Our findings give empirical support to the role of compositional generalization in human behavior, implicate abstract representations as its neural implementation, and illustrate that these representations can be embedded into ANNs by designing simple and efficient pretraining procedures., Comment: NeurIPS 2022. 24 pages (10 pages main text), 17 figures (8 main text figures)

Messenger RNA-based medicines hold immense potential, as evidenced by their rapid deployment as COVID-19 vaccines. However, worldwide distribution of mRNA molecules has been limited by their thermostability, which is fundamentally limited by the intrinsic instability of RNA molecules to a chemical degradation reaction called in-line hydrolysis. Predicting the degradation of an RNA molecule is a key task in designing more stable RNA-based therapeutics. Here, we describe a crowdsourced machine learning competition ("Stanford OpenVaccine") on Kaggle, involving single-nucleotide resolution measurements on 6043 102-130-nucleotide diverse RNA constructs that were themselves solicited through crowdsourcing on the RNA design platform Eterna. The entire experiment was completed in less than 6 months, and 41% of nucleotide-level predictions from the winning model were within experimental error of the ground truth measurement. Furthermore, these models generalized to blindly predicting orthogonal degradation data on much longer mRNA molecules (504-1588 nucleotides) with improved accuracy compared to previously published models. Top teams integrated natural language processing architectures and data augmentation techniques with predictions from previous dynamic programming models for RNA secondary structure. These results indicate that such models are capable of representing in-line hydrolysis with excellent accuracy, supporting their use for designing stabilized messenger RNAs. The integration of two crowdsourcing platforms, one for data set creation and another for machine learning, may be fruitful for other urgent problems that demand scientific discovery on rapid timescales.

The edge states in the fractional quantum Hall systems at filling factor $\nu=1/3$ are studied by the density matrix renormalization group method. It is shown that the density oscillation induced by the local boundary condition at the edge is characterized by the wave number of the minimum magnetoroton excitation, and this structure is partially reconstructed with the change in the confinement potential shape. In particular, the $\nu=1$ counterpropagating edge channel appears with the change in the chemical potential, which is consistent with recent experiments on heat transport. The stability of the bulk states against the change in the number of electrons confirms that the bulk part of the fractional quantum Hall state is incompressible, while the edge state is compressible.

Understanding neurocognitive computations will require not just localizing cognitive information distributed throughout the brain but also determining how that information got there. We review recent advances in linking empirical and simulated brain network organization with cognitive information processing. Building on these advances, we offer a new framework for understanding the role of connectivity in cognition - network coding (encoding/decoding) models. These models utilize connectivity to specify the transfer of information via neural activity flow processes, successfully predicting the formation of cognitive representations in empirical neural data. The success of these models supports the possibility that localized neural functions mechanistically emerge (are computed) from distributed activity flow processes that are specified primarily by connectivity patterns.

Thermodynamic properties of the twisted three-leg spin tube under magnetic field are studied by the finite-$T$ density-matrix renormalization group method. The specific heat, spin, and chiral susceptibilities of the infinite system are calculated for both the original and its low-energy effective models. The obtained results show that the presence of the chirality is observed as a clear peak in the specific heat at low temperature and the contribution of the chirality dominates the low-temperature part of the specific heat as the exchange coupling along the spin tube decreases. The peak structures in the specific heat, spin, and chiral susceptibilities are strongly modified near the quantum phase transition where the critical behaviors of the spin and chirality correlations change. These results confirm that the chirality plays a major role in characteristic low-energy behaviors of the frustrated spin systems., Comment: 6 pages, 7 figures