Search

We present the Brain Treebank, a large-scale dataset of electrophysiological neural responses, recorded from intracranial probes while 10 subjects watched one or more Hollywood movies. Subjects watched on average 2.6 Hollywood movies, for an average viewing time of 4.3 hours, and a total of 43 hours. The audio track for each movie was transcribed with manual corrections. Word onsets were manually annotated on spectrograms of the audio track for each movie. Each transcript was automatically parsed and manually corrected into the universal dependencies (UD) formalism, assigning a part of speech to every word and a dependency parse to every sentence. In total, subjects heard over 38,000 sentences (223,000 words), while they had on average 168 electrodes implanted. This is the largest dataset of intracranial recordings featuring grounded naturalistic language, one of the largest English UD treebanks in general, and one of only a few UD treebanks aligned to multimodal features. We hope that this dataset serves as a bridge between linguistic concepts, perception, and their neural representations. To that end, we present an analysis of which electrodes are sensitive to language features while also mapping out a rough time course of language processing across these electrodes. The Brain Treebank is available at https://BrainTreebank.dev/, Comment: 36 pages, 17 figures; Accepted at NeurIPS Dataset and Benchmarks 2024

In this paper we look for the convex hull of a set using the geometric evolution by minimal curvature of a hypersurface that surrounds the set. To find the convex hull, we study the large time behavior of solutions to an obstacle problem for the level set formulation of the geometric flow driven by the minimum of the principal curvatures (that coincides with the mean curvature flow only in two dimensions). We prove that the superlevel set where the solution to this obstacle problem is positive converges as time goes to infinity to the convex hull of the obstacle. Our approach is based on a game-theoretic approximation for this geometric flow that is inspired by previous results for the mean curvature flow.

Humans solve problems by following existing rules and procedures, and also by leaps of creativity to redefine those rules and objectives. To probe these abilities, we developed a new benchmark based on the game Baba Is You where an agent manipulates both objects in the environment and rules, represented by movable tiles with words written on them, to reach a specified goal and win the game. We test three state-of-the-art multi-modal large language models (OpenAI GPT-4o, Google Gemini-1.5-Pro and Gemini-1.5-Flash) and find that they fail dramatically when generalization requires that the rules of the game must be manipulated and combined., Comment: 8 pages, 8 figures

We use (multi)modal deep neural networks (DNNs) to probe for sites of multimodal integration in the human brain by predicting stereoencephalography (SEEG) recordings taken while human subjects watched movies. We operationalize sites of multimodal integration as regions where a multimodal vision-language model predicts recordings better than unimodal language, unimodal vision, or linearly-integrated language-vision models. Our target DNN models span different architectures (e.g., convolutional networks and transformers) and multimodal training techniques (e.g., cross-attention and contrastive learning). As a key enabling step, we first demonstrate that trained vision and language models systematically outperform their randomly initialized counterparts in their ability to predict SEEG signals. We then compare unimodal and multimodal models against one another. Because our target DNN models often have different architectures, number of parameters, and training sets (possibly obscuring those differences attributable to integration), we carry out a controlled comparison of two models (SLIP and SimCLR), which keep all of these attributes the same aside from input modality. Using this approach, we identify a sizable number of neural sites (on average 141 out of 1090 total sites or 12.94%) and brain regions where multimodal integration seems to occur. Additionally, we find that among the variants of multimodal training techniques we assess, CLIP-style training is the best suited for downstream prediction of the neural activity in these sites., Comment: ICML 2024; 23 pages, 11 figures

Using nonequilibrium computer simulations, we study the response of ferromagnetic nanofilaments, consisting of stabilized onedimensional chains of ferromagnetic nanoparticles, under external rotating magnetic fields. In difference with their analogous microscale and stiff counterparts, which have been actively studied in recent years, nonequilibrium properties of rather flexible nanoparticle filaments remain mostly unexplored. By progressively increasing the modeling details, we are able to evidence the qualitative impact of main interactions that can not be neglected at the nanoscale, showing that filament flexibility, thermal fluctuations and hydrodynamic interactions contribute independently to broaden the range of synchronous frequency response in this system. Furthermore, we also show the existence of a limited set of characteristic dynamic filament configurations and discuss in detail the asynchronous response, which at finite temperature becomes probabilistic.

We study the positivity and asymptotic behaviour of nonnegative solutions of a general nonlocal fast diffusion equation, \[\partial_t u + \mathcal{L}\varphi(u) = 0,\] and the interplay between these two properties. Here $\mathcal{L}$ is a stable-like operator and $\varphi$ is a singular nonlinearity. We start by analysing positivity by means of a weak Harnack inequality satisfied by a related elliptic (nonlocal) equation. Then we use this positivity to establish the asymptotic behaviour: under certain hypotheses on the nonlocal operator and nonlinearity, our solutions behave asymptotically as the Barenblatt solution of the standard fractional fast diffusion equation. The main difficulty stems from the generality of the operator, which does not allow the use of the methods that were available for the fractional Laplacian. Our results are new even in the case where $\varphi$ is a power.

In this paper we introduce two new fractional versions of the Laplacian. The first one is based on the classical formula that writes the usual Laplacian as the sum of the eigenvalues of the Hessian. The second one comes from looking at the classical fractional Laplacian as the mean value (in the sphere) of the 1-dimensional fractional Laplacians in lines with directions in the sphere. To obtain this second new fractional operator we just replace the mean value by the mid-range of 1-dimensional fractional Laplacians with directions in the sphere. For these two new fractional operators we prove a comparison principle for viscosity sub and supersolutions and then we obtain existence and uniqueness for the Dirichlet problem. We also show that solutions are $C^\gamma$ smooth up to the boundary when the exterior datum is also H\"older continuous. Finally, we prove that for the first operator we recover the classical Laplacian in the limit as $s\nearrow 1$.

In an introductory paper on dollar game played on a graph, Matt Baker wrote the following: ``The total number of borrowing moves required to win the game when playing the 'borrowing binge strategy' is independent of which borrowing moves you do in which order! Note, however, that it is usually possible to win in fewer moves by employing lending moves in combination with borrowing moves. The optimal strategy when one uses both kinds of moves is not yet understood.'' In this article, we give a lower bound on the minimum number $ M_{\text{min}} $ of such moves of an optimal algorithm in terms of the number of moves $ M_0 $ of the borrowing binge strategy. Concretely, we have: $ M_{\text{min}} \geq \frac{M_0}{n-1} $ where $ n $ is the number of vertices of the graph. This bound is tight., Comment: 9 pages, 3 figures

We create a reusable Transformer, BrainBERT, for intracranial recordings bringing modern representation learning approaches to neuroscience. Much like in NLP and speech recognition, this Transformer enables classifying complex concepts, i.e., decoding neural data, with higher accuracy and with much less data by being pretrained in an unsupervised manner on a large corpus of unannotated neural recordings. Our approach generalizes to new subjects with electrodes in new positions and to unrelated tasks showing that the representations robustly disentangle the neural signal. Just like in NLP where one can study language by investigating what a language model learns, this approach opens the door to investigating the brain by what a model of the brain learns. As a first step along this path, we demonstrate a new analysis of the intrinsic dimensionality of the computations in different areas of the brain. To construct these representations, we combine a technique for producing super-resolution spectrograms of neural data with an approach designed for generating contextual representations of audio by masking. In the future, far more concepts will be decodable from neural recordings by using representation learning, potentially unlocking the brain like language models unlocked language., Comment: 9 pages, 6 figures, ICLR 2023

The mixing time of the Markov chain induced by a policy limits performance in real-world continual learning scenarios. Yet, the effect of mixing times on learning in continual reinforcement learning (RL) remains underexplored. In this paper, we characterize problems that are of long-term interest to the development of continual RL, which we call scalable MDPs, through the lens of mixing times. In particular, we theoretically establish that scalable MDPs have mixing times that scale polynomially with the size of the problem. We go on to demonstrate that polynomial mixing times present significant difficulties for existing approaches, which suffer from myopic bias and stale bootstrapped estimates. To validate our theory, we study the empirical scaling behavior of mixing times with respect to the number of tasks and task duration for high performing policies deployed across multiple Atari games. Our analysis demonstrates both that polynomial mixing times do emerge in practice and how their existence may lead to unstable learning behavior like catastrophic forgetting in continual learning settings., Comment: Accepted at NeurIPS 2022

The options framework is a popular approach for building temporally extended actions in reinforcement learning. In particular, the option-critic architecture provides general purpose policy gradient theorems for learning actions from scratch that are extended in time. However, past work makes the key assumption that each of the components of option-critic has independent parameters. In this work we note that while this key assumption of the policy gradient theorems of option-critic holds in the tabular case, it is always violated in practice for the deep function approximation setting. We thus reconsider this assumption and consider more general extensions of option-critic and hierarchical option-critic training that optimize for the full architecture with each update. It turns out that not assuming parameter independence challenges a belief in prior work that training the policy over options can be disentangled from the dynamics of the underlying options. In fact, learning can be sped up by focusing the policy over options on states where options are actually likely to terminate. We put our new algorithms to the test in application to sample efficient learning of Atari games, and demonstrate significantly improved stability and faster convergence when learning long options., Comment: AAAI 2020

Deep learning models for semantics are generally evaluated using naturalistic corpora. Adversarial methods, in which models are evaluated on new examples with known semantic properties, have begun to reveal that good performance at these naturalistic tasks can hide serious shortcomings. However, we should insist that these evaluations be fair -that the models are given data sufficient to support the requisite kinds of generalization. In this paper, we define and motivate a formal notion of fairness in this sense. We then apply these ideas to natural language inference by constructing very challenging but provably fair artificial datasets and showing that standard neural models fail to generalize in the required ways; only task-specific models that jointly compose the premise and hypothesis are able to achieve high performance, and even these models do not solve the task perfectly.

Compositionality is a key strategy for addressing combinatorial complexity and the curse of dimensionality. Recent work has shown that compositional solutions can be learned and offer substantial gains across a variety of domains, including multi-task learning, language modeling, visual question answering, machine comprehension, and others. However, such models present unique challenges during training when both the module parameters and their composition must be learned jointly. In this paper, we identify several of these issues and analyze their underlying causes. Our discussion focuses on routing networks, a general approach to this problem, and examines empirically the interplay of these challenges and a variety of design decisions. In particular, we consider the effect of how the algorithm decides on module composition, how the algorithm updates the modules, and if the algorithm uses regularization.

Standard evaluations of deep learning models for semantics using naturalistic corpora are limited in what they can tell us about the fidelity of the learned representations, because the corpora rarely come with good measures of semantic complexity. To overcome this limitation, we present a method for generating data sets of multiply-quantified natural language inference (NLI) examples in which semantic complexity can be precisely characterized, and we use this method to show that a variety of common architectures for NLI inevitably fail to encode crucial information; only a model with forced lexical alignments avoids this damaging information loss.

Lack of performance when it comes to continual learning over non-stationary distributions of data remains a major challenge in scaling neural network learning to more human realistic settings. In this work we propose a new conceptualization of the continual learning problem in terms of a temporally symmetric trade-off between transfer and interference that can be optimized by enforcing gradient alignment across examples. We then propose a new algorithm, Meta-Experience Replay (MER), that directly exploits this view by combining experience replay with optimization based meta-learning. This method learns parameters that make interference based on future gradients less likely and transfer based on future gradients more likely. We conduct experiments across continual lifelong supervised learning benchmarks and non-stationary reinforcement learning environments demonstrating that our approach consistently outperforms recently proposed baselines for continual learning. Our experiments show that the gap between the performance of MER and baseline algorithms grows both as the environment gets more non-stationary and as the fraction of the total experiences stored gets smaller., Comment: ICLR 2019

Neural networks have excelled at many NLP tasks, but there remain open questions about the performance of pretrained distributed word representations and their interaction with weight initialization and other hyperparameters. We address these questions empirically using attention-based sequence-to-sequence models for natural language inference (NLI). Specifically, we compare three types of embeddings: random, pretrained (GloVe, word2vec), and retrofitted (pretrained plus WordNet information). We show that pretrained embeddings outperform both random and retrofitted ones in a large NLI corpus. Further experiments on more controlled data sets shed light on the contexts for which retrofitted embeddings can be useful. We also explore two principled approaches to initializing the rest of the model parameters, Gaussian and orthogonal, showing that the latter yields gains of up to 2.9% in the NLI task., Comment: This manuscript dates from late Winter 2016

The mechanical properties of PC12 living cells have been studied at the nanoscale with a Force Feedback Microscope using two experimental approaches. Firstly, the local mechanical impedance of the cell membrane has been mapped simultaneously to the cell morphology at constant force. As the force of the interaction is gradually increased, we observed the appearance of the sub-membrane cytoskeleton. We shall compare the results obtained with this method with the measurement of other existing techniques. Secondly, a spectroscopic investigation has been performed varying the indentation of the tip in the cell membrane and consequently the force applied on it. In contrast with conventional dynamic atomic force microscopy techniques, here the small oscillation amplitude of the tip is not necessarily imposed at the cantilever first eigenmode. This allows the user to arbitrarily choose the excitation frequency in developing spectroscopic AFM techniques. The mechanical response of the PC12 cell membrane is found to be frequency dependent in the 1 kHz - 10 kHz range. The damping coefficient is reproducibly observed to decrease when the excitation frequency is increased., Comment: 8 pages, 8 figures

This work analyzes the problem of community structure in real-world networks based on the synchronization of nonidentical coupled chaotic R\"{o}ssler oscillators each one characterized by a defined natural frequency, and coupled according to a predefined network topology. The interaction scheme contemplates an uniformly increasing coupling force to simulate a society in which the association between the agents grows in time. To enhance the stability of the correlated states that could emerge from the synchronization process, we propose a parameterless mechanism that adapts the characteristic frequencies of coupled oscillators according to a dynamic connectivity matrix deduced from correlated data. We show that the characteristic frequency vector that results from the adaptation mechanism reveals the underlying community structure present in the network., Comment: 21 pages, 7 figures; Chaos, Solitons & Fractals (2012)

Statistical evidence of the influence of neighborhood topology on the performance of particle swarm optimization (PSO) algorithms has been shown in many works. However, little has been done about the implications could have the percolation threshold in determining the topology of this neighborhood. This work addresses this problem for individuals that, like robots, are able to sense in a limited neighborhood around them. Based on the concept of percolation threshold, and more precisely, the disk percolation model in 2D, we show that better results are obtained for low values of radius, when individuals occasionally ask others their best visited positions, with the consequent decrease of computational complexity. On the other hand, since percolation threshold is a universal measure, it could have a great interest to compare the performance of different hybrid PSO algorithms.

Apparent competition between species is believed to be one of the principle driving forces that structure ecological communities, although the precise mecha nisms have yet to be characterized. Here we develop a model system that isolates phage-mediated interactions by neutralizing resource competition using two genetically identical Bordetella bronchiseptica strains that differ only in that one is the carrier of a phage and the other is susceptible to the phage. We observe and quantify the competitive advantage of the bacterial strain bearing the prophage in both invading and in resisting invasion by bacteria susceptible to the phage, and use our measurements to develop a mathematical model of phage-mediated competition. The model predicts, and experimental evidence confirms, that the competitive advantage conferred by the phage depends only on the relative phage pathology and is independent of other phage and host parameters. This work combines experimental and mathematical approaches to the study of phage-driven competition, and provides an experimentally tested framework for evaluation of the effects of pathogens/parasites on interspecific competition., Comment: 10pages, 8 figures