Search

Understanding connections between artificial and biological intelligent systems can reveal fundamental principles of general intelligence. While many artificial intelligence models have a neuroscience counterpart, such connections are largely missing in Transformer models and the self-attention mechanism. Here, we examine the relationship between interacting attention heads and human episodic memory. We focus on induction heads, which contribute to in-context learning in Transformer-based large language models (LLMs). We demonstrate that induction heads are behaviorally, functionally, and mechanistically similar to the contextual maintenance and retrieval (CMR) model of human episodic memory. Our analyses of LLMs pre-trained on extensive text data show that CMR-like heads often emerge in the intermediate and late layers, qualitatively mirroring human memory biases. The ablation of CMR-like heads suggests their causal role in in-context learning. Our findings uncover a parallel between the computational mechanisms of LLMs and human memory, offering valuable insights into both research fields., Comment: NeurIPS 2024

Replay is a powerful strategy to promote learning in artificial intelligence and the brain. However, the conditions to generate it and its functional advantages have not been fully recognized. In this study, we develop a modular reinforcement learning model that could generate replay. We prove that replay generated in this way helps complete the task. We also analyze the information contained in the representation and provide a mechanism for how replay makes a difference. Our design avoids complex assumptions and enables replay to emerge naturally within a task-optimized paradigm. Our model also reproduces key phenomena observed in biological agents. This research explores the structural biases in modular ANN to generate replay and its potential utility in developing efficient RL.

While advances in artificial intelligence and neuroscience have enabled the emergence of neural networks capable of learning a wide variety of tasks, our understanding of the temporal dynamics of these networks remains limited. Here, we study the temporal dynamics during learning of Hebbian Feedforward (HebbFF) neural networks in tasks of continual familiarity detection. Drawing inspiration from the field of network neuroscience, we examine the network's dynamic reconfiguration, focusing on how network modules evolve throughout learning. Through a comprehensive assessment involving metrics like network accuracy, modular flexibility, and distribution entropy across diverse learning modes, our approach reveals various previously unknown patterns of network reconfiguration. In particular, we find that the emergence of network modularity is a salient predictor of performance, and that modularization strengthens with increasing flexibility throughout learning. These insights not only elucidate the nuanced interplay of network modularity, accuracy, and learning dynamics but also bridge our understanding of learning in artificial and biological realms.

Any reinforcement learning system must be able to identify which past events contributed to observed outcomes, a problem known as credit assignment. A common solution to this problem is to use an eligibility trace to assign credit to recency-weighted set of experienced events. However, in many realistic tasks, the set of recently experienced events are only one of the many possible action events that could have preceded the current outcome. This suggests that reinforcement learning can be made more efficient by allowing credit assignment to any viable preceding state, rather than only those most recently experienced. Accordingly, we examine ``Predecessor Features'', the fully bootstrapped version of van Hasselt's ``Expected Trace'', an algorithm that achieves this richer form of credit assignment. By maintaining a representation that approximates the expected sum of past occupancies, this algorithm allows temporal difference (TD) errors to be propagated accurately to a larger number of predecessor states than conventional methods, greatly improving learning speed. The algorithm can also be naturally extended from tabular state representation to feature representations allowing for increased performance on a wide range of environments. We demonstrate several use cases for Predecessor Features and compare its performance with other approaches., Comment: Accepted to RLDM 2022

Prioritized experience replay is a reinforcement learning technique whereby agents speed up learning by replaying useful past experiences. This usefulness is quantified as the expected gain from replaying the experience, a quantity often approximated as the prediction error (TD-error). However, recent work in neuroscience suggests that, in biological organisms, replay is prioritized not only by gain, but also by "need" -- a quantity measuring the expected relevance of each experience with respect to the current situation. Importantly, this term is not currently considered in algorithms such as prioritized experience replay. In this paper we present a new approach for prioritizing experiences for replay that considers both gain and need. Our proposed algorithms show a significant increase in performance in benchmarks including the Dyna-Q maze and a selection of Atari games.

Human learning is a complex process in which future behavior is altered via the reorganization of brain activity and connectivity. It remains unknown whether activity and connectivity differentially reorganize during learning, and, if so, how that differential reorganization tracks stages of learning across distinct brain areas. Here, we address this gap in knowledge by measuring brain activity and functional connectivity in a longitudinal fMRI experiment in which healthy adult human participants learn the values of novel objects over the course of four days. An increasing similarity in activity or functional connectivity across subjects during learning reflects reorganization toward a common functional architecture. We assessed the presence of reorganization in activity and connectivity both during value learning and during the resting-state, allowing us to differentiate common elicited processes from intrinsic processes. We found a complex and dynamic reorganization of brain connectivity and activity--as a function of time, space, and performance--that occurs while subjects learn. Spatially localized brain activity reorganizes across the brain to a common functional architecture early in learning, and this reorganization tracks early learning performance. In contrast, spatially distributed connectivity reorganizes across the brain to a common functional architecture as training progresses, and this reorganization tracks later learning performance. Particularly good performance is associated with a sticky connectivity, that persists into the resting state. Broadly, our work uncovers distinct principles of reorganization in activity and connectivity at different phases of value learning, which inform the ongoing study of learning processes more generally.

A fundamental cognitive process is the ability to map value and identity onto objects as we learn about them. Exactly how such mental constructs emerge and what kind of space best embeds this mapping remains incompletely understood. Here we develop tools to quantify the space and organization of such a mapping, thereby providing a framework for studying the geometric representations of neural responses as reflected in functional MRI. Considering how human subjects learn the values of novel objects, we show that quick learners have a higher dimensional geometric representation than slow learners, and hence more easily distinguishable whole-brain responses to objects of different value. Furthermore, we find that quick learners display a more compact embedding of their neural responses and hence have a higher ratio of their task-based dimension to their embedding dimension -- consistent with a greater efficiency of cognitive coding. Lastly, we investigate the neurophysiological drivers of high dimensional patterns at both regional and voxel levels, and we complete our study with a complementary test of the distinguishability of associated whole-brain responses. Our results demonstrate a spatial organization of neural responses characteristic of learning, and offer a suite of geometric measures applicable to the study of efficient coding in higher-order cognitive processes more broadly.

Learning requires the traversal of inherently distinct cognitive states to produce behavioral adaptation. Yet, tools to explicitly measure these states with non-invasive imaging -- and to assess their dynamics during learning -- remain limited. Here, we describe an approach based on a novel application of graph theory in which points in time are represented by network nodes, and similarities in brain states between two different time points are represented as network edges. We use a graph-based clustering technique to identify clusters of time points representing canonical brain states, and to assess the manner in which the brain moves from one state to another as learning progresses. We observe the presence of two primary states characterized by either high activation in sensorimotor cortex or high activation in a frontal-subcortical system. Flexible switching among these primary states and other less common states becomes more frequent as learning progresses, and is inversely correlated with individual differences in learning rate. These results are consistent with the notion that the development of automaticity is associated with a greater freedom to use cognitive resources for other processes. Taken together, our work offers new insights into the constrained, low dimensional nature of brain dynamics characteristic of early learning, which give way to less constrained, high-dimensional dynamics in later learning., Comment: 36 pages, 7 figures

Human learning is a complex phenomenon that requires adaptive processes across a range of temporal and spacial scales. While our understanding of those processes at single scales has increased exponentially over the last few years, a mechanistic understanding of the entire phenomenon has remained elusive. We propose that progress has been stymied by the lack of a quantitative framework that can account for the full range of neurophysiological and behavioral dynamics both across scales in the systems and also across different types of learning. We posit that network neuroscience offers promise in meeting this challenge. Built on the mathematical fields of complex systems science and graph theory, network neuroscience embraces the interconnected and hierarchical nature of human learning, offering insights into the emergent properties of adaptability. In this review, we discuss the utility of network neuroscience as a tool to build a quantitative framework in which to study human learning, which seeks to explain the full chain of events in the brain from sensory input to motor output, being both biologically plausible and able to make predictions about how an intervention at a single level of the chain may cause alterations in another level of the chain. We close by laying out important remaining challenges in network neuroscience in explicitly bridging spatial scales at which neurophysiological processes occur, and underscore the utility of such a quantitative framework for education and therapy., Comment: 9 pages, 5 figures

Human brain dynamics can be profitably viewed through the lens of statistical mechanics, where neurophysiological activity evolves around and between local attractors representing preferred mental states. Many physically-inspired models of these dynamics define the state of the brain based on instantaneous measurements of regional activity. Yet, recent work in network neuroscience has provided initial evidence that the brain might also be well-characterized by time-varying states composed of locally coherent activity or functional modules. Here we study this network-based notion of brain state to understand how functional modules dynamically interact with one another to perform cognitive functions. We estimate the functional relationships between regions of interest (ROIs) by fitting a pair-wise maximum entropy model to each ROI's pattern of allegiance to functional modules. Local minima in this model represent attractor states characterized by specific patterns of modular structure. The clustering of local minima highlights three classes of ROIs with similar patterns of allegiance to community states. Visual, attention, sensorimotor, and subcortical ROIs tend to form a single functional community. The remaining ROIs tend to form a putative executive control community or a putative default mode and salience community. We simulate the brain's dynamic transitions between these community states using a Markov Chain Monte Carlo random walk. We observe that simulated transition probabilities between basins resemble empirically observed transitions between community allegiance states in resting state fMRI data. These results collectively offer a view of the brain as a dynamical system that transitions between basins of attraction characterized by coherent activity in small groups of brain regions, and that the strength of these attractors depends on the cognitive computations being performed.

Value guides behavior. With knowledge of stimulus values and action consequences, behaviors that maximize expected reward can be selected. Prior work has identified several brain structures critical for representing both stimuli and their values. Yet, it remains unclear how these structures interact with one another and with other regions of the brain to support the dynamic acquisition of value-related knowledge. Here, we use a network neuroscience approach to examine how BOLD functional networks change as 20 healthy human subjects learn the values of novel visual stimuli over the course of four consecutive days. We show that connections between regions of the visual, frontal, and cingulate cortices become increasingly stronger as learning progresses, and that these changes are primarily confined to the temporal core of the network. These results demonstrate that functional networks dynamically track behavioral improvement in value judgments, and that interactions between network communities form predictive biomarkers of learning., Comment: 27 pages; 6 figures

Human skill learning requires fine-scale coordination of distributed networks of brain regions that are directly linked to one another by white matter tracts to allow for effective information transmission. Yet how individual differences in these anatomical pathways may impact individual differences in learning remains far from understood. Here, we test the hypothesis that individual differences in the organization of structural networks supporting task performance predict individual differences in the rate at which humans learn a visuo-motor skill. Over the course of 6 weeks, twenty-two healthy adult subjects practiced a discrete sequence production task, where they learned a sequence of finger movements based on discrete visual cues. We collected structural imaging data during four MRI scanning sessions spaced approximately two weeks apart, and using deterministic tractography, structural networks were generated for each participant to identify streamlines that connect cortical and sub-cortical brain regions. We observed that increased white matter connectivity linking early visual (but not motor) regions was associated with a faster learning rate. Moreover, we observed that the strength of multi-edge paths between motor and visual modules was also correlated with learning rate, supporting the role of polysynaptic connections in successful skill acquisition. Our results demonstrate that the combination of diffusion imaging and tractography-based connectivity can be used to predict future individual differences in learning capacity, particularly when combined with methods from network science and graph theory., Comment: 11 pages, 6 figures