Search

One of the most important and challenging application areas for complex machine learning methods is to predict, characterize and model rich, multi-dimensional, neural data. Recent advances in neural recording techniques have made it possible to monitor the activities of a large number of neurons across different brain regions as animals perform behavioural tasks. This poses the critical challenge of establishing links between neural activity at a microscopic scale, which might for instance represent sensory input, and at a macroscopic scale, which then generates behaviour. Predominant modeling methods apply rather disjoint techniques to these scales; by contrast, we suggest an end-to-end model which exploits recent developments of flexible, but tractable, neural network point-process models to characterize dependencies between stimuli, actions, and neural data. We apply this model to a public dataset collected using Neuropixel probes in mice performing a visually-guided behavioural task as well as a synthetic dataset produced from a hierarchical network model with reciprocally connected sensory and integration circuits intended to characterize animal behaviour in a fixed-duration motion discrimination task. We show that our model outperforms previous approaches and contributes novel insights into the relationships between neural activities and behaviour.

Significance “What I cannot efficiently break, I cannot understand.” Understanding the vulnerabilities of human choice processes allows us to detect and potentially avoid adversarial attacks. We develop a general framework for creating adversaries for human decision-making. The framework is based on recent developments in deep reinforcement learning models and recurrent neural networks and can in principle be applied to any decision-making task and adversarial objective. We show the performance of the framework in three tasks involving choice, response inhibition, and social decision-making. In all of the cases the framework was successful in its adversarial attack. Furthermore, we show various ways to interpret the models to provide insights into the exploitability of human choice., Adversarial examples are carefully crafted input patterns that are surprisingly poorly classified by artificial and/or natural neural networks. Here we examine adversarial vulnerabilities in the processes responsible for learning and choice in humans. Building upon recent recurrent neural network models of choice processes, we propose a general framework for generating adversarial opponents that can shape the choices of individuals in particular decision-making tasks toward the behavioral patterns desired by the adversary. We show the efficacy of the framework through three experiments involving action selection, response inhibition, and social decision-making. We further investigate the strategy used by the adversary in order to gain insights into the vulnerabilities of human choice. The framework may find applications across behavioral sciences in helping detect and avoid flawed choice.

[This is a preprint of the manuscript published in Cognition: Bröker, Love, & Dayan (2022).] Humans continuously categorise inputs, but only rarely receive explicit feedback as to whether or not they are correct. This implies that they may be integrating unsupervised information together with their sparse supervised data -- a form of semi-supervised learning. However, experiments testing semi-supervised learning are rare, and are bedevilled with conflicting results about whether the unsupervised information affords any benefit. Here, we suggest that one important factor that has been paid insufficient attention is the alignment between subjects' internal representations of the stimulus material and the experimenter-defined representations that determine success in the tasks. Subjects' representations are shaped by prior biases and experience, and unsupervised learning can only be successful if the alignment suffices. Otherwise, unsupervised learning might harmfully strengthen incorrect assumptions. To test this hypothesis, we conducted an experiment in which subjects initially categorise items along a salient, but task-irrelevant, dimension, and only recover the correct categories when sufficient feedback draws their attention to the subtle, task-relevant, stimulus dimensions. By withdrawing feedback at different stages along this learning curve, we tested whether unsupervised learning improves or worsens performance when internal stimulus representations and task are sufficiently or insufficiently aligned, respectively. Our results demonstrate that unsupervised learning can indeed have opposing effects on subjects' learning. We also discuss factors limiting the degree to which such effects can be predicted from momentary performance. Our work implies that predicting and understanding human category learning in particular tasks requires assessment and consideration of the representational spaces that subjects entertain for the materials involved in those tasks. These considerations not only apply to studies in the lab, but could also help improve the design of tutoring systems and instruction.

Summary Background Benzodiazepine-refractory, or established, status epilepticus is thought to be of similar pathophysiology in children and adults, but differences in underlying aetiology and pharmacodynamics might differentially affect response to therapy. In the Established Status Epilepticus Treatment Trial (ESETT) we compared the efficacy and safety of levetiracetam, fosphenytoin, and valproate in established status epilepticus, and here we describe our results after extending enrolment in children to compare outcomes in three age groups. Methods In this multicentre, double-blind, response-adaptive, randomised controlled trial, we recruited patients from 58 hospital emergency departments across the USA. Patients were eligible for inclusion if they were aged 2 years or older, had been treated for a generalised convulsive seizure of longer than 5 min duration with adequate doses of benzodiazepines, and continued to have persistent or recurrent convulsions in the emergency department for at least 5 min and no more than 30 min after the last dose of benzodiazepine. Patients were randomly assigned in a response-adaptive manner, using Bayesian methods and stratified by age group ( 65 years), to levetiracetam, fosphenytoin, or valproate. All patients, investigators, study staff, and pharmacists were masked to treatment allocation. The primary outcome was absence of clinically apparent seizures with improved consciousness and without additional antiseizure medication at 1 h from start of drug infusion. The primary safety outcome was life-threatening hypotension or cardiac arrhythmia. The efficacy and safety outcomes were analysed by intention to treat. This study is registered in ClinicalTrials.gov , NCT01960075 . Findings Between Nov 3, 2015, and Dec 29, 2018, we enrolled 478 patients and 462 unique patients were included: 225 children (aged 65 years). 175 (38%) patients were randomly assigned to levetiracetam, 142 (31%) to fosphenyltoin, and 145 (31%) were to valproate. Baseline characteristics were balanced across treatments within age groups. The primary efficacy outcome was met in those treated with levetiracetam for 52% (95% credible interval 41–62) of children, 44% (33–55) of adults, and 37% (19–59) of older adults; with fosphenytoin in 49% (38–61) of children, 46% (34–59) of adults, and 35% (17–59) of older adults; and with valproate in 52% (41–63) of children, 46% (34–58) of adults, and 47% (25–70) of older adults. No differences were detected in efficacy or primary safety outcome by drug within each age group. With the exception of endotracheal intubation in children, secondary safety outcomes did not significantly differ by drug within each age group. Interpretation Children, adults, and older adults with established status epilepticus respond similarly to levetiracetam, fosphenytoin, and valproate, with treatment success in approximately half of patients. Any of the three drugs can be considered as a potential first-choice, second-line drug for benzodiazepine-refractory status epilepticus. Funding National Institute of Neurological Disorders and Stroke, National Institutes of Health.

Adversarial examples are carefully crafted input patterns that are surprisingly poorly classified by artificial and/or natural neural networks. Here we examine adversarial vulnerabilities in the processes responsible for learning and choice in humans. Building upon recent recurrent neural network models of choice processes, we propose a general framework for generating adversarial opponents that can shape the choices of individuals in particular decision-making tasks towards the behavioural patterns desired by the adversary. We show the efficacy of the framework through two experiments involving action selection and response inhibition. We further investigate the strategy used by the adversary in order to gain insights into the vulnerabilities of human choice. The framework may find applications across behavioural sciences in helping detect and avoid flawed choice.

How does the brain learn how to plan? We reverse-engineer people's underlying learning mechanisms by combining rational process models of cognitive plasticity with recently developed empirical methods that allow us to trace the temporal evolution of people's planning strategies. We find that our Learned Value of Computation model (LVOC) accurately captures people's average learning curve. However, there were also substantial individual differences in metacognitive learning that are best understood in terms of multiple different learning mechanisms -- including strategy selection learning. Furthermore, we observed that LVOC could not fully capture people's ability to adaptively decide when to stop planning. We successfully extended the LVOC model to address these discrepancies. Our models broadly capture people's ability to improve their decision mechanisms and represent a significant step towards reverse-engineering how the brain learns increasingly more effective cognitive strategies through its interaction with the environment.

Individual characteristics in human decision-making are often quantified by fitting a parametric cognitive model to subjects’ behavior and then studying differences between them in the associated parameter space. However, these models often fit behavior more poorly than recurrent neural networks (RNNs), which are more flexible and make fewer assumptions about the underlying decision-making processes. Unfortunately, the parameter and latent activity spaces of RNNs are generally high-dimensional and uninterpretable, making it hard to use them to study individual differences. Here, we show how to benefit from the flexibility of RNNs while representing individual differences in a low-dimensional and interpretable space. To achieve this, we propose a novel end-to-end learning framework in which an encoder is trained to map the behavior of subjects into a low-dimensional latent space. These low-dimensional representations are used to generate the parameters of individual RNNs corresponding to the decision-making process of each subject. We introduce terms into the loss function that ensure that the latent dimensions are informative and disentangled, i.e., encouraged to have distinct effects on behavior. This allows them to align with separate facets of individual differences. We illustrate the performance of our framework on synthetic data as well as a dataset including the behavior of patients with psychiatric disorders.

Contemporary reinforcement learning (RL) theory suggests that potential choices can be evaluated by strategies that may or may not be sensitive to the computational structure of tasks. A paradigmatic model-free (MF) strategy simply repeats actions that have been rewarded in the past; by contrast, model-sensitive (MS) strategies exploit richer information associated with knowledge of task dynamics. MF and MS strategies should typically be combined, because they have complementary statistical and computational strengths; however, this tradeoff between MF/MS RL has mostly only been demonstrated in humans, often with only modest numbers of trials. We trained rhesus monkeys to perform a two-stage decision task designed to elicit and discriminate the use of MF and MS methods. A descriptive analysis of choice behaviour revealed directly that the structure of the task (of MS importance) and the reward history (of MF and MS importance) significantly influenced both choice and response vigour. A detailed, trial-by-trial computational analysis confirmed that choices were made according to a combination of strategies, with a dominant influence of a particular form of model sensitivity that persisted over weeks of testing. The residuals from this model necessitated development of a new combined RL model which incorporates a particular credit assignment weighting procedure. Finally, response vigor exhibited a subtly different collection of MF and MS influences. These results provide new illumination onto RL behavioural processes in non-human primates., Author summary We routinely solve planning problems in which present decisions have consequences in the future. These pose complex computational and statistical problems and are addressed by multiple systems in the brain which use different solutions to these problems, and which may compete and cooperate. We trained two rhesus monkeys on a paradigmatic planning task that transparently reveals canonical aspects of different strategies. We performed a detailed behavioral analysis using methods of reinforcement learning on choice and reaction time to reveal conjoint influences and structural interactions of different sources of information. We show the strengths and limitations of these analyses, at the same time as we provide a novel perspective on how different learning systems interact for choice in non-human primates.

Humans and other animals are able to discover underlying statistical structure in their environments and exploit it to achieve efficient and effective performance. However, such structure is often difficult to learn and use because it is obscure, involving long-range temporal dependencies. Here, we analysed behavioural data from an extended experiment with rats, showing that the subjects learned the underlying statistical structure, albeit suffering at times from immediate inferential imperfections as to their current state within it. We accounted for their behaviour using a Hidden Markov Model, in which recent observations are integrated with the recollections of an imperfect memory. We found that over the course of training, subjects came to track their progress through the task more accurately, a change that our model largely attributed to decreased forgetting. This ‘learning to remember’ decreased reliance on recent observations, which may be misleading, in favour of a longer-term memory.Author summaryHumans and other animals possess the remarkable ability to find and exploit patterns and structures in their experience of a complex and varied world. However, such structures are often temporally extended and latent or hidden, being only partially correlated with immediate observations of the world. This makes it essential to integrate current and historical information, and creates a challenging statistical and computational problem.Here, we examine the behaviour of rats facing a version of this challenge posed by a brain-stimulation reward task. We find that subjects learned the general structure of the task, but struggled when immediate observations were misleading. We captured this behaviour with a model in which subjects integrated evidence from their observations together with a memory whose imperfections accounted for their errors.The subjects’ performance improved markedly over successive sessions, allowing them to overcome misleading observations. According to the model, this arose from a process of ‘learning to remember’ in which subjects became better at employing more reliable past observations to determine the hidden state of the world.

Popular computational models of decision-making make specific assumptions about learning processes that may cause them to underfit observed behaviours. Here we suggest an alternative method using recurrent neural networks (RNNs) to generate a flexible family of models that have sufficient capacity to represent the complex learning and decision- making strategies used by humans. In this approach, an RNN is trained to predict the next action that a subject will take in a decision-making task and, in this way, learns to imitate the processes underlying subjects’ choices and their learning abilities. We demonstrate the benefits of this approach using a new dataset drawn from patients with either unipolar (n = 34) or bipolar (n = 33) depression and matched healthy controls (n = 34) making decisions on a two-armed bandit task. The results indicate that this new approach is better than baseline reinforcement-learning methods in terms of overall performance and its capacity to predict subjects’ choices. We show that the model can be interpreted using off-policy simulations and thereby provides a novel clustering of subjects’ learning processes—something that often eludes traditional approaches to modelling and behavioural analysis., Author summary Computational models of decision-making provide a quantitative characterisation of the learning and choice processes behind human actions. Designing a computational model is often based on manual engineering with an iterative process to examine the consistency between different aspects of the model and the empirical data. In practice, however, inconsistencies between the model and observed behaviours can remain hidden behind examined summary statistics. To address this limitation, we developed a recurrent neural network (RNNs) as a flexible type of model that can automatically characterize human decision-making processes without requiring tweaking and engineering. To show the benefits of this new approach, we collected data on a decision-making task conducted on subjects with either bipolar or unipolar depression, as well as healthy controls. The results showed that, indeed, important aspects of decision-making remained uncaptured by typical computational models and even their enhanced variants, but were captured by RNNs automatically. Further, we were able to show that the nature of such processes can be unveiled by simulating the model under various conditions. This new approach can be used, therefore, as a standalone model of decision-making or as a baseline model to evaluate how well other candidate models fit observed data.

The temporal difference (TD) learning framework is a major paradigm for understanding value-based decision making and related neural activities (e.g., dopamine activity). The representation of time in neural processes modeled by a TD framework, however, is poorly understood. To address this issue, we propose a TD formulation that separates the time of the operator (neural valuation processes), which we refer to as internal time, from the time of the observer (experiment), which we refer to as conventional time. We provide the formulation and theoretical characteristics of this TD model based on internal time, called internal-time TD, and explore the possible consequences of the use of this model in neural value-based decision making. Due to the separation of the two times, internal-time TD computations, such as TD error, are expressed differently, depending on both the time frame and time unit. We examine this operator-observer problem in relation to the time representation used in previous TD models. An internal time TD value function exhibits the co-appearance of exponential and hyperbolic discounting at different delays in intertemporal choice tasks. We further examine the effects of internal time noise on TD error, the dynamic construction of internal time, and the modulation of internal time with the internal time hypothesis of serotonin function. We also relate the internal TD formulation to research on interval timing and subjective time.

Significance Locus coeruleus (LC) integrity in cognitively normal older adults is a potentially important preclinical marker in dementia. Our study establishes a link between variability in LC integrity and cognitive decline related to noradrenergic modulation in old age. We find that in older adults, reduced LC integrity explains lower memory performance. This effect was more pronounced for memory related to negative events, and accompanied by increased pupil diameter size in response to negative events. The study provides a strong motivation for future research investigating the role of LC integrity in healthy, as well as in pathological, aging., The locus coeruleus (LC) is the principal origin of noradrenaline in the brain. LC integrity varies considerably across healthy older individuals, and is suggested to contribute to altered cognitive functions in aging. Here we test this hypothesis using an incidental memory task that is known to be susceptible to noradrenergic modulation. We used MRI neuromelanin (NM) imaging to assess LC structural integrity and pupillometry as a putative index of LC activation in both younger and older adults. We show that older adults with reduced structural LC integrity show poorer subsequent memory. This effect is more pronounced for emotionally negative events, in accord with a greater role for noradrenergic modulation in encoding salient or aversive events. In addition, we found that salient stimuli led to greater pupil diameters, consistent with increased LC activation during the encoding of such events. Our study presents novel evidence that a decrement in noradrenergic modulation impacts on specific components of cognition in healthy older adults. The findings provide a strong motivation for further investigation of the effects of altered LC integrity in pathological aging.

It is normally, if tacitly, assumed that nature came before poetry, and music somewhere in between the two. Mallarmé’s ‘Bucolique’, in the folds of its syntax, demonstrates why this belief is unsustainable. None of those concepts really functions unless, at a more or less hidden level, each can behave as if it were both anterior to and born of the others. Mallarmé does not assert this as a rational truth; rather, he shows this behaviour at work. Poetry, in this view, is both the consequence and the precondition of nature — and vice versa. Hence, perhaps, ecopoetry is the only poetry.

OBJECTIVE: Cranial computed tomography (CT) scans are frequently obtained in the evaluation of blunt head trauma in children. These scans may detect unexpected incidental findings. The objectives of this study were to determine the prevalence and significance of incidental findings on cranial CT scans in children evaluated for blunt head trauma. METHODS: This was a secondary analysis of a multicenter study of pediatric blunt head trauma. Patients RESULTS: Of the 43 904 head-injured children enrolled in the parent study, 15 831 underwent CT scans, and these latter patients serve as the study cohort. On 670 of these scans, nontraumatic findings were identified, with 16 excluded due to previously known abnormalities or surgeries. The remaining 654 represent a 4% prevalence of incidental findings. Of these, 195 (30%), representing 1% of the overall sample, warranted immediate intervention or outpatient follow-up. CONCLUSIONS: A small but important number of children evaluated with CT scans after blunt head trauma had incidental findings. Physicians who order cranial CTs must be prepared to interpret incidental findings, communicate with families, and ensure appropriate follow-up. There are ethical implications and potential health impacts of informing patients about incidental findings.

Objective To examine emergency department (ED) staff's knowledge of traumatic stress in children, attitudes toward providing psychosocial care, and confidence in doing so, and also to examine differences in these outcomes according to demographic, professional, and organizational characteristics, and training preferences. Study design We conducted an online survey among staff in ED and equivalent hospital departments, based on the Psychological First Aid and Distress-Emotional Support-Family protocols. Main analyses involved descriptive statistics and multiple regressions. Respondents were 2648 ED staff from 87 countries (62.2% physicians and 37.8% nurses; mean years of experience in emergency care was 9.5 years with an SD of 7.5 years; 25.2% worked in a low- or middle-income country). Results Of the respondents, 1.2% correctly answered all 7 knowledge questions, with 24.7% providing at least 4 correct answers. Almost all respondents (90.1%) saw all 18 identified aspects of psychosocial care as part of their job. Knowledge and confidence scores were associated with respondent characteristics (eg, years of experience, low/middle vs high-income country), although these explained no more than 11%-18% of the variance. Almost all respondents (93.1%) wished to receive training, predominantly through an interactive website or one-off group training. A small minority (11.1%) had previously received training. Conclusions More education of ED staff regarding child traumatic stress and psychosocial care appears needed and would be welcomed. Universal education packages that are readily available can be modified for use in the ED.

One character in Cortázar's novel (Persio) truly believes in cosmic rhythm. This belief is characteristic of a magical view of the universe central to 1960s (proto-‘New Age’) counterculture. The other characters in Los Premios, like the implied narrator, reject Persio's essentialism; they dismiss the notion that there is really any rhythm common to art, humanity, and the universe. However, there are key points in the narrative, inspired by falling in love and by works of art, at which their world does appear patterned by just such a rhythm, a ‘swing cósmico’. The novel itself turns out to depend on the intermittent conviction of this rhythm, not objectively embedded in anything, but always seen, living, and dying in time; the price of art is the acceptance of this rhythmed mortality.

Of course, ‘bien sur’ . . . Derrida would never have found the time to finish anything, to get to the end of anything, to see anything through properly, to do what he would have wanted to do. But perhaps the time that he would never have had was only one kind: the philosophical kind, the kind we are aware of as distinctively human. Perhaps there is another time, animal and poetic, which he did find. The paragraph from L’animal que donc je suis (The Animal That Therefore I am) of which I have quoted the first sentence continues thus

To describe the patterns of referral and use of resources for patients with psychiatric-related visits presenting to pediatric emergency departments (EDs) in a pediatric research network.We conducted a retrospective chart review of a random sample of patients (approximately 10 charts per month per site) who presented with psychiatric-related visits in 2002 to 4 pediatric EDs in the Pediatric Emergency Care Applied Research Network. Emergency department resource use variables evaluated included the use of consultation services, restraints, and laboratory tests as well as ED length of stay.We reviewed 462 patient visits with a psychiatric-related ED diagnosis. Mean (SD) age was 12.8 (3.7) years, 52% were male, and 49% were African American. The most common chief complaints were suicidality (47%), aggression/agitation (42%), and anxiety/depression (27%), alone or in combination. Ninety percent of patients (range across sites, 83%-94%) had a mental health consult in the ED, 5% were restrained (range, 3%-9%), and 35% had a laboratory test performed (range, 15%-63%). Mean (SD) ED length of stay was 5.1 (5.4) hours, and 52% were admitted (93% to a psychiatric bed, including transfers to separate psychiatric facilities).Children with psychiatric-related visits seem to require substantial ED resources. Interventions are needed to reduce the burden on the ED by increasing the linkage to mental health services, particularly for suicidal youths.

The selectivities of neurons in primary visual cortex are often considered to be adapted to the statistics of natural images. Accordingly, simple cell-like tuning emerges when unsupervised learning models that seek sparse representations of input probabilities are trained on natural scenes. However, orientation tuning develops before structured vision starts, rendering these previous results moot as models of activity-dependent development. A more stringent examination of such models comes from experiments demonstrating altered neural response properties in goggle-reared kittens. We show that an unsupervised learning model of cortical responsivity accounts well for the dramatic effects of stimulus driven development during goggle-rearing.

No sensory stimulus is an island unto itself; rather, it can only properly be interpreted in light of the stimuli that surround it in space and time. This can result in entertaining illusions and puzzling results in psychological and neurophysiological experiments. We concentrate on perhaps the best studied test case, namely orientation or tilt, which gives rise to the notorious tilt illusion and the adaptation tilt after-effect. We review the empirical literature and discuss the computational and statistical ideas that are battling to explain these conundrums, and thereby gain favour as more general accounts of cortical processing.

The recently developed ‘two-step’ behavioural task promises to differentiate model-based from model-free reinforcement learning, while generating neurophysiologically-friendly decision datasets with parametric variation of decision variables. These desirable features have prompted its widespread adoption. Here, we analyse the interactions between a range of different strategies and the structure of transitions and outcomes in order to examine constraints on what can be learned from behavioural performance. The task involves a trade-off between the need for stochasticity, to allow strategies to be discriminated, and a need for determinism, so that it is worth subjects’ investment of effort to exploit the contingencies optimally. We show through simulation that under certain conditions model-free strategies can masquerade as being model-based. We first show that seemingly innocuous modifications to the task structure can induce correlations between action values at the start of the trial and the subsequent trial events in such a way that analysis based on comparing successive trials can lead to erroneous conclusions. We confirm the power of a suggested correction to the analysis that can alleviate this problem. We then consider model-free reinforcement learning strategies that exploit correlations between where rewards are obtained and which actions have high expected value. These generate behaviour that appears model-based under these, and also more sophisticated, analyses. Exploiting the full potential of the two-step task as a tool for behavioural neuroscience requires an understanding of these issues., Author Summary Planning is the use of a predictive model of the consequences of actions to guide decision making. Planning plays a critical role in human behaviour, but isolating its contribution is challenging because it is complemented by control systems which learn values of actions directly from the history of reinforcement, resulting in automatized mappings from states to actions often termed habits. Our study examined a recently developed behavioural task which uses choices in a multi-step decision tree to differentiate planning from value-based control. We compared various strategies using simulations, showing a range that produce behaviour that resembles planning but in fact arises as a fixed mapping from particular sorts of states to action. These results show that when a planning problem is faced repeatedly, sophisticated automatization strategies may be developed which identify that there are in fact a limited number of relevant states of the world each with an appropriate fixed or habitual response. Understanding such strategies is important for the design and interpretation of tasks which aim to isolate the contribution of planning to behaviour. Such strategies are also of independent scientific interest as they may contribute to automatization of behaviour in complex environments.

Reciprocating interactions represent a central feature of all human exchanges. They have been the target of various recent experiments, with healthy participants and psychiatric populations engaging as dyads in multi-round exchanges such as a repeated trust task. Behaviour in such exchanges involves complexities related to each agent’s preference for equity with their partner, beliefs about the partner’s appetite for equity, beliefs about the partner’s model of their partner, and so on. Agents may also plan different numbers of steps into the future. Providing a computationally precise account of the behaviour is an essential step towards understanding what underlies choices. A natural framework for this is that of an interactive partially observable Markov decision process (IPOMDP). However, the various complexities make IPOMDPs inordinately computationally challenging. Here, we show how to approximate the solution for the multi-round trust task using a variant of the Monte-Carlo tree search algorithm. We demonstrate that the algorithm is efficient and effective, and therefore can be used to invert observations of behavioural choices. We use generated behaviour to elucidate the richness and sophistication of interactive inference., Author Summary Agents interacting in games with multiple rounds must model their partner’s thought processes over extended time horizons. This poses a substantial computational challenge that has restricted previous behavioural analyses. By taking advantage of recent advances in algorithms for planning in the face of uncertainty, we demonstrate how these formal methods can be extended. We use a well studied social exchange game called the trust task to illustrate the power of our method, showing how agents with particular cognitive and social characteristics can be expected to interact, and how to infer the properties of individuals from observing their behaviour.

Roger Kowalski's poem ‘L'autre face’, published in 1960, was set to music by Jean-Claude Risset some two decades later in a piece with the same title for soprano and computer-generated tape. The poem is a powerfully self-reflexive piece, foregrounding its status as a work on paper, in which the voice of the poet becomes audible only through an imaginative act of interpretation. This relationship between paper, voice, and interpretation is fundamental to the aesthetic tradition which the poem belongs. Risset's soprano part, like classical music generally, is similarly a work on paper, awaiting a voice; but his computer-generated tape is not. Does this mean we cannot interpret it in the same way? This essay argues that Kowalski's work and Risset's, despite the latter's new medium, both belong to a high art tradition in which ghostly voices emerge from beyond the physical substance of the work—whatever that substance may be.

Humans routinely formulate plans in domains so complex that even the most powerful computers are taxed. To do so, they seem to avail themselves of many strategies and heuristics that efficiently simplify, approximate, and hierarchically decompose hard tasks into simpler subtasks. Theoretical and cognitive research has revealed several such strategies; however, little is known about their establishment, interaction, and efficiency. Here, we use model-based behavioral analysis to provide a detailed examination of the performance of human subjects in a moderately deep planning task. We find that subjects exploit the structure of the domain to establish subgoals in a way that achieves a nearly maximal reduction in the cost of computing values of choices, but then combine partial searches with greedy local steps to solve subtasks, and maladaptively prune the decision trees of subtasks in a reflexive manner upon encountering salient losses. Subjects come idiosyncratically to favor particular sequences of actions to achieve subgoals, creating novel complex actions or “options.”

Gaussian scale mixture models offer a top-down description of signal generation that captures key bottom-up statistical characteristics of filter responses to images. However, the pattern of dependence among the filters for this class of models is prespecified. We propose a novel extension to the gaussian scale mixturemodel that learns the pattern of dependence from observed inputs and thereby induces a hierarchical representation of these inputs. Specifically, we propose that inputs are generated by gaussian variables (modeling local filter structure), multiplied by a mixer variable that is assigned probabilistically to each input from a set of possible mixers. We demonstrate inference of both components of the generative model, for synthesized data and for different classes of natural images, such as a generic ensemble and faces. For natural images, the mixer variable assignments show invariances resembling those of complex cells in visual cortex; the statistics of the gaussian components of the model are in accord with the outputs of divisive normalization models. We also show how our model helps interrelate a wide range of models of image statistics and cortical processing.

The representation of hierarchically structured knowledge in systems using distributed patterns of activity is an abiding concern for the connectionist solution of cognitively rich problems. Here, we use statistical unsupervised learning to consider semantic aspects of structured knowledge representation. We meld unsupervised learning notions formulated for multilinear models with tensor product ideas for representing rich information. We apply the model to images of faces.

Solving in an efficient manner many different optimal control tasks within the same underlying environment requires decomposing the environment into its computationally elemental fragments. We suggest how to find fragmentations using unsupervised, mixture model, learning methods on data derived from optimal value functions for multiple tasks, and show that these fragmentations are in accord with observable structure in the environments. Further, we present evidence that such fragments can be of use in a practical reinforcement learning context, by facilitating online, actor-critic learning of multiple goals MDPs.

A venerable history of classical work on autoassociative memory has significantly shaped our understanding of several features of the hippocampus, and most prominently of its CA3 area, in relation to memory storage and retrieval. However, existing theories of hippocampal memory processing ignore a key biological constraint affecting memory storage in neural circuits: the bounded dynamical range of synapses. Recent treatments based on the notion of metaplasticity provide a powerful model for individual bounded synapses; however, their implications for the ability of the hippocampus to retrieve memories well and the dynamics of neurons associated with that retrieval are both unknown. Here, we develop a theoretical framework for memory storage and recall with bounded synapses. We formulate the recall of a previously stored pattern from a noisy recall cue and limited-capacity (and therefore lossy) synapses as a probabilistic inference problem, and derive neural dynamics that implement approximate inference algorithms to solve this problem efficiently. In particular, for binary synapses with metaplastic states, we demonstrate for the first time that memories can be efficiently read out with biologically plausible network dynamics that are completely constrained by the synaptic plasticity rule, and the statistics of the stored patterns and of the recall cue. Our theory organises into a coherent framework a wide range of existing data about the regulation of excitability, feedback inhibition, and network oscillations in area CA3, and makes novel and directly testable predictions that can guide future experiments., Author Summary Memory is central to nervous system function and has been a particular focus for studies of the hippocampus. However, despite many clues, we understand little about how memory storage and retrieval is implemented in neural circuits. In particular, while many previous studies considered the amount of information that can be stored in synaptic connections under biological constraints on the dynamic range of synapses, how much of this information can be successfully recovered by neural dynamics during memory retrieval remains unclear. Here, we use a top-down approach to address this question: we assume memories are laid down in bounded synapses by biologically relevant plasticity rules and then derive from first principles how the neural circuit should behave during recall in order to retrieve these memories most efficiently. We show that the resulting recall dynamics are consistent with a wide variety of properties of hippocampal area CA3, across a range of biophysical levels – from synapses, through neurons, to circuits. Furthermore, our approach allows us to make novel and experimentally testable predictions about the link between the structure, dynamics, and function of CA3 circuitry.

That familiarity should gate learning is a neurobiological and computational commonplace, and has been particularly investigated in the context of the hippocampus. We show its critical importance in continuous attractor models, using the formation of place fields in hippocampal area CA3 as an example. Without it, biased sampling of places in an environment, coming from purely random exploration, leads to degenerate attractors and prevents the formation of place fields. Conversely, appropriate use of familiarity information during learning can counteract the sampling bias, resulting in a normal place cell representation.

Bayesian planning is a formally elegant approach to learning optimal behaviour under model uncertainty, trading off exploration and exploitation in an ideal way. Unfortunately, planning optimally in the face of uncertainty is notoriously taxing, since the search space is enormous. In this paper we introduce a tractable, sample-based method for approximate Bayes-optimal planning which exploits Monte-Carlo tree search. Our approach avoids expensive applications of Bayes rule within the search tree by sampling models from current beliefs, and furthermore performs this sampling in a lazy manner. This enables it to outperform previous Bayesian model-based reinforcement learning algorithms by a significant margin on several well-known benchmark problems. As we show, our approach can even work in problems with an infinite state space that lie qualitatively out of reach of almost all previous work in Bayesian exploration.

It is well known that for finite-sized networks, onestep retrieval in the autoassociative Willshaw net is a suboptimal way to extract the information stored in the synapses. Iterative retrieval strategies are much better, but have hitherto only had heuristic justification. We show how they emerge naturally from considerations of probabilistic inference under conditions of noisy and partial input and a corrupted weight matrix. We start from the conditional probability distribution over possible patterns for retrieval. This contains all possible information that is available to an observer of the network and the initial input. Since this distribution is over exponentially many patterns, we use it to develop two approximate, but tractable, iterative retrieval methods. One performs maximum likelihood inference to find the single most likely pattern, using the (negative log of the) conditional probability as a Lyapunov function for retrieval. In physics terms, if storage errors are present, then the modified iterative update equations contain an additional antiferromagnetic interaction term and site dependent threshold values. The second method makes a mean field assumption to optimize a tractable estimate of the full conditional probability distribution. This leads to iterative mean field equations which can be interpreted in terms of a network of neurons with sigmoidal responses but with the same interactions and thresholds as in the maximum likelihood update equations. In the absence of storage errors, both models become very similiar to the Willshaw model, where standard retrieval is iterated using a particular form of linear threshold strategy.

Binocular rivalry is the alternating percept that can result when the two eyes see different scenes. Recent psychophysical evidence supports the notion that some aspects of binocular rivalry bear functional similarities to other bistable percepts. We build a model based on the hypothesis (Logothetis & Schall, 1989; Leopold & Logothetis, 1996; Logothetis, Leopold, & Sheinberg, 1996) that alternation can be generated by competition between top-down cortical explanations for the inputs, rather than by direct competition between the inputs. Recent neurophysiological evidence shows that some binocular neurons are modulated with the changing percept; others are not, even if they are selective between the stimuli presented to the eyes. We extend our model to a hierarchy to address these effects.

Ce texte de Sand pose d'evidents problemes de narratologie. L'A. se penche plus particulierement sur la question de l'identite du narrateur en essayant en particulier d'elucider les relations qui existent entre le narrateur extradiegetique des quatre premieres parties, et le narrateur intradiegetique de la conclusion

Highlights • Task irrelevant backgrounds of adapting and test stimuli influence tilt aftereffect. • Matched backgrounds enable transfer from second to first-order orientations. • Mismatched backgrounds reduce normally large, within-order aftereffects. • This background similarity effect is robust for weak foreground orientations. • Null aftereffect in many studies may be partially caused by a background mismatch., A wealth of studies has found that adapting to second-order visual stimuli has little effect on the perception of first-order stimuli. This is physiologically and psychologically troubling, since many cells show similar tuning to both classes of stimuli, and since adapting to first-order stimuli leads to aftereffects that do generalize to second-order stimuli. Focusing on high-level visual stimuli, we recently proposed the novel explanation that the lack of transfer arises partially from the characteristically different backgrounds of the two stimulus classes. Here, we consider the effect of stimulus backgrounds in the far more prevalent, lower-level, case of the orientation tilt aftereffect. Using a variety of first- and second-order oriented stimuli, we show that we could increase or decrease both within- and cross-class adaptation aftereffects by increasing or decreasing the similarity of the otherwise apparently uninteresting or irrelevant backgrounds of adapting and test patterns. Our results suggest that similarity between background statistics of the adapting and test stimuli contributes to low-level visual adaptation, and that these backgrounds are thus not discarded by visual processing but provide contextual modulation of adaptation. Null cross-adaptation aftereffects must also be interpreted cautiously. These findings reduce the apparent inconsistency between psychophysical and neurophysiological data about first- and second-order stimuli.

Receptive fields acquired through unsupervised learning of sparse representations of natural scenes have similar properties to primary visual cortex (V1) simple cell receptive fields. However, what drives in vivo development of receptive fields remains controversial. The strongest evidence for the importance of sensory experience in visual development comes from receptive field changes in animals reared with abnormal visual input. However, most sparse coding accounts have considered only normal visual input and the development of monocular receptive fields. Here, we applied three sparse coding models to binocular receptive field development across six abnormal rearing conditions. In every condition, the changes in receptive field properties previously observed experimentally were matched to a similar and highly faithful degree by all the models, suggesting that early sensory development can indeed be understood in terms of an impetus towards sparsity. As previously predicted in the literature, we found that asymmetries in inter-ocular correlation across orientations lead to orientation-specific binocular receptive fields. Finally we used our models to design a novel stimulus that, if present during rearing, is predicted by the sparsity principle to lead robustly to radically abnormal receptive fields., Author Summary The responses of neurons in the primary visual cortex (V1), a region of the brain involved in encoding visual input, are modified by the visual experience of the animal during development. For example, most neurons in animals reared viewing stripes of a particular orientation only respond to the orientation that the animal experienced. The responses of V1 cells in normal animals are similar to responses that simple optimisation algorithms can learn when trained on images. However, whether the similarity between these algorithms and V1 responses is merely coincidental has been unclear. Here, we used the results of a number of experiments where animals were reared with modified visual experience to test the explanatory power of three related optimisation algorithms. We did this by filtering the images for the algorithms in ways that mimicked the visual experience of the animals. This allowed us to show that the changes in V1 responses in experiment were consistent with the algorithms. This is evidence that the precepts of the algorithms, notably sparsity, can be used to understand the development of V1 responses. Further, we used our model to propose a novel rearing condition which we expect to have a dramatic effect on development.

Intrinsic motivation involves internally governed drives for exploration, curiosity, and play. These shape subjects over the course of development and beyond to explore to learn and expand the actions they are capable of performing and to acquire skills that can be useful in future domains. We adopt a utilitarian view of this learning process, treating it in terms of exploration bonuses that arise from distributions over the structure of the world that imply potential benefits from generalizing knowledge and skills to subsequent environments. We discuss how functionally and architecturally different controllers may realize these bonuses in different ways.

Finding the Bayesian balance between exploration and exploitation in adaptive optimal control is in general intractable. This paper shows how to compute suboptimal estimates based on a certainty equivalence approximation (Cozzolino, Gonzalez-Zubieta & Miller, 1965) arising from a form of dual control. This system-atizes and extends existing uses of exploration bonuses in reinforcement learning (Sutton, 1990). The approach has two components: a statistical model of uncertainty in the world and a way of turning this into exploratory behavior. This general approach is applied to two-dimensional mazes with moveable barriers and its performance is compared with Sutton's DYNA system.

An unsupervised learning algorithm for a multilayer network of stochastic neurons is described. Bottom-up "recognition" connections convert the input into representations in successive hidden layers, and top-down "generative" connections reconstruct the representation in one layer from the representation in the layer above. In the "wake" phase, neurons are driven by recognition connections, and generative connections are adapted to increase the probability that they would reconstruct the correct activity vector in the layer below. In the "sleep" phase, neurons are driven by generative connections, and recognition connections are adapted to increase the probability that they would produce the correct activity vector in the layer above.

If different causes can interact on any occasion to generate a set of patterns, then systems modeling the generation have to model the interaction too. We discuss a way of combining multiple causes that is based on the Integrated Segmentation and Recognition architecture of Keeler et al. (1991). It is more cooperative than the scheme embodied in the mixture of experts architecture, which insists that just one cause generate each output, and more competitive than the noisy-or combination function, which was recently suggested by Saund (1994a,b). Simulations confirm its efficacy.

v w c i m t w d s m d e l t T here is an odd irony associated with the by-now almost ineluctable tie between reinforcement learning’s temporal difference learning rule (1) and the phasic activity of dopamine eurons (2). Although temporal difference learning was designed o enable the acquisition of whole sequences of actions and predicions, a task that its ancestors (3) would flub, there are very few irect tests of this characteristic. In a penetrating new study, Wasum and colleagues (4) measured and manipulated dopamine in a equence learning task for sucrose reward, revealing four correlates f the neuromodulator: its fineand gross-scale dynamics during earning, its involvement in two aspects of energizing behavior, and ts necessity for learning. The task is straightforward. Rats were placed in an operant hamber in which one (distal) lever was freely available to be ressed at any time. A response on this lever caused a second proximal; mildly pretrained) lever to emerge, and when this was ressed, a sucrose pellet was delivered to a food well and the roximal lever was retracted. Over the course of training, the anials learned to make increasingly efficient sequential responses: istal press, proximal press, and consumption. To determine how dopamine transmission is modulated during he development of this behavior, the authors used fast-scan cyclic oltammetry (FSCV) to measure changes in dopamine concentraion in the ventral striatum. A strength of FSCV is that it allows eliable trial-by-trial monitoring of phasic dopamine transmission n individual animals. Wassum and colleagues used this to their dvantage to demonstrate the dynamic modulation of dopamine s the animals improved their efficiency on the sequence. They sed FSCV in three groups of animals. For one group (labeled day 0), easurements were made on the very first day they were introuced to the sequence task. For the second and third groups, FSCV as performed after moderate (day 5) or extensive (day 10) trainng. Furthermore, in a separate experiment, the authors used the onspecific dopamine antagonist flupenthixol systemically to esablish whether dopamine release was required for the appropriate equence of actions to be learned and performed. Treating this as an instrumental learning task (albeit with some mportant Pavlovian elements, such as the sound of the proximal ever extending or retracting), we might expect phasic dopamine to eport an appetitively signed temporal difference prediction error. t least to a first approximation, this would imply that early in earning, dopamine release would increase to the unexpected reard following the proximal press; but late in learning this release ould be absent because the reward would have been well preicted and so inspire no prediction error. Nonetheless, even at the nd of training, unexpected, unearned rewards should still lead to opamine release. This is exactly what was observed and is an mportant novel finding because past FSCV studies of free operant nstrumental conditioning have used cocaine as a reward (5). The

Adaptation aftereffects offer a critical window onto sensory processing in the brain. However, such sensory processing is hierarchical, progressing from the extraction of simple features to the representation of complex patterns. The way that adaptation depends on coordinated changes across different levels of the hierarchy has been studied. However, when a given adapting stimulus produces both a low- and a high-level aftereffect, it remains unclear whether the high-level aftereffect is a passive reflection of low-level adaptation, or whether it is generated, at least partially, de novo in high-level areas. We assembled the two key ingredients needed for investigating this question psychophysically. One ingredient involves perceptual tasks that depend rather exclusively on low or high levels of processing, and yet involve partially identical stimuli that inspire cross-level adaptation. For this, we considered the discrimination of curvature or facial expression using curves or cartoon faces. The other ingredient is spatial or temporal stimulus manipulations that limit adaptation to either low or high levels. For this, we used crowding and brief presentations. We found that crowding an adapting curve with flanking curves reduces the curvature aftereffect much more than the facial-expression aftereffect, and vice versa for crowding the adapting face with flanking faces. Additionally, reducing adaptation time to a cartoon face diminishes the curvature aftereffect more drastically than the facial-expression aftereffect. These results suggest that high-level aftereffects, even when generated by a low-level adaptor, are not completely inherited from lower levels, and offer a window into the determining factors.

Spatial context in images induces perceptual phenomena associated with salience and modulates the responses of neurons in primary visual cortex (V1). However, the computational and ecological principles underlying contextual effects are incompletely understood. We introduce a model of natural images that includes grouping and segmentation of neighboring features based on their joint statistics, and we interpret the firing rates of V1 neurons as performing optimal recognition in this model. We show that this leads to a substantial generalization of divisive normalization, a computation that is ubiquitous in many neural areas and systems. A main novelty in our model is that the influence of the context on a target stimulus is determined by their degree of statistical dependence. We optimized the parameters of the model on natural image patches, and then simulated neural and perceptual responses on stimuli used in classical experiments. The model reproduces some rich and complex response patterns observed in V1, such as the contrast dependence, orientation tuning and spatial asymmetry of surround suppression, while also allowing for surround facilitation under conditions of weak stimulation. It also mimics the perceptual salience produced by simple displays, and leads to readily testable predictions. Our results provide a principled account of orientation-based contextual modulation in early vision and its sensitivity to the homogeneity and spatial arrangement of inputs, and lends statistical support to the theory that V1 computes visual salience., Author Summary One of the most important and enduring hypotheses about the way that mammalian brains process sensory information is that they are exquisitely attuned to the statistical structure of the natural world. This allows them to come, over the course of development, to represent inputs in a way that reflects the facets of the environment that were responsible. We focus on the case of information about the local orientation of visual input, a basic level feature for which a wealth of phenomenological observations are available to constrain and validate computational models. We suggest a new account which focuses on the statistics of orientations at nearby locations in visual space, and captures data on how such contextual information modulates both the responses of neurons in the primary visual cortex, and the corresponding psychophysical percepts. Our approach thus helps elucidate the computational and ecological principles underlying contextual processing in early vision; provides a number of predictions that are readily testable with existing experimental approaches; and indicates a possible route for examining whether similar computational principles and operations also support higher-level visual functions.

When planning a series of actions, it is usually infeasible to consider all potential future sequences; instead, one must prune the decision tree. Provably optimal pruning is, however, still computationally ruinous and the specific approximations humans employ remain unknown. We designed a new sequential reinforcement-based task and showed that human subjects adopted a simple pruning strategy: during mental evaluation of a sequence of choices, they curtailed any further evaluation of a sequence as soon as they encountered a large loss. This pruning strategy was Pavlovian: it was reflexively evoked by large losses and persisted even when overwhelmingly counterproductive. It was also evident above and beyond loss aversion. We found that the tendency towards Pavlovian pruning was selectively predicted by the degree to which subjects exhibited sub-clinical mood disturbance, in accordance with theories that ascribe Pavlovian behavioural inhibition, via serotonin, a role in mood disorders. We conclude that Pavlovian behavioural inhibition shapes highly flexible, goal-directed choices in a manner that may be important for theories of decision-making in mood disorders., Author Summary Planning is tricky because choices we make now affect future choices, and future choices and outcomes should guide current choices. Because there are exponentially many combinations of future choices and actions, brute-force approaches that consider all possible combinations work only for trivially small problems. Here, we describe how humans use a simple Pavlovian strategy to cut an expanding decision tree down to a computationally manageable size. We find that humans use this strategy even when it is disadvantageous, and that the tendency to use it is related to mild depressive symptoms. The findings, we suggest, can be interpreted within a theoretical framework which relates Pavlovian behavioural inhibition to serotonin and mood disorders.

Complexities in the relationship between value and choice are two central sources of anomalies. First, the different systems can disagree about their values. Actions involve such things as picking a stimulus or pressing a button. The environment specifies a set of rules governing the transitions between states depending on the action chosen. The trouble is that a tree typically grows exponentially with the number of layers considered, making this extremely difficult. Model-based and model-free controls are ways of doing this, which differ in the information about the environment they use and the computations they perform. The model-free system would learn the utility of pressing the lever but would not have the informational wherewithal to realize that this utility had changed when the cheese had been poisoned. Pavlovian control is also based on predictions of affectively important outcomes such as rewards and punishments. However, rather than determining the choices that would lead to the acquisition or avoidance of these outcomes, it expresses a set of hard-wired preparatory and consummatory choices. © 2012 Elsevier Inc. All rights reserved.

Chemotaxis plays a crucial role in many biological processes, including nervous system development. However, fundamental physical constraints limit the ability of a small sensing device such as a cell or growth cone to detect an external chemical gradient. One of these is the stochastic nature of receptor binding, leading to a constantly fluctuating binding pattern across the cell's array of receptors. This is analogous to the uncertainty in sensory information often encountered by the brain at the systems level. Here we derive analytically the Bayes-optimal strategy for combining information from a spatial array of receptors in both one and two dimensions to determine gradient direction. We also show how information from more than one receptor species can be optimally integrated, derive the gradient shapes that are optimal for guiding cells or growth cones over the longest possible distances, and illustrate that polarized cell behavior might arise as an adaptation to slowly varying environments. Together our results provide closed-form predictions for variations in chemotactic performance over a wide range of gradient conditions.

Estimation of returns over time, the focus of temporal difference (TD) algorithms, imposes particular constraints on good function approximators or representations. Appropriate generalization between states is determined by how similar their successors are, and representations should follow suit. This paper shows how TD machinery can be used to learn such representations, and illustrates, using a navigation task, the appropriately distributed nature of the result.