Search

Reward-based learning and decision-making are prime candidates to understand symptoms of attention deficit hyperactivity disorder (ADHD). However, only limited evidence is available regarding the neurocomputational underpinnings of the alterations seen in ADHD. This particularly concerns the flexible behavioral adaption in dynamically changing environments, which is challenging for individuals with ADHD. One previous study points to elevated choice switching in adolescent ADHD, which was accompanied by disrupted learning signals in medial prefrontal cortex.In the present study, we investigated young adults with ADHD (n=17, 18-32 years) and age and sex matched controls (n=17, 18-30 years) using a probabilistic reversal learning experiment during functional magnetic resonance imaging (fMRI). The task requires continuous learning to guide flexible behavioral adaptation to changing reward contingencies. To disentangle the neurocomputational underpinnings of the behavioral data, we used detailed reinforcement learning (RL) models, which informed the analysis of fMRI data.ADHD patients performed worse than controls particularly in trials before reversals, i.e., when reward contingencies were stable. This pattern resulted from ‘noisy’ choice switching regardless of previous feedback. RL modelling showed decreased reinforcement sensitivity and enhanced learning rates for negative feedback in ADHD patients. At the neural level, this was reflected in diminished representation of choice probability in the left posterior parietal cortex in ADHD. Moreover, modelling showed a marginal reduction of learning about the unchosen option, which was paralleled by an equally marginal reduction in learning signals incorporating the unchosen option in the left ventral striatum.Taken together, we show that flexible behavioral adaptation in the context of dynamically changing reward contingencies is impaired in ADHD. This is due to excessive choice switching (‘hyper-flexibility’), which can be detrimental or beneficial depending on the learning environment. Computationally, this results from blunted sensitivity to reinforcement. We detected neural correlates of this blunted sensitivity to reinforcement in the attention-control network, specifically in the parietal cortex. These neurocomputational findings are promising but remain preliminary due to the relatively small sample size.

Alcohol use disorder (AUD) is characterized by a combination of symptoms including excessive craving, loss of control, and progressive neglect of alternative pleasures. A mechanistic understanding of what drives these symptoms is needed to improve diagnostic stratification and to develop new treatment and prevention strategies for AUD. To date, there is no consensus regarding a unifying mechanistic framework that accounts for the different symptoms of AUD. Reinforcement learning (RL) and economic choice theories may be key to elucidating the underlying processes of symptom development and maintenance in AUD. These algorithms may account for the different behavioral and physiological phenomena and are suited to dissect mechanisms linked to different symptoms of AUD. We here review different RL and economic choice models and how they map onto three symptoms of AUD: (1) cue-induced craving, (2) neglect of alternative rewards, and (3) consumption despite adverse consequences. For each symptom and theory, we describe findings from animal and human studies. In humans, we focus on empirical studies that investigated RL models in the context of treatment outcome in AUD. The review indicates important gaps to be addressed in the future by highlighting the challenges in transferring findings from RL and economic choice studies to clinical application. We also critically evaluate the potential and pitfalls of a symptom-oriented approach and highlight the importance of elucidating the role of learning and decision-making processes across diagnostic boundaries.

Introduction: Over the last decades, our understanding of the cognitive, motivational, and neural processes involved in addictive behavior has increased enormously. A plethora of laboratory-based and cross-sectional studies has linked cognitive-behavioral measures to between-subject differences in drinking behavior. However, such laboratory-based studies inevitably suffer from small sample sizes and the inability to link temporal fluctuations in task measures to fluctuations in real-life substance use. To overcome these problems, several existing behavioral tasks have been transferred to smartphones to allow studying cognition in the field. Method: In this narrative review, we first summarize studies that used existing behavioral tasks in the laboratory and self-reports of substance use with ecological momentary assessment (EMA) in the field. Next, we review studies on psychometric properties of smartphone-based behavioral tasks. Finally, we review studies that used both smartphone-based tasks and self-reports with EMA in the field. Results: Overall, studies were scarce and heterogenous both in tasks and in study outcomes. Nevertheless, existing findings are promising and point toward several methodological recommendations: concerning psychometrics, studies show that – although more systematic studies are necessary – task validity and reliability can be improved, for example, by analyzing several measurement sessions at once rather than analyzing sessions separately. Studies that use tasks in the field, moreover, show that power can be improved by choosing sampling schemes that combine time-based with event-based sampling, rather than relying on time-based sampling alone. Increasing sampling frequency can further increase power. However, as this also increases the burden to participants, more research is necessary to determine the ideal sampling frequency for each task. Conclusion: Although more research is necessary to systematically study both the psychometrics of smartphone-based tasks and the frequency at which task measures fluctuate, existing studies are promising and reveal important methodological recommendations useful for researchers interested in implementing behavioral tasks in EMA studies.

Background and Hypothesis Current frameworks propose that delusions result from aberrant belief updating due to altered prediction error (PE) signaling and misestimation of environmental volatility. We aimed to investigate whether behavioral and neural signatures of belief updating are specifically related to the presence of delusions or generally associated with manifest schizophrenia. Methods Our cross-sectional design includes human participants (n[female/male] = 66[25/41]), stratified into four groups: healthy participants with minimal (n = 22) or strong delusional-like ideation (n = 18), and participants with diagnosed schizophrenia with minimal (n = 13) or strong delusions (n = 13), resulting in a 2 × 2 design, which allows to test for the effects of delusion and diagnosis. Participants performed a reversal learning task with stable and volatile task contingencies during fMRI scanning. We formalized learning with a hierarchical Gaussian filter model and conducted model-based fMRI analysis regarding beliefs of outcome uncertainty and volatility, precision-weighted PEs of the outcome- and the volatility-belief. Results Patients with schizophrenia as compared to healthy controls showed lower accuracy and heightened choice switching, while delusional ideation did not affect these measures. Participants with delusions showed increased precision-weighted PE-related neural activation in fronto-striatal regions. People with diagnosed schizophrenia overestimated environmental volatility and showed an attenuated neural representation of volatility in the anterior insula, medial frontal and angular gyrus. Conclusions Delusional beliefs are associated with altered striatal PE-signals. Juxtaposing, the potentially unsettling belief that the environment is constantly changing and weaker neural encoding of this subjective volatility seems to be associated with manifest schizophrenia, but not with the presence of delusional ideation.

Self-regulation, the ability to guide behavior according to one’s goals, plays an integral role in understanding loss of control behaviors a pertinent example being substance use disorders (SUD). Yet, experimental tasks that measure processes underlying self-regulation are not easy to deploy in contexts where such behaviors often occur, namely in real life situations outside the laboratory. Moreover, lab-based experimental tasks are criticized for poor test–retest reliability and a lack of construct validity. These concerns might in part explain why ecological validity of experimental measures—their ability to predict real-life behavior—is low. To address these shortcomings, we assessed the reliability and construct validity of four smartphone-based experimental tasks designed to measure cognitive control and decision-making. To facilitate future clinical applicability we recruited a large (N=488) sample of individuals with SUD. Joint modeling of measurement sessions increased the reliability of task measures from moderate to good and often excellent levels. In line with theories of cognitive control and motivation, three latent factors reflecting cognitive control and decision-making in the context of gains and losses best described the data. As proof of concept, we show that a latent cognitive control score based on joint modeling, yielded stronger correlations with drinking behavior than single task scores based on separate modeling. These findings indicate that in individuals with SUD, smartphone-based ambulatory experimental assessments can reliably index functions of cognitive control and decision-making, with plausible construct validity. Our findings provide evidence for rich possibilities arising from longitudinal experimental studies in SUD as well as in psychiatry, neuroscience, and psychology more generally.

Child maltreatment gives rise to atypical patterns of social functioning with peers which might be particularly pronounced in early adolescence when peer influence typically peaks. Yet, few neuroimaging studies in adolescents use peer interaction paradigms to parse neural correlates of distinct maltreatment exposures. This fMRI study examines effects of abuse, neglect, and emotional maltreatment (EM) among 98 youth (n = 58 maltreated; n = 40 matched controls) using an event-related Cyberball paradigm affording assessment of both social exclusion and inclusion across early and mid-adolescence (≤13.5 years, n = 50; >13.5 years, n = 48). Younger adolescents showed increased activation to social exclusion versus inclusion in regions implicated in mentalizing (e.g., superior temporal gyrus). Individual exposure-specific analyses suggested that neglect and EM coincided with less reduction of activation to social exclusion relative to inclusion in the dorsal anterior cingulate cortex/pre-supplementary motor area (dACC/pre-SMA) among younger versus older adolescents. Integrative follow-up analyses showed that EM accounted for this dACC/pre-SMA activation pattern over and above other exposures. Moreover, age-independent results within respective exposure groups revealed that greater magnitude of neglect predicted blunted exclusion-related activity in the parahippocampal gyrus, while EM predicted increased activation to social exclusion in the precuneus/posterior cingulate cortex.

Task-based measures that capture neurocognitive processes can help bridge the gap between brain and behavior. To transfer tasks to clinical application, reliability is a crucial benchmark because it imposes an upper bound to potential correlations with other variables (e.g., symptom or brain data). However, the reliability of many task readouts is low. In this study, we scrutinized the retest reliability of a probabilistic reversal learning task (PRLT) that is frequently used to characterize cognitive flexibility in psychiatric populations. We analyzed data from N = 40 healthy subjects, who completed the PRLT twice. We focused on how individual metrics are derived, i.e., whether data were partially pooled across participants and whether priors were used to inform estimates. We compared the reliability of the resulting indices across sessions, as well as the internal consistency of a selection of indices. We found good to excellent reliability for behavioral indices as derived from mixed-effects models that included data from both sessions. The internal consistency was good to excellent. For indices derived from computational modeling, we found excellent reliability when using hierarchical estimation with empirical priors and including data from both sessions. Our results indicate that the PRLT is well equipped to measure individual differences in cognitive flexibility in reinforcement learning. However, this depends heavily on hierarchical modeling of the longitudinal data (whether sessions are modeled separately or jointly), on estimation methods, and on the combination of parameters included in computational models. We discuss implications for the applicability of PRLT indices in psychiatric research and as diagnostic tools.

Re-directing attention to objects in working memory can enhance their representational fidelity. However, how this attentional enhancement of memory representations is implemented across distinct, sensory and cognitive-control brain network is unspecified. The present fMRI experiment leverages psychophysical modelling and multivariate auditory- pattern decoding as behavioral and neural proxies of mnemonic fidelity. Listeners performed an auditory syllable pitch-discrimination task and received retro-active cues to selectively attend to a to-be-probed syllable in memory. Accompanied by increased neural activation in fronto-parietal and cingulo-opercular networks, valid retro-cues yielded faster and more perceptually sensitive responses in recalling acoustic detail of memorized syllables. Information about the cued auditory object was decodable from hemodynamic response patterns in superior temporal sulcus (STS), fronto-parietal, and sensorimotor regions. However, among these regions retaining auditory memory objects, neural fidelity in the left STS and its enhancement through attention-to-memory best predicted individuals’ gain in auditory memory recall precision. Our results demonstrate how functionally discrete brain regions differentially contribute to the attentional enhancement of memory representations.

Physical and mental well-being during the COVID-19 pandemic is typically assessed via surveys, which might make it difficult to conduct longitudinal studies and might lead to data suffering from recall bias. Ecological momentary assessment (EMA) driven smartphone apps can help alleviate such issues, allowing for in situ recordings. Implementing such an app is not trivial, necessitates strict regulatory and legal requirements, and requires short development cycles to appropriately react to abrupt changes in the pandemic. Based on an existing app framework, we developed Corona Health, an app that serves as a platform for deploying questionnaire-based studies in combination with recordings of mobile sensors. In this paper, we present the technical details of Corona Health and provide first insights into the collected data. Through collaborative efforts from experts from public health, medicine, psychology, and computer science, we released Corona Health publicly on Google Play and the Apple App Store (in July 2020) in eight languages and attracted 7290 installations so far. Currently, five studies related to physical and mental well-being are deployed and 17,241 questionnaires have been filled out. Corona Health proves to be a viable tool for conducting research related to the COVID-19 pandemic and can serve as a blueprint for future EMA-based studies. The data we collected will substantially improve our knowledge on mental and physical health states, traits and trajectories as well as its risk and protective factors over the course of the COVID-19 pandemic and its diverse prevention measures., publishedVersion

BackgroundMethylphenidate (MPH) is the first-line pharmacological treatment of attention-deficit/hyperactivity disorder (ADHD). MPH binds to the dopamine (DA) transporter (DAT), which has high density in the striatum. Assessments of the striatal dopamine transporter by single positron emission computed tomography (SPECT) in childhood and adolescent patients are rare but can provide insight in how effects of MPH affect DAT availability. The aim of our within-subject study was to investigate the effect of MPH on DAT availability and how responsivity to MPH in DAT availability is linked to clinical symptoms and cognitive functioning.MethodsThirteen adolescent male patients (9-16 years) with diagnosis of ADHD according to DSM-IV and long-term stimulant medication (for at least 6 months) with MPH were assessed twice within 7 days using SPECT after application of I-123-ß-CIT to examine DAT binding potential (DAT BP). SPECT measures took place in on and off-MPH status balanced for order across participants. A virtual-reality continuous-performance test was performed at each time point. Further clinical symptoms were assessed for baseline off-MPH.ResultsOn-MPH status was associated with a highly significant decrease (−27,6%) of striatal DAT BP as compared to off-MPH (t=4.93, pConclusionOur findings corroborate previous reports from mainly adult samples that MPH reduces striatal DAT BP availability and suggest higher off-MPH DAT BP, likely reflecting low baseline DA levels, as a marker of symptom severity. More speculatively, regional specific responsivity of DAT BP to MPH may reflect treatment response with respect to cognitive functioning. However, implications from this small patient sample should be treated with caution and warrant replication.

Consuming more energy than is expended may reflect a failure of control over eating behaviour in obesity. Behavioural control arises from a balance between two dissociable strategies of reinforcement learning: model-free and model-based. We hypothesized that weight status relates to an imbalance in reliance on model-based and model-free control, and that it may do so in a linear or quadratic manner. To test this, 90 healthy participants in a wide BMI range (normal-weight (n=31), overweight (n=29), obese (n=30)) performed a sequential decision-making task. The primary analysis indicated that obese participants relied less on model-based control than overweight and normal-weight participants, with no difference between overweight and normal-weight participants. In line, secondary continuous analyses revealed a negative linear, but not quadratic, relationship between BMI and model-based control. Computational modelling of choice behaviour suggested that a mixture of both strategies was shifted towards less model-based control in obese participants. Furthermore, exploratory analyses of separate weights for model-free and model-based control showed stronger reliance on model-free control with increased BMI. Our findings suggest that obesity may indeed be related to an imbalance in behavioural control as expressed in a phenotype of less model-based control potentially resulting from enhanced reliance on model-free computations.

Individuals differ in how they learn from experience. In Pavlovian conditioning models, where cues predict reinforcer delivery at a different goal location, some animals-called sign-trackers-come to approach the cue, whereas others, called goal-trackers, approach the goal. In sign-trackers, model-free phasic dopaminergic reward-prediction errors underlie learning, which renders stimuli 'wanted'. Goal-trackers do not rely on dopamine for learning and are thought to use model-based learning. We demonstrate this double dissociation in 129 male humans using eye-tracking, pupillometry and functional magnetic resonance imaging informed by computational models of sign- and goal-tracking. We show that sign-trackers exhibit a neural reward prediction error signal that is not detectable in goal-trackers. Model-free value only guides gaze and pupil dilation in sign-trackers. Goal-trackers instead exhibit a stronger model-based neural state prediction error signal. This model-based construct determines gaze and pupil dilation more in goal-trackers.

One of the major risk factors for global death and disability is alcohol, tobacco, and illicit drug use. While there is increasing knowledge with respect to individual factors promoting the initiation and maintenance of substance use disorders (SUDs), disease trajectories involved in losing and regaining control over drug intake (ReCoDe) are still not well described. Our newly formed German Collaborative Research Centre (CRC) on ReCoDe has an interdisciplinary approach funded by the German Research Foundation (DFG) with a 12-year perspective. The main goals of our research consortium are (i) to identify triggers and modifying factors that longitudinally modulate the trajectories of losing and regaining control over drug consumption in real life, (ii) to study underlying behavioral, cognitive, and neurobiological mechanisms, and (iii) to implicate mechanism-based interventions. These goals will be achieved by: (i) using mobile health (m-health) tools to longitudinally monitor the effects of triggers (drug cues, stressors, and priming doses) and modify factors (eg, age, gender, physical activity, and cognitive control) on drug consumption patterns in real-life conditions and in animal models of addiction; (ii) the identification and computational modeling of key mechanisms mediating the effects of such triggers and modifying factors on goal-directed, habitual, and compulsive aspects of behavior from human studies and animal models; and (iii) developing and testing interventions that specifically target the underlying mechanisms for regaining control over drug intake.

BackgroundReward-based decision-making is impaired in patients with schizophrenia (PSZ) as reflected by increased choice switching. The underlying cognitive and motivational processes as well as associated neural signatures remain unknown. Reinforcement Learning (RL) and hierarchical Bayesian learning account for choice switching in different ways. We hypothesized that enhanced choice switching, as seen in PSZ during reward-based decision-making, relates to higher-order beliefs about environmental volatility and examined the associated neural activity.Methods46 medicated PSZ and 43 healthy controls (HC) performed a reward-based decision-making task requiring flexible responses to changing action-outcome contingencies during functional Magnetic Resonance Imaging (fMRI). Detailed computational modeling of choice data was performed, including RL and the hierarchical Gaussian filter (HGF). Trajectories of learning from computational modeling informed the analysis of fMRI data.ResultsA three-level HGF accounted best for the observed choice data. This model revealed a heightened initial belief about environmental volatility and a stronger influence of volatility on lower-level learning of action-outcome contingencies in PSZ as compared to HC. This was replicated in an independent sample of non-medicated PSZ. Beliefs about environmental volatility were reflected by higher activity in dorsolateral prefrontal cortex of PSZ as compared to HC.ConclusionsOur study suggests that PSZ inferred the environment as overly volatile, which may explain increased choice switching. In PSZ, activity in dorsolateral prefrontal cortex was more strongly related to beliefs about environmental volatility. Our computational phenotyping approach may provide useful information to dissect clinical heterogeneity and could improve prediction of outcome.

Reinforcement learning involves a balance between model-free (MF) and model-based (MB) systems. Recent studies suggest that individuals with either pharmacologically enhanced levels of dopamine (DA) or higher baseline levels of DA exhibit more MB control. However, it remains unknown whether such pharmacological effects depend on baseline DA.Here, we investigated whether effects of L-DOPA on the balance of MB/MF control depend on ventral striatal baseline DA. Sixty participants had two functional magnetic resonance imaging (fMRI) scans while performing a two-stage sequential decision-making task under 150 mg L-DOPA or placebo (counterbalanced), followed by a 4-hour 18F-DOPA positron emission tomography (PET) scan (on a separate occasion).We found an interaction between baseline DA levels and L-DOPA induced changes in MB control. Individuals with higher baseline DA levels showed a greater L-DOPA induced enhancement in MB control. Surprisingly, we found a corresponding drug-by-baseline DA interaction on MF, but not MB learning signals in the ventromedial prefrontal cortex. We did not find a significant interaction between baseline DA levels and L-DOPA effects on MF control or MB/MF balance.In sum, our findings point to a baseline dependency of L-DOPA effects on differential aspects of MB and MF control. Individual differences in DA washout may be an important moderator of L-DOPA effects. Overall, our findings complement the general notion where higher DA levels is related to a greater reliance on MB control. Although the relationship between phasic DA firing and MF learning is conventionally assumed in the animal literature, the relationship between DA and MF control is not as straightforward and requires further clarification.

Dopamine underlies important aspects of cognition, and has been suggested to boost cognitive performance. However, how dopamine modulates the large-scale cortical dynamics during cognitive performance has remained elusive. Using functional MRI during a working memory task in healthy young human listeners (N=22), we investigated the effect of levodopa (L-dopa) on two aspects of cortical dynamics, blood oxygen-level-dependent (BOLD) signal variability and the functional connectome of large-scale cortical networks. We here show that enhanced dopaminergic signaling modulates the two potentially interrelated aspects of large-scale cortical dynamics during cognitive performance, and the degree of these modulations is able to explain inter-individual differences in L-dopa-induced behavioral benefits. Relative to placebo, L-dopa increased BOLD signal variability in task-relevant temporal, inferior frontal, parietal and cingulate regions. On the connectome level, however, L-dopa diminished functional integration across temporal and cingulo-opercular regions. This hypo-integration was expressed as a reduction in network efficiency and modularity in more than two thirds of the participants and to different degrees. Hypo-integration co-occurred with relative hyper-connectivity in paracentral lobule and precuneus, as well as posterior putamen. Both, L-dopa-induced BOLD signal variability modulation and functional connectome modulations proved predictive of an individual’s L-dopa-induced gain in behavioral performance, namely response speed and perceptual sensitivity. Lastly, L-dopa-induced modulations of BOLD signal variability were correlated with L-dopa-induced modulation of nodal connectivity and network efficiency. Our findings underline the role of dopamine in maintaining the dynamic range of, and communication between, cortical systems, and their explanatory power for inter-individual differences in benefits from dopamine during cognitive performance.

Theoretical models and preceding studies have described age-related alterations in neuronal activation of frontoparietal regions in a working memory (WM) load-dependent manner. However, to date, underlying neuronal mechanisms of these WM load-dependent activation changes in aging remain poorly understood. The aim of this study was to investigate these mechanisms in terms of effective connectivity by application of dynamic causal modeling with Bayesian Model Selection. Eighteen healthy younger (age: 20-32 years) and 32 older (60-75 years) participants performed an n-back task with 3 WM load levels during functional magnetic resonance imaging (fMRI). Behavioral and conventional fMRI results replicated age group by WM load interactions. Importantly, the analysis of effective connectivity derived from dynamic causal modeling, indicated an age- and performance-related reduction in WM load-dependent modulation of connectivity from dorsolateral prefrontal cortex to inferior parietal lobule. This finding provides evidence for the proposal that age-related WM decline manifests as deficient WM load-dependent modulation of neuronal top-down control and can integrate implications from theoretical models and previous studies of functional changes in the aging brain.

Schizophrenia is a heterogeneous spectrum disorder often associated with detrimental negative symptoms. In recent years, computational approaches to psychiatry have attracted growing attention. Negative symptoms have shown some overlap with general cognitive impairments and were also linked to impaired motivational processing in brain circuits implementing reward prediction. In this review, we outline how computational approaches may help to provide a better understanding of negative symptoms in terms of the potentially underlying behavioural and biological mechanisms. First, we describe the idea that negative symptoms could arise from a failure to represent reward expectations to enable flexible behavioural adaptation. It has been proposed that these impairments arise from a failure to use prediction errors to update expectations. Important previous studies focused on processing of so-called model-free prediction errors where learning is determined by past rewards only. However, learning and decision-making arise from multiple cognitive mechanisms functioning simultaneously, and dissecting them via well-designed tasks in conjunction with computational modelling is a promising avenue. Second, we move on to a proof-of-concept example on how generative models of functional imaging data from a cognitive task enable the identification of subgroups of patients mapping on different levels of negative symptoms. Combining the latter approach with behavioural studies regarding learning and decision-making may allow the identification of key behavioural and biological parameters distinctive for different dimensions of negative symptoms versus a general cognitive impairment. We conclude with an outlook on how this computational framework could, at some point, enrich future clinical studies.

Background: Working-memory impairment is a core cognitive dysfunction in people with schizophrenia and people at mental high risk. Recent imaging studies on working memory have suggested that abnormalities in prefrontal activation and in connectivity between the frontal and parietal regions could be neural underpinnings of the different stages of psychosis. However, it remains to be explored whether comparable alterations are present in people with subclinical levels of psychosis, as experienced by a small proportion of the general population who neither seek help nor show constraints in daily functioning.Methods: We compared 24 people with subclinical high delusional ideation and 24 people with low delusional ideation. Both groups performed an n-back working-memory task during functional magnetic resonance imaging. We characterized frontoparietal effective connectivity using dynamic causal modelling.Results: Compared to people who had low delusional ideation, people with high delusional ideation showed a significant increase in dorsolateral prefrontal activation during the working-memory task, as well as reduced working-memory-dependent parietofrontal effective connectivity in the left hemisphere. Group differences were not evident at the behavioural level.Limitations: The current experimental design did not distinguish among the working-memory subprocesses; it remains unexplored whether differences in connectivity exist at that level.Conclusion: These findings suggest that alterations in the working-memory network are also present in a nonclinical population with psychotic experiences who do not display cognitive deficits. They also suggest that alterations in working-memory-dependent connectivity show a putative continuity along the spectrum of psychotic symptoms.

Dopamine potentially unites two important roles: one in addiction, being involved in most substances of abuse including alcohol, and a second one in a specific type of learning, namely model-free temporal-difference reinforcement learning. Theories of addiction have long suggested that drugs of abuse may usurp dopamine's role in learning. Here, we briefly review the preclinical literature to motivate specific hypotheses about model-free temporal-difference learning and then review the imaging evidence in the drug of abuse with the most substantial societal consequences: alcohol. Despite the breadth of the literature, only a few studies have examined the predictions directly, and these provide at best inconclusive evidence for the involvement of temporal-difference learning alterations in alcohol dependence. We discuss the difficulties of testing the theory in humans, make specific suggestions, and close with a focus on the interaction with other learning mechanisms.

Background Generative models of neuroimaging data, such as dynamic causal models (DCMs), are commonly used for inferring effective connectivity from individual subject data. Recently introduced “generative embedding” approaches have used DCM-based connectivity parameters for supervised classification of individual patients or to find unknown subgroups in heterogeneous groups using unsupervised clustering methods. New method We present a novel framework which combines DCMs with finite mixture models into a single hierarchical model. This approach unifies the inference of connectivity parameters in individual subjects with inference on population structure, i.e. the existence of subgroups defined by model parameters, and allows for empirical Bayesian estimates of a subject’s connectivity based on subgroup-specific prior distributions. We introduce a Markov chain Monte Carlo sampling method for inversion of this hierarchical generative model. Results This paper formally introduces the idea behind our novel concept and demonstrates the face validity of the model in application to both simulated data as well as an empirical fMRI dataset from healthy controls and patients with schizophrenia. Comparison with existing method(s) The analysis of our empirical fMRI data demonstrates that our approach results in superior model evidence than the conventional non-hierarchical inversion of DCMs. Conclusions In this paper, we have presented a novel unified framework to jointly infer the effective connectivity parameters in DCMs for multiple subjects and, at the same time, discover connectivity-defined cluster structure of the whole population, using a mixture model approach.

PLoS Computational Biology, 14 (8), ISSN:1553-734X, ISSN:1553-7358

In the present study, we explored possible alterations in the default mode network (DMN) and its functional connectivity in 41 schizophrenia patients and 42 age-matched healthy controls. Schizophrenia patients displayed reduced activation in the ventromedial prefrontal cortex, left superior temporal gyrus including auditory cortex and temporal pole. Psychophysiological interaction analysis revealed reduced connectivity between left superior temporal gyrus including auditory cortex and the left temporal pole in schizophrenia patients compared to healthy subjects.

It is suggested that acute stress shifts behavioral control from goal-directed, model-based toward habitual, model-free strategies. Recent findings indicate that interindividual differences in the cortisol stress response influence model-based decision-making. Although not yet investigated in humans, animal studies show that chronic stress also shifts decision-making toward more habitual behavior. Here, we ask whether acute stress and individual vulnerability factors, such as stress reactivity and previous exposure to stressful life events, impact the balance between model-free and model-based control systems. To test this, 39 male participants (21-30 years old) were exposed to a potent psychosocial stressor (Trier Social Stress Test) and a control condition in a within-subjects design before they performed a sequential decision-making task which evaluates the balance between the two systems. Physiological and subjective stress reactivity was assessed before, during, and after acute stress exposure. By means of computational modeling, we demonstrate that interindividual variability in stress reactivity predicts impairments in model-based decision-making. Whereas acute psychosocial stress did not alter model-based behavioral control, we found chronic and acute stress to interact in their detrimental effect on decision-making: subjects with high but not low chronic stress levels as indicated by stressful life events exhibited reduced model-based control in response to acute psychosocial stress. These findings emphasize that stress reactivity and chronic stress play an important role in mediating the relationship between stress and decision-making. Our results might stimulate new insights into the interplay between chronic and acute stress, attenuated model-based control, and the pathogenesis of various psychiatric diseases.

Dual system theories suggest that behavioral control is parsed between a deliberative "model-based" and a more reflexive "model-free" system. A balance of control exerted by these systems is thought to be related to dopamine neurotransmission. However, in the absence of direct measures of human dopamine, it remains unknown whether this reflects a quantitative relation with dopamine either in the striatum or other brain areas. Using a sequential decision task performed during functional magnetic resonance imaging, combined with striatal measures of dopamine using [(18)F]DOPA positron emission tomography, we show that higher presynaptic ventral striatal dopamine levels were associated with a behavioral bias toward more model-based control. Higher presynaptic dopamine in ventral striatum was associated with greater coding of model-based signatures in lateral prefrontal cortex and diminished coding of model-free prediction errors in ventral striatum. Thus, interindividual variability in ventral striatal presynaptic dopamine reflects a balance in the behavioral expression and the neural signatures of model-free and model-based control. Our data provide a novel perspective on how alterations in presynaptic dopamine levels might be accompanied by a disruption of behavioral control as observed in aging or neuropsychiatric diseases such as schizophrenia and addiction.

Individuals differ in how they learn from experience. In Pavlovian conditioning models, where cues predict reinforcer delivery at a different goal location, some animals-called sign-trackers-come to approach the cue, whereas others, called goal-trackers, approach the goal. In sign-trackers, model-free phasic dopaminergic reward-prediction errors underlie learning, which renders stimuli 'wanted'. Goal-trackers do not rely on dopamine for learning and are thought to use model-based learning. We demonstrate this double dissociation in 129 male humans using eye-tracking, pupillometry and functional magnetic resonance imaging informed by computational models of sign- and goal-tracking. We show that sign-trackers exhibit a neural reward prediction error signal that is not detectable in goal-trackers. Model-free value only guides gaze and pupil dilation in sign-trackers. Goal-trackers instead exhibit a stronger model-based neural state prediction error signal. This model-based construct determines gaze and pupil dilation more in goal-trackers.

Research on the neural substrates of drug reward, withdrawal and relapse has yet to be translated into significant advances in the treatment of addiction. One potential reason is that this research has not captured a common feature of human addiction: progressive social exclusion and marginalization. We propose that research aimed at understanding the neural mechanisms that link these processes to drug seeking and drug taking would help to make addiction neuroscience research more clinically relevant.

Substance-dependent individuals often lack the ability to adjust decisions flexibly in response to the changes in reward contingencies. Prediction errors (PEs) are thought to mediate flexible decision-making by updating the reward values associated with available actions. In this study, we explored whether the neurobiological correlates of PEs are altered in alcohol dependence. Behavioral, and functional magnetic resonance imaging (fMRI) data were simultaneously acquired from 34 abstinent alcohol-dependent patients (ADP) and 26 healthy controls (HC) during a probabilistic reward-guided decision-making task with dynamically changing reinforcement contingencies. A hierarchical Bayesian inference method was used to fit and compare learning models with different assumptions about the amount of task-related information subjects may have inferred during the experiment. Here, we observed that the best-fitting model was a modified Rescorla-Wagner type model, the “double-update” model, which assumes that subjects infer the knowledge that reward contingencies are anti-correlated, and integrate both actual and hypothetical outcomes into their decisions. Moreover, comparison of the best-fitting model's parameters showed that ADP were less sensitive to punishments compared to HC. Hence, decisions of ADP after punishments were loosely coupled with the expected reward values assigned to them. A correlation analysis between the model-generated PEs and the fMRI data revealed a reduced association between these PEs and the BOLD activity in the dorsolateral prefrontal cortex (DLPFC) of ADP. A hemispheric asymmetry was observed in the DLPFC when positive and negative PE signals were analyzed separately. The right DLPFC activity in ADP showed a reduced correlation with positive PEs. On the other hand, ADP, particularly the patients with high dependence severity, recruited the left DLPFC to a lesser extent than HC for processing negative PE signals. These results suggest that the DLPFC, which has been linked to adaptive control of action selection, may play an important role in cognitive inflexibility observed in alcohol dependence when reinforcement contingencies change. Particularly, the left DLPFC may contribute to this impaired behavioral adaptation, possibly by impeding the extinction of the actions that no longer lead to a reward., Highlights • Alcohol-dependent patients (ADP) had difficulty adapting to the reversals. • The impaired adaptation was associated with a decrease in punishment sensitivity. • The dorsolateral prefrontal cortex (DLPFC) of ADP failed to track prediction errors. • A reduced tracking of the negative prediction error was present in the left DLPFC. • The clinical severity of dependence was correlated with abnormal DLPFC activity.

Despite its clinical relevance and the recent recognition as a diagnostic category in the DSM-5, binge eating disorder (BED) has rarely been investigated from a cognitive neuroscientific perspective targeting a more precise neurocognitive profiling of the disorder. BED patients suffer from a lack of behavioral control during recurrent binge eating episodes and thus fail to adapt their behavior in the face of negative consequences, eg, high risk for obesity. To examine impairments in flexible reward-based decision-making, we exposed BED patients (n=22) and matched healthy individuals (n=22) to a reward-guided decision-making task during functional resonance imaging (fMRI). Performing fMRI analysis informed via computational modeling of choice behavior, we were able to identify specific signatures of altered decision-making in BED. On the behavioral level, we observed impaired behavioral adaptation in BED, which was due to enhanced switching behavior, a putative deficit in striking a balance between exploration and exploitation appropriately. This was accompanied by diminished activation related to exploratory decisions in the anterior insula/ventro-lateral prefrontal cortex. Moreover, although so-called model-free reward prediction errors remained intact, representation of ventro-medial prefrontal learning signatures, incorporating inference on unchosen options, was reduced in BED, which was associated with successful decision-making in the task. On the basis of a computational psychiatry account, the presented findings contribute to defining a neurocognitive phenotype of BED.

Adolescence is a critical maturation period for human cognitive control and executive function. In this study, a large sample of adolescents (n = 85) performed a reversal learning task during functional magnetic resonance imaging. We analyzed behavioral data using a reinforcement learning model to provide individually fitted parameters and imaging data with regard to reward prediction errors (PE). Following a model-based approach, we formed two groups depending on whether individuals tended to update expectations predominantly for the chosen stimulus or also for the unchosen one. These groups significantly differed in their problem behavior score obtained using the child behavior checklist (CBCL) and in a measure of their developmental stage. Imaging results showed that dorsolateral striatal areas covaried with PE. Participants who relied less on learning based on task structure showed less prefrontal activation compared with participants who relied more on task structure. An exploratory analysis revealed that PE-related activity was associated with pubertal development in prefrontal areas, insula and anterior cingulate. These findings support the hypothesis that the prefrontal cortex is implicated in mediating flexible goal-directed behavioral control.

Huntington's disease (HD) is an autosomal dominant neurodegenerative condition characterized by a triad of movement disorder, neuropsychiatric symptoms and cognitive deficits. The striatum is particularly vulnerable to the effects of mutant huntingtin, and cell loss can already be found in presymptomatic stages. Since the striatum is well known for its role in reinforcement learning, we hypothesized to find altered behavioral and neural responses in HD patients in a probabilistic reinforcement learning task performed during functional magnetic resonance imaging. We studied 24 HD patients without central nervous system (CNS)-active medication and 25 healthy controls. Twenty HD patients and 24 healthy controls were able to complete the task. Computational modeling was used to calculate prediction error values and estimate individual parameters. We observed that gray matter density and prediction error signals during the learning task were related to disease stage. HD patients in advanced disease stages appear to use a less complex strategy in the reversal learning task. In contrast, HD patients in early disease stages show intact encoding of learning signals in the degenerating left ventral striatum. This effect appears to be lost with disease progression.