Search

SUMMARY Hippocampal maps incorporate reward information, yet the functional contributions of the anterior and posterior hippocampal divisions (AH and PH) to reinforcement learning remain unclear. Here, we examined exploration and exploitation of a continuous unidimensional task with a basis function reinforcement learning model. In model-based fMRI analyses, we found doubly dissociated representations along the hippocampal long axis: state-wise reward prediction error signals in the PH (tail) and global value maximum signals in the AH (anterior body). PH prediction error signals predicted exploration whereas AH global value maximum signals predicted exploitation. AH-mediated exploitation depended on value representations compressed across episodes and options. PH responses to reinforcement were early and phasic while AH responses were delayed and evolved throughout learning. During choice, AH (head) displayed goal cell-like responses to the global value maximum. In summary, granular reinforcement representations in PH facilitate exploration and compressed representations of the value maximum in AH facilitate exploitation.

Laboratory studies of value-based decision-making often involve choosing among a few discrete actions. Yet in natural environments, we encounter a multitude of options whose values may be unknown or poorly estimated. Given that our cognitive capacity is bounded, in complex environments, it becomes hard to solve the challenge of whether to exploit an action with known value or search for even better alternatives. In reinforcement learning, the intractable exploration/exploitation tradeoff is typically handled by controlling the temperature parameter of the softmax stochastic exploration policy or by encouraging the selection of uncertain options.We describe how selectively maintaining high-value actions in a manner that reduces their information content helps to resolve the exploration/exploitation dilemma during a reinforcement-based timing task. By definition of the softmax policy, the information content (i.e., Shannon’s entropy) of the value representation controls the shift from exploration to exploitation. When subjective values for different response times are similar, the entropy is high, inducing exploration. Under selective maintenance, entropy declines as the agent preferentially maps the most valuable parts of the environment and forgets the rest, facilitating exploitation. We demonstratein silicothat this memory-constrained algorithm performs as well as cognitively demanding uncertainty-driven exploration, even though the latter yields a more accurate representation of the contingency.We found that human behavior was best characterized by a selective maintenance model. Information dynamics consistent with selective maintenance were most pronounced in better-performing subjects, in those with higher non-verbal intelligence, and in learnable vs. unlearnable contingencies. Entropy of value traces shaped human exploration behavior (response time swings), whereas uncertainty-driven exploration was not supported by Bayesian model comparison. In summary, when the action space is large, strategic maintenance of value information reduces cognitive load and facilitates the resolution of the exploration/exploitation dilemma.Author summaryA much-debated question is whether humans explore new options at random or selectively explore unfamiliar options. We show that uncertainty-driven exploration recovers a more accurate picture of simulated environments, but typically does not lead to greater success in foraging. The alternative approach of mapping the most valuable parts of the world accurately while having only approximate knowledge of the rest is just as successful, requires less representational capacity, and provides a better explanation of human behavior. Furthermore, when searching among a multitude of response times, people cannot indefinitely maintain information about every experience. A good strategy for someone with limited memory capacity is to selectively maintain a valuable subset of options and gradually forget the rest. In simulated worlds, a player with this strategy was as successful as a player that represented all previous experiences. When learning a time-varying contingency, humans behaved in a manner consistent with a selective maintenance account. The amount of information retained under this strategy is high early in learning, encouraging exploration, and declines after one has discovered valuable response times.

Perceived burdensomeness is thought to contribute to suicide risk. However, suicidal behavior is clinically and psychologically heterogeneous. Does a high level of perceived burdensomeness differentiate medically serious suicidal acts, most closely resembling death by suicide, from less serious ones? How is perceived burdensomeness related to dysfunctional personality dimensions implicated in suicide? We sought to answer these questions in a cross-sectional, case-control study of adults, aged 42 years or older (n = 165).Participants were suicidal depressed with history of high-lethality and low-lethality attempts, depressed with serious suicidal ideation, depressed non-suicidal, and psychiatrically healthy controls. Following detailed clinical characterization, we assessed perceived burdensomeness, the Big Five, impulsivity, and anger rumination.Low-lethality attempters reported the highest levels of perceived burdensomeness, followed by ideators, high-lethality attempters, non-suicidal depressed, and healthy controls. Group differences were robust to confounders, including demographics, severity of depression, and physical illness burden. In suicide attempters, perceived burdensomeness scaled positively with neuroticism, impulsivity, and anger and negatively with extraversion, conscientiousness, and age.Our findings suggest that perceived burdensomeness is most prominent in a subgroup of younger individuals with lower-lethality suicide attempts and a dysfunctional interpersonal style. Older adults with high-lethality attempts are surprisingly more resilient to the feelings of burdensomeness. Copyright © 2016 John WileySons, Ltd.

Identifying the predictors of late-life depression that are amenable to change may lead to interventions that result in better and faster remission. Thus, the authors investigated the impact of two different strategies for coping with physical illness on depression in older, primary care patients. Health-oriented goal engagement strategies involve the investment of cognitive and behavioral resources to achieve health goals. Conversely, disengagement strategies involve the withdrawal of these resources from obsolete or unattainable health goals, combined with goal restructuring.The participants were 271 adults aged59 years who took part in a two-year randomized clinical trial for treating depression in older adults (Prevention of Suicide in Primary Care Elderly: Collaborative Trial). The use of engagement and disengagement strategies, along with other risk factors for depression, were included in a tree-structured survival analysis to identify subgroups of individuals at risk for not achieving depression remission.The use of disengagement strategies predicted earlier remission of depression, particularly among more severely depressed older patients. The use of engagement strategies did not predict earlier remission.Interventions that encourage disengagement from unattainable health goals may promote remission from depression in older, primary care patients.