Search

Rewards influence information processing in the primary sensory areas specialized to process stimuli from a specific sensory modality. In real life situations, we receive sensory inputs not only from one single modality, but stimuli are often multisensory. It is however not known whether the reward-driven modulation of perception follows the same principles when reward is cued through a single or multiple sensory modalities. We previously showed that task-irrelevant reward cues modulate perception both intra- as well as cross-modally, likely through a putative enhancement in the integration of the stimulus parts into a coherent object. In this study, we explicitly test this possibility by assessing whether reward enhances the integration of unisensory components of a multisensory object in accordance with thesupra-additiveprinciple of multisensory integration. Towards this aim, we designed a simple detection task using reward predicting cues that were either unisensory (auditory or visual, both above the detection threshold) or multisensory (audiovisual). We conducted two experiments, behavioral (experiment 1) and simultaneous behavioral and neuroimaging testing (experiment 2). We expected that reward speeds up reaction times in response to all stimulus configurations, and that additionally the reward effects in multisensory cues fulfill thesupra-additiveprinciple of multisensory integration. We observed that reward decreased response times in both experiments with the strongest effect found for the multisensory stimuli in experiment 1. However, this behavioral effect did not fulfill thesupra-additiveprinciple. Neuroimaging results demonstrated sensory supra-additivity at the classical areas involved in multisensory integration such as the Superior Temporal areas (STS), while reward modulation was found in the midbrain and fronto-parietal areas, reflecting the typical areas that receive dopaminergic projections. However, reward did not enhance thesupra-additivityin the STS compared to a no reward condition. Instead, we observed that some of the reward-related areas showed asub-additivemodulation by rewards and areas exhibiting a weakersupra-additiveresponse to audiovisual stimuli, namely the fusiform gyrus, were modulated by rewards of audiovisual stimuli as measured by a conjunction analysis. Overall, our results indicate that reward does not enhance the multisensory integration through asupra-additiverule. These findings inspire a model where reward and sensory integration processes are regulated by two independent mechanisms, where sensory information is integrated at an early stage in asupra-additivemanner, while reward modulates perception at a later stagesub-additively. Moreover, an associative area in the Fusiform gyrus exhibits a convergence of both reward and multisensory integration signals, indicating that it may be ahubto integrate different types of signals including rewards to disambiguate the information from different sensory modalities.

Past reward associations may be signaled by stimuli from different sensory modalities, however it remains unclear how different types of reward-associated stimuli modulate perception. In this human fMRI study, we employed a paradigm involving a visual discrimination task, where a visual target was simultaneously presented with either an intra-(visual) or a cross-modal (auditory) cue that was previously associated with rewards. We hypothesized that depending on the sensory modality of the cues distinct neural pathways underlie the value-driven modulation of visual areas. Two steps of analyses were conducted: first, using a multivariate approach, we confirmed that previously reward-associated cues enhanced the target representation in the early visual areas. Then, using effective connectivity analysis, we tested three possible patterns of communication across the brain regions that could underlie the modulation of visual cortex: a direct pathway from the frontal valuation areas to the visual areas, a mediated pathway through the attention-related areas, and a mediated pathway that additionally involved distinct sensory association areas for auditory and visual rewards. We found evidence for the third model and demonstrate that reward-related information is communicated across the valuation and attention-related brain regions such as the intraparietal sulcus across for both visual and auditory cues. Additionally, the long-range communication of reward information also involved the superior temporal areas in case of auditory reward-associated stimuli. These results suggest that in the presence of previously rewarded stimuli from different sensory modalities, a combination of domain-general and domain-specific mechanisms are recruited across the brain to adjust visual processing.

Standard neuroeconomics theories state that the value of different classes of stimuli, for instance the hedonic value of food versus music, is transformed to a common reference scale that is independent of their sensory properties. However, adaptive behaviour in a multimodal and dynamic environment requires that our brain also encodes information about the sensory features of reward predicting stimuli. Whether and how a common code for value could integrate information about the sensory features of rewarding stimuli remains inadequately understood. By employing stimuli from auditory and visual modalities as reward predicting cues in a value-based decision-making task, we were able to vary the reward value and sensory modality independently and dissociate neural codes of auditory and visual rewards in frontal areas using fMRI. Univariate fMRI analysis revealed modality-specific and modality-general value representations in orbitofrontal cortex (OFC) and ventromedial prefrontal cortex (vmPFC), respectively. Crucially, modality-specific representations were highly selective as they were only activated when participants believed that the corresponding sensory modality was associated with reward and were absent when the task involved instruction-based rather than value-based choices. Moreover, we show that modality-specific value representations are supported by the presence of the effective connectivity between each primary sensory area and the corresponding OFC activation and further between modality-specific value representations in OFC and vmPFC, only when the sensory modality to be chosen is associated with reward and absent otherwise. Our results indicate the presence of both modality-specific and modality-general representations of reward value and reveal mechanisms through which the interaction between the sensory cortices and the two types of representation guides value-based decisions.

Supplementary Text and Figures for "Differential effects of intra-modal and cross-modal reward value on perception: ERP evidence", Vkhrushev et al.

Reward modulates perceptual encoding and its neuronal correlates (Baldassi & Simoncini, 2011; Bayer et al., 2017; Hickey, Chelazzi, & Theeuwes, 2010; Hughes, Mathan, & Yeung, 2013; Leo & Noppeney, 2014; Pooresmaeili et al., 2014; San Martín, Appelbaum, Huettel, & Woldorff, 2016; Serences, 2008). Similarly, selective attention affects behavioral and neural responses. The similarity between reward and attention has raised the question of whether they reflect identical or distinct cognitive processes (Maunsell, 2004). Previous studies have provided divergent results regarding this question. On one hand, some studies have shown that reward and attention are not distinguishable from each other (Stanisor, Van Der Togt, Pennartz, & Roelfsema, 2013). This idea is also supported by the observation that rewarding signals automatically capture attention (Hickey et al., 2010; Hickey & Van Zoest, 2012; Yantis, Anderson, Wampler, & Laurent, 2012) and hence all effects of reward could potentially be exerted through attention. On the other hand, other studies have shown that attention and reward are distinguishable processes as reward effects are unchanged by the amount of attentional load (Baldassi & Simoncini, 2011) and occur during separate stages of processing captured by distinct ERP components (Bachman, Wang, Gamble, & Woldorff, 2020; Bayer et al., 2017). Here we draw on our previous findings where we showed that reward cues from a different sensory modality (audition) can affect perception in vision (Pooresmaeili et al., 2014) and that these effects are less spatially confined than typical attentional effects. Furthermore, we have recently shown that behavioral and EEG correlates of reward effects differ between cross-modal and within-modal signals (unplublished data, see the preregistration in https://osf.io/47wxr/). Taken together, these previous results from our lab indicate that the involvement of attentional mechanisms in the processing of rewarding cues differs for cross-modal and within-modal rewards. In the current study we ask 2 questions related to these findings: 1. Do the behavioral and neural effects of reward occur in the absence of attention (i.e. in an unattended location)? 2. Does the interaction between reward and attention depend on the sensory modality of reward cues? To answer these questions we modified a task developed by Talsma (2005) where it is possible to control independently both attention and reward. In this task, participants perform a change detection task where the change occurs only in 20% of all trials. We use bimodal stimuli with 4 different sensory configurations based on how the reward is delivered and manipulate the spatial attention across blocks to different sides of the display.

When comparing themselves with others, people often perceive their own actions and behaviour favourably. This phenomenon is often categorised as a bias of attribution, with favourable self-evaluation resulting from differing explanations of one’s own behaviour and that of others. However, studies on availability biases offer an alternative explanation, ascribing egocentric biases to the inherent sensory asymmetries between performing an action and merely observing it. In this study, we used a paradigm that allowed us to directly compare these two distinct sources of bias. Participants perceived the tasks they performed to be harder than the tasks they observed, but demonstrated no bias driven by favourable self-evaluation. Furthermore, the degree of overestimation of the difficulty of performed tasks was magnified as overall task difficulty increased. These findings suggest that egocentric biases are in part derived from sensory asymmetries inherent to the first-person perspective.

Numerous different objects are simultaneously visible in a person's visual field, competing for attention. This competition has been shown to affect eye‐movements and early neural responses toward stimuli, while the role of a stimulus' emotional meaning for mechanisms of overt attention shifts under competition is unclear. The current study combined EEG and eye‐tracking to investigate effects of competition and emotional content on overt shifts of attention to human face stimuli. Competition prolonged the latency of the P1 component and of saccades, while faces showing emotional expressions elicited an early posterior negativity (EPN). Remarkably, the emotion‐related modulation of the EPN was attenuated when two stimuli were competing for attention compared to non‐competition. In contrast, no interaction effects of emotional expression and competition were observed on other event‐related potentials. This finding indicates that competition can decelerate attention shifts in general and also diminish the emotion‐driven attention capture, measured through the smaller effects of emotional expression on EPN amplitude. Reduction of the brain's responsiveness to emotional content in the presence of distractors contradicts models that postulate fully automatic processing of emotions.

Over the last decades, several studies have demonstrated that conscious and unconscious reward incentives both affect performance in physical and cognitive tasks, suggesting that goal-pursuit can arise from an unconscious will. Whether the planning of goal-directed saccadic eye movements during an effortful task can also be affected by subliminal reward cues has not been systematically investigated. We employed a novel task where participants made several eye movements back and forth between a fixation point and a number of peripheral targets. The total number of targets visited by the eyes in a fixed amount of time determined participants' monetary gain. The magnitude of the reward at stake was briefly shown at the beginning of each trial and masked by pattern images superimposed in time so that at shorter display durations participants perceived reward incentives subliminally. We found a main effect of reward across all display durations as higher reward enhanced participants' oculomotor effort measured as the frequency and peak velocity of saccades. This effect was strongest for consciously perceived rewards but also occurred when rewards were subliminally perceived. Although we did not find a statistically significant dissociation between the reward-related modulation of different saccadic parameters, across two experiments the most robust effect of subliminal rewards was observed for the modulation of the saccadic frequency but not the peak velocity. These results suggest that multiple indices of oculomotor effort can be incentivized by subliminal rewards and that saccadic frequency may provide the most sensitive indicator of subliminal incentivization of eye movements.

Stimuli associated with high reward modulate perception and such value-driven effects have been shown to originate from the modulation of the earliest stages of sensory processing in the brain. In natural environments objects comprise multiple features (imagine a rolling soccer ball, with its black and white patches and the swishing sound made during its motion), where each feature may signal different associations with previously encountered rewards. How perception of such an object is affected by the value associations of its constituent parts is unknown. The present study compares intra- and cross-modal value-driven effects on behavioral and electrophysiological correlates of visual perception. Human participants first learned the reward associations of visual and auditory cues. Subsequently, they performed a visual orientation discrimination task in the presence of previously rewarded visual or auditory cues (intra- and cross-modal cues, respectively) that were concurrently presented with the target stimulus. During the conditioning phase, when reward associations were learned and reward cues were the target of the task, reward value of both modalities enhanced the electrophysiological correlates of sensory processing in visual cortex. During the post-conditioning phase, when reward delivery was halted and previously rewarded stimuli were task-irrelevant, cross-modal value-enhanced behavioral measures of visual sensitivity whereas intra-modal value led to a trend for suppression. A similar pattern of modulations was found in the simultaneously recorded event-related potentials (ERPs) of posterior electrodes. We found an early (90-120 ms) suppression of ERPs evoked by high-value, intra-modal stimuli. Cross-modal cues led to a later value-driven modulation, with an enhancement of response positivity for high-compared to low-value stimuli starting at the N1 window (180-250 ms) and extending to the P3 (300-600 ms) responses of the posterior electrodes. These results indicate that visual cortex is modulated by the reward value of visual as well as auditory cues. Previously rewarded, task-irrelevant cues from the same or different sensory modality have a different effect on visual perception, as intra-modal high-value cues may interfere with the target processing, whereas cross-modal high-value cues boost the perception of the target.

Estimating invested effort is a core dimension for evaluating own and others’ actions, and views on the relationship between effort and rewards are deeply ingrained in various societal attitudes. Internal representations of effort, however, are inherently noisy, e.g. due to the variability of sensorimotor and visceral responses to physical exertion. The uncertainty in effort judgments is further aggravated when there is no direct access to the internal representations of exertion – such as when estimating the effort of another person. Bayesian cue integration suggests that this uncertainty can be resolved by incorporating additional cues that are predictive of effort, e.g. received rewards. We hypothesized that judgments about the effort spent on a task will be influenced by the magnitude of received rewards. Additionally, we surmised that such influence might further depend on individual beliefs regarding the relationship between hard work and prosperity, as exemplified by a conservative work ethic. To test these predictions, participants performed an effortful task interleaved with a partner and were informed about the obtained reward before rating either their own or the partner’s effort. We show that higher rewards led to higher estimations of exerted effort in self-judgments, and this effect was even more pronounced for other-judgments. In both types of judgment, computational modelling revealed that reward information and sensorimotor markers of exertion were combined in a Bayes-optimal manner in order to reduce uncertainty. Remarkably, the extent to which rewards influenced effort judgments was associated with conservative world-views, indicating links between this phenomenon and general beliefs about the relationship between effort and earnings in society. Open-Access-Publikationsfonds 2020 peerReviewed

In everyday life, faces with emotional expressions quickly attract attention and eye-movements. To study the neural mechanisms of such emotion-driven attention by means of event-related brain potentials (ERPs), tasks that employ covert shifts of attention are commonly used, in which participants need to inhibit natural eye-movements towards stimuli. It remains, however, unclear how shifts of attention to emotional faces with and without eye-movements differ from each other. The current preregistered study aimed to investigate neural differences between covert and overt emotion-driven attention. We combined eye-tracking with measurements of ERPs to compare shifts of attention to faces with happy, angry or neutral expressions when eye-movements were either executed (Go conditions) or withheld (No-go conditions). Happy and angry faces led to larger EPN amplitudes, shorter latencies of the P1 component and faster saccades, suggesting that emotional expressions significantly affected shifts of attention. Several ERPs (N170, EPN, LPC), were augmented in amplitude when attention was shifted with an eye-movement, indicating an enhanced neural processing of faces if eye-movements had to be executed together with a reallocation of attention. However, the modulation of ERPs by facial expressions did not differ between the Go and No-go conditions, suggesting that emotional content enhances both covert and overt shifts of attention. In summary, our results indicate that overt and covert attention shifts differ but are comparably affected by emotional content.

Humans can use their previous experience in form of statistical priors to improve decisions. It is, however, unclear how such priors are learned and represented. Importantly, it has remained elusive whether prior learning is independent of the sensorimotor system involved in the learning process or not, as both modality-specific and modality-general learning have been reported in the past. Here, we used a saccadic eye movement task to probe the learning and representation of a spatial prior across a few trials. In this task, learning occurs in an unsupervised manner and through encountering trial-by-trial visual hints drawn from a distribution centered on the target location. Using a model-comparison approach, we found that participants’ prior knowledge is largely represented in the form of their previous motor actions, with minimal influence from the previously seen visual hints. By using two different motor contexts for response (looking either at the estimated target location, or exactly opposite to it), we could further compare whether prior experience obtained in one motor context can be transferred to the other. Although learning curves were highly similar, and participants seemed to use the same strategy for both response types, they could not fully transfer their knowledge between contexts, as performance and confidence ratings dropped after a switch of the required response. Together, our results suggest that humans preferably use the internal representations of their previous motor actions, rather than past incoming sensory information, to form statistical sensorimotor priors on the timescale of a few trials.

Humans can use their previous experience in form of statistical priors to improve decisions. It is however unclear how such priors are learned and represented. Importantly, it has remained elusive whether prior learning is independent of the sensorimotor system involved in the learning process or not, as both modality-specific and modality-general learning have been reported in the past. Here, we used a saccadic eye movement task to probe the learning and representation of a spatial prior across a few trials. In this task, learning occurs in an unsupervised manner and through encountering trial-by-trial visual hints drawn from a distribution centered on the target location. Using a model-comparison approach, we found that participants’ prior knowledge is largely represented in the form of their previous motor actions, with minimal influence from the previously seen visual hints. By using two different motor contexts for response (looking either at the estimated target location, or exactly opposite to it), we could further compare whether prior experience obtained in one motor context can be transferred to the other. Although learning curves were highly similar, and participants seemed to use the same strategy for both response types, they could not transfer their knowledge between contexts, as performance and confidence ratings dropped to naïve levels after a switch of the required response. Together, our results suggest that humans preferably use the internal representations of their previous motor actions, rather than past incoming sensory information, to form statistical sensorimotor priors on the timescale of a few trials.

Over the last decades, several studies have demonstrated that conscious and unconscious reward incentives both affect performance in physical and cognitive tasks, suggesting that goal-pursuit can arise from an unconscious will. Whether the planning of goal-directed saccadic eye movements during an effortful task can also be affected by subliminal reward cues has not been systematically investigated. We employed a novel task where participants had to make several eye movements back and forth between a fixation point and a number of peripheral targets. The total number of targets visited by the eyes in a fixed amount of time determined participants’ monetary gain. The magnitude of the reward at stake was briefly shown at the beginning of each trial and was masked by pattern images superimposed in time. We found that when reward cues were fully visible and thus consciously perceived, higher reward enhanced all saccade parameters. However, a dissociation was observed between the effects of subliminal rewards on saccade initiation and peak velocities. While truly subliminal reward cues did increase the number of saccades, they did not enhance saccades’ peak velocity. Additionally, participants who had reached a truly subliminal level of reward perception showed a decrement in accuracy as a function of reward across all visibility levels, as saccade endpoint error was larger when higher reward incentives were expected. This suboptimal speed-accuracy trade-off did not occur in the supraliminal group. These results suggest that although saccades’ initiation can be triggered by subconscious mechanisms, conscious awareness is required to optimally adjust the velocity and accuracy of eye movements based on the expected rewards.

One of the most important tasks for the visual system is to construct an internal representation of the spatial properties of objects, including their size. Size perception includes a combination of bottom-up (retinal inputs) and top-down (e.g., expectations) information, which makes the estimates of object size malleable and susceptible to numerous contextual cues. For example, it has been shown that size perception is prone to adaptation: brief previous presentations of larger or smaller adapting stimuli at the same region of space changes the perceived size of a subsequent test stimulus. Large adapting stimuli cause the test to appear smaller than its veridical size and vice versa. Here, we investigated whether size adaptation is susceptible to attentional modulation. First, we measured the magnitude of adaptation aftereffects for a size discrimination task. Then, we compared these aftereffects (on average 15-20%) with those measured while participants were engaged, during the adaptation phase, in one of the two highly demanding central visual tasks: Multiple Object Tracking (MOT) or Rapid Serial Visual Presentation (RSVP). Our results indicate that deploying visual attention away from the adapters did not significantly affect the distortions of perceived size induced by adaptation, with accuracy and precision in the discrimination task being almost identical in all experimental conditions. Taken together, these results suggest that visual attention does not play a key role in size adaptation, in line with the idea that this phenomenon can be accounted for by local gain control mechanisms within area V1.

Reward value guides goal-directed behavior and modulates early sensory processing. Rewarding stimuli are often multisensory but it is not known how reward value is combined across sensory modalities. Here we show that the integration of reward value critically depends on whether the distinct sensory inputs are perceived to emanate from the same multisensory object. We systematically manipulated the congruency in monetary reward values and the relative spatial positions of co-occurring auditory and visual stimuli that served as bimodal distractors during an oculomotor task. The amount of interference induced by the distractors was used as an indicator of their perceptual salience. Our results across two experiments show that when reward value is linked to each modality separately, the value congruence between vision and audition determines the combined salience of the bimodal distractors. However, reward value of vision wins over the value of audition if visual and auditory stimuli have been experienced as belonging to the same audiovisual object prior to the learning of the reward values. The perceived spatial alignment of auditory and visual stimuli is a prerequisite for the integration of their reward values, as no effect of reward value was observed when the two modalities were perceived to be misaligned. These results show that in a task that highly relies on the processing of visual spatial information, the reward values from multiple sensory modalities are integrated with each other, each with their respective weights. This weighting depends on the congruency in reward values, exposure history, and spatial co-localization.

Effort constitutes a major part of cost-benefit calculations underlying decision making. Therefore, estimating the effort someone has spent on a task is a core dimension for evaluating own and others’ actions. It has been previously shown that self-judgments of effort are influenced by the magnitude of obtained rewards. It is unclear, however, whether the influence of reward on effort estimations is limited to self-judgments or whether reward incorporation represents a general computational principle when judging effort. Here we show that people also integrate reward magnitude when judging the effort exerted by others. Participants (N=48) performed an effortful sensorimotor task interleaved with a partner, while rating either their own or the other person’s effort. After each trial but before the effort rating, both participants were informed about the obtained reward. We found that higher rewards led to higher estimations of exerted effort, in self-as well as other-judgments, and this effect was more pronounced for other-judgments. In both types of judgment, computational modelling revealed that reward information and the perceived level of exertion were combined in a Bayes optimal manner to form effort estimates. Remarkably, the extent to which rewards influenced effort judgments was positively correlated with conservative world-views, indicating that the basic computations underlying this behavioural phenomenon might be related to more general beliefs about the association between effort and reward in the society. The integration of reward information into retrospective effort judgments underscores the convergence of multiple information sources that supports adaptive learning and decision making in social contexts.

Societal norms exert a powerful influence on our decisions. Behaviours motivated by norms, however, do not always concur with the responses mandated by decision relevant information potentially generating a conflict. To probe the interplay between normative and informational influences, we examined how prosocial norms impact on perceptual decisions subjects made in the context of a simultaneous presentation of social information. Participants displayed a bias in their perceptual decisions towards that mandated by social information. However, normative prescriptions modulated this bias bi-directionally depending on whether norms mandated a decision in accord or contrary to the contextual social information. At a neural level, the addition of a norms increased activity in prefrontal cortex and modulated functional connectivity between prefrontal and parietal areas. The bi-directional effect of our norms was captured by differential activations when participants decided against the social information. When norms indicated a decision in line with social information, non-compliance modulated lateral prefrontal cortex activity. By contrast, when norms mandated a decision against social information norm compliance increased activity in the anterior cingulate cortex. Hence, social norms changed the balance between a reliance on perceptual and social information by modulating brain activity in regions associated with response inhibition and conflict monitoring.

Deciding whether a stimulus is the “same” or “different” from a previous presented one involves integrating among the incoming sensory information, working memory, and perceptual decision making. Visual selective attention plays a crucial role in selecting the relevant information that informs a subsequent course of action. Previous studies have mainly investigated the role of visual attention during the encoding phase of working memory tasks. In this study, we investigate whether manipulation of bottom-up attention by changing stimulus visual salience impacts on later stages of memory-based decisions. In two experiments, we asked subjects to identify whether a stimulus had either the same or a different feature to that of a memorized sample. We manipulated visual salience of the test stimuli by varying a task-irrelevant feature contrast. Subjects chose a visually salient item more often when they looked for matching features and less often so when they looked for a nonmatch. This pattern of results indicates that salient items are more likely to be identified as a match. We interpret the findings in terms of capacity limitations at a comparison stage where a visually salient item is more likely to exhaust resources leading it to be prematurely parsed as a match.

How do you make a decision if you do not know the rules of the game? Models of sensory decision-making suggest that choices are slow if evidence is weak, but they may only apply if the subject knows the task rules. Here, we asked how the learning of a new rule influences neuronal activity in the visual (area V1) and frontal cortex (area FEF) of monkeys. We devised a new icon-selection task. On each day, the monkeys saw 2 new icons (small pictures) and learned which one was relevant. We rewarded eye movements to a saccade target connected to the relevant icon with a curve. Neurons in visual and frontal cortex coded the monkey's choice, because the representation of the selected curve was enhanced. Learning delayed the neuronal selection signals and we uncovered the cause of this delay in V1, where learning to select the relevant icon caused an early suppression of surrounding image elements. These results demonstrate that the learning of a new rule causes a transition from fast and random decisions to a more considerate strategy that takes additional time and they reveal the contribution of visual and frontal cortex to the learning process. This work was supported by NWO (Brain and Cognition grant 433-09-208 and ALW grant 823-02-010), the European Union Seventh Framework Program (project 269921 “BrainScaleS” and Marie-Curie Action PITN-GA-2011-290011), and an ERC advanced grant (Cortic_al_gorithms; no 339490) awarded to P.R.R.

The visual system encodes the features of visual stimuli as well as their behavioral relevance. Stimuli with a high luminance contrast evoke more activity in the visual cortex than stimuli with a low contrast. At the same time, attended stimuli evoke more activity than nonattended stimuli. There is a debate about how visual features and attention jointly determine neuronal activity in the visual cortex. Some studies suggested that attention increases apparent contrast (Reynolds et al., 2000), others that attention amplifies responses by a constant factor (Williford and Maunsell, 2006), and yet others that attention and contrast have largely additive effects (Buracas and Boynton, 2007; Thiele et al., 2009). The influence of attention on contrast sensitivity differs between neurons, raising the possibility that attention and contrast could be coded conjointly in a population of neurons. Here we investigate this possibility by recording neuronal activity at multiple sites in the primary visual cortex of macaque monkeys using multielectrode recording techniques and support vector machines to decode attended stimuli as well as stimulus contrast. We find that many, but not all, V1 neurons are influenced by attention and that the effects of attention and contrast are additive on average. Stimulus contrast can be decoded from neuronal responses not strongly modulated by attention, whereas the attended stimulus can be decoded as the difference in activity of cells that are influenced by attention and cells that are not. The success of the approach suggests that visual attention and stimulus contrast are represented by largely separable codes. Copyright © 2010 the authors.

Significance Retrospective reevaluation of effort is a pervasive aspect of everyday life, such as when we assess our professional satisfaction after knowing the ensuing outcomes. Previous studies have focused on the interaction of effort and reward when a choice is to be made, whereas retrospective interactions have been largely ignored. Here we show that humans revise their estimation of effort after receiving a reward. When rewarded more than average, subjects tend to overestimate their effort, with a converse effect observed for low rewards. The size of this bias depends strongly on the contingency between reward magnitude and task difficulty and is dynamically adjusted when changes occur in these contingencies. These results reveal a sophisticated mechanism to cope with reward–effort inconsistencies.

Summary Background Object-based attention can group image elements of spatially extended objects into coherent representations, but its mechanisms have remained unclear. The mechanisms for object-based attention may include shape-selective neurons in higher visual cortical areas that feed back to lower areas to simultaneously enhance the representation of all image elements of a relevant shape. Another, nonexclusive mechanism is the spread of attention in early visual cortex according to Gestalt rules, which could successively add new elements to a growing object representation. Results We investigated the dynamics of object-based attention in the primary visual cortex of monkeys trained to perform a contour-grouping task. The animals mentally traced a target curve through the visual field and ignored a distracting curve. Neuronal responses elicited by the target curve were enhanced relative to those elicited by distracting curve. Remarkably, the response enhancement was delayed for neurons with receptive fields farther from the start of the tracing process. We could therefore measure propagation speed and found that it was low if curves were nearby and that it increased if curves were far apart. The results are well explained by an "attentional growth-cone" model, which holds that the response enhancement can spread in multiple visual cortical areas with different receptive field sizes at a speed of approximately 50 ms per receptive field. Conclusions Our findings support an active role for early visual areas in object-based attention because neurons in these areas gradually spread enhanced activity over the representation of relevant objects.

Cross-modal interactions are very common in perception. An important feature of many perceptual stimuli is their reward-predicting properties, the utilization of which is essential for adaptive behavior. What is unknown is whether reward associations in one sensory modality influence perception of stimuli in another modality. Here we show that auditory stimuli with high-reward associations increase the sensitivity of visual perception, even when sounds and reward associations are both irrelevant for the visual task. This increased sensitivity correlates with a change in stimulus representation in the visual cortex, indexed by increased multivariate decoding accuracy in simultaneously acquired functional MRI data. Univariate analysis showed that reward associations modulated responses in regions associated with multisensory processing in which the strength of modulation was a better predictor of the magnitude of the behavioral effect than the modulation in classical reward regions. Our findings demonstrate a value-driven cross-modal interaction that affects perception and stimulus encoding, with a resemblance to well-described modulatory effects of attention. We suggest that multisensory processing areas may mediate the transfer of value signals across senses.

Models of visual attention hold that top-down signals from frontal cortex influence information processing in visual cortex. It is unknown whether situations exist in which visual cortex actively participates in attentional selection. To investigate this question, we simultaneously recorded neuronal activity in the frontal eye fields (FEF) and primary visual cortex (V1) during a curve-tracing task in which attention shifts are object-based. We found that accurate performance was associated with similar latencies of attentional selection in both areas and that the latency in both areas increased if the task was made more difficult. The amplitude of the attentional signals in V1 saturated early during a trial, whereas these selection signals kept increasing for a longer time in FEF, until the moment of an eye movement, as if FEF integrated attentional signals present in early visual cortex. In erroneous trials, we observed an interareal latency difference because FEF selected the wrong curve before V1 and imposed its erroneous decision onto visual cortex. The neuronal activity in visual and frontal cortices was correlated across trials, and this trial-to-trial coupling was strongest for the attended curve. These results imply that selective attention relies on reciprocal interactions within a large network of areas that includes V1 and FEF.

In natural scenes, objects rarely occur in isolation but appear within a spatiotemporal context. Here, we show that the perceived size of a stimulus is significantly affected by the context of the scene: brief previous presentation of larger or smaller adapting stimuli at the same region of space changes the perceived size of a test stimulus, with larger adapting stimuli causing the test to appear smaller than veridical and vice versa. In a human fMRI study, we measured the blood oxygen level-dependent activation (BOLD) responses of the primary visual cortex (V1) to the contours of large-diameter stimuli and found that activation closely matched the perceptual rather than the retinal stimulus size: the activated area of V1 increased or decreased, depending on the size of the preceding stimulus. A model based on local inhibitory V1 mechanisms simulated the inward or outward shifts of the stimulus contours and hence the perceptual effects. Our findings suggest that area V1 is actively involved in reshaping our perception to match the short-term statistics of the visual scene.

The perceptual salience and visibility of image elements is influenced by other elements in their vicinity. The perceptual effect of image elements on an adjacent target element depends on their relative orientation. Collinear flanking elements usually improve sensitivity for the target element, whereas orthogonal elements have a weaker effect. It is believed that the collinear flankers exert these effects through lateral interactions between neurons in the primary visual cortex (area V1), but the precise mechanisms underlying these contextual interactions remain unknown. Here, we directly examined this question by recording the effects of flankers on the responses of V1 neurons at parafoveal representations while monkeys performed a fixation task or a contrast detection task. We found, unexpectedly, that collinear flankers reduce the monkeys' perceptual sensitivity for a central target element. This behavioral effect was explained by a flanker-induced increase in the activity of V1 neurons in the absence of the central target stimulus, which reduced the amplitude of the target response. Our results indicate that the dominant effect of collinear flankers in parafoveal vision is suppression and suggest that these suppressive effects are caused by a decrease in the dynamic range of neurons coding the central target. Copyright © 2010 the authors.

Neurons in the frontal eye fields (FEFs) register incoming visual information and select visual stimuli that are relevant for behavior. Here we investigated the timing of the visual response and the timing of selection by recording from single FEF neurons in a curve-tracing task that requires shifts of attention followed by an oculomotor response. We found that the behavioral selection signal in area FEF had a latency of 147 ms and that it was delayed substantially relative to the visual response, which occurred 50 ms after stimulus presentation. We compared the FEF responses to activity previously recorded in the primary visual cortex (area V1) during the same task. Visual responses in area V1 preceded the FEF responses, but the latencies of selection signals in areas V1 and FEF were similar. The similarity of timing of selection signals in structures at opposite ends of the visual cortical processing hierarchy supports the view that stimulus selection occurs in an interaction between widely separated cortical regions. Copyright © 2009 The American Physiological Society.

Previous studies have proposed a variety of mechanisms by which attention influences neuronal activity. Here we investigated the mechanisms of attention in the striate cortex of monkeys performing a spatial or an object-based attention task at various stimulus contrasts and compared neuronal contrast response functions with and without attention. Our data are best described by an “additive” interaction: The influence of attention on the neuronal response is relatively independent of the stimulus contrast, at least when the stimulus has enough contrast to become visible. This shows that attention adds to the neuronal responses in a largely contrast invariant manner. These data support recent functional magnetic resonance imaging studies and suggest that feedback from higher areas exerts a constant attentional drive that is mostly task not stimulus driven.

We are perplexed by Clarke et al.’s (2013) criticisms on our recent contribution to Journal of Vision (Pooresmaeili, Cicchini, Morrone, & Burr, 2012). Our group has long championed the idea that perceptual processing of information can be anchored in a dynamic coordinate system that need not correspond to the instantaneous retinal representation. Our recent evidence shows that temporal duration (Burr, Tozzi, & Morrone, 2007; Morrone, Cicchini, & Burr, 2010), orientation (Zimmermann, Morrone, Fink, & Burr, 2013), motion (Melcher & Morrone, 2003; Turi & Burr, 2012) and saccadic error-correction (Zimmermann, Burr, & Morrone, 2011) are all processed to some extent in spatiotopic coordinates. Imaging studies reinforce these studies (d’Avossa et al., 2007; Crespi et al., 2011). Much earlier, we showed that the processing of smoothly moving objects was not anchored in instantaneous, retinotopic coordinates, but in the reference frame given by the trajectory of motion. There is an effective interpolation along the trajectory, so temporal offsets in spatially collinear stimuli causes them to appear spatially offset, corresponding to the physical reality of stimuli moving over large regions of space, behind occluders (Burr, 1979; Burr & Ross, 1979). Our explanation for this surprising effect was that it could be a direct consequence of the spatiotemporal orientation of the impulsive response of motion detectors, providing the spatiotemporal reference frame needed to account for the interactions between time and space (Burr & Ross, 1986; Burr, Ross, & Morrone, 1986; Burr & Ross, 2004; Nishida, 2004). Recently, we have applied the concept of spatiotemporal oriented receptive fields to account for ‘‘predictive remapping,’’ the ‘‘nonretinotopic’’ effects that occur on each saccadic eye-movement (Burr & Morrone, 2010; Burr & Morrone, 2012; Cicchini, Binda, Burr, & Morrone, 2012). We were most impressed by the compelling demonstrations of Herzog’s group, clearly showing that the reference frame of processing is not the instantaneous retinal position, but is flexible, depending not only on real physical motion, but on an illusory apparent motion where the stimuli do not actually move (Boi, Ogmen, Krummenacher, Otto, & Herzog, 2009). This seemed to us important, worthy of quantitative measurement and modeling, particularly to see whether these new effects may fall within the framework that so successfully explained previous demonstrations, such as spatiotemporal interpolation. It is reassuring that Clarke et al. (2013) confirm our results, albeit with some variability between subjects. But more importantly add a very nice result in showing that our simplified version of the ‘‘litmus test’’ can be enhanced by attending to the motion. This is an excellent point that we overlooked. The strength of this type of motion is well known to depend on attention (Cavanagh, 1992), and it is indeed interesting that the strength of motion-induced effects depends not only on the physical conditions, but on internal states such as attention. Perhaps attention may also provide the flexibility in choosing the most appropriate scale for analysis, which in this case would be lower, given that attention is diverted to the periphery. This would add strength to our model, and an idea worth following up