Search

Sensory receptive fields are large enough that they can contain more than one perceptible stimulus. How, then, can the brain encode information abouteachof the stimuli that may be present at a given moment? We recently showed that when more than one stimulus is present, single neurons can fluctuate between coding one vs. the other(s) across some time period, suggesting a form of neural multiplexing of different stimuli (Caruso et al., 2018). Here we investigate (a) whether such coding fluctuations occur in early visual cortical areas; (b) how coding fluctuations are coordinated across the neural population; and (c) how coordinated coding fluctuations depend on the parsing of stimuli into separate vs. fused objects. We found coding fluctuations do occur in macaque V1 but only when the two stimuli form separate objects. Such separate objects evoked a novel pattern of V1 spike count (“noise”) correlations involving distinct distributions of positive and negative values. This bimodal correlation pattern was most pronounced among pairs of neurons showing the strongest evidence for coding fluctuations or multiplexing. Whether a given pair of neurons exhibited positive or negative correlations depended on whether the two neurons both responded better to the same object or had different object preferences. Distinct distributions of spike count correlations based on stimulus preferences were also seen in V4 for separate objects but not when two stimuli fused to form one object. These findings suggest multiple objects evoke different response dynamics than those evoked by single stimuli, lending support to the multiplexing hypothesis and suggesting a means by which information about multiple objects can be preserved despite the apparent coarseness of sensory coding.Significance StatementHow the brain separates information about multiple objects despite overlap in the neurons responsive to each item is not well understood. Here we show that some neurons in V1 exhibit coding fluctuations in response to two objects, and that these coding fluctuations are coordinated at the population level in ways that are not observed for single objects. Broadly similar results were obtained in V4. These response dynamics lend support to the hypothesis that information about individual objects may be multiplexed across the neural population, preserving information about each item despite the coarseness of sensory coding.

Summary Visual attention allows observers to change the influence of different parts of a visual scene on their behavior, suggesting that information can be flexibly shared between visual cortex and neurons involved in decision making. We investigated the neural substrate of flexible information routing by analyzing the activity of populations of visual neurons in the medial temporal area (MT) and oculo-motor neurons in the superior colliculus (SC) while rhesus monkeys switched spatial attention. We demonstrated that attention increases the efficacy of visuomotor communication: trial-to-trial variability in SC population activity could be better predicted by the activity of the MT population (and vice versa) when attention was directed toward their joint receptive fields. Surprisingly, this improvement in prediction was not explained by changes in the dimensionality of the shared subspace or in the magnitude of local or shared pairwise noise correlations. These results lay a foundation for future theoretical and experimental studies into how visual attention can flexibly change information flow between sensory and decision neurons.

Neuronal population responses to sensory stimuli are remarkably flexible. The responses of neurons in visual cortex depend on stimulus properties (e.g. contrast), processes that affect all stages of visual processing (e.g. adaptation), and cognitive processes (e.g attention or task switching). The effects of all of these processes on trial-averaged responses of individual neurons are well-described by divisive normalization, in which responses are scaled by the total stimulus drive. Normalization describes how a staggering variety of sensory, cognitive, and motor processes affect individual neurons (Carandini and Heeger, 2012), but whether different normalization processes could be mediated by the same mechanism remains poorly understood. We and others recently showed that attention has low rank effects on the covariability of populations of neurons in visual area V4 (Rabinowitz et al., 2015; Ecker et al., 2016; Huang et al., 2019), which strongly constrains mechanistic models mechanism (Huang et al., 2019). We hypothesized that measuring changes in population covariability associated with other normalization processes could clarify whether they might share a mechanism. Our experimental design included measurements in multiple visual areas using four normalization processes. We found that contrast, adaptation, attention, and task switching affect the responses of populations of neurons in primate visual cortex in a similarly low rank way. These results suggest that a given circuit uses a common mechanism to perform many forms of normalization and likely reflect a general principle that applies to a wide range of brain areas and sensory, cognitive, or motor processes.

Normalization, which divisively scales neuronal responses to multiple stimuli, is thought to underlie many sensory, motor, and cognitive processes. In every study where it has been investigated, neurons measured in the same brain area under identical conditions exhibit a range of normalization, ranging from suppression by nonpreferred stimuli (strong normalization) to additive responses to combinations of stimuli (no normalization). Normalization has been hypothesized to arise from interactions between neuronal populations, either in the same or different brain areas, but current models of normalization are not mechanistic and focus on trial-averaged responses. To gain insight into the mechanisms underlying normalization, we examined interactions between neurons that exhibit different degrees of normalization. We recorded from multiple neurons in three cortical areas while rhesus monkeys viewed superimposed drifting gratings. We found that neurons showing strong normalization shared less trial-to-trial variability with other neurons in the same cortical area and more variability with neurons in other cortical areas than did units with weak normalization. Furthermore, the cortical organization of normalization was not random: neurons recorded on nearby electrodes tended to exhibit similar amounts of normalization. Together, our results suggest that normalization reflects a neuron's role in its local network and that modulatory factors like normalization share the topographic organization typical of sensory tuning properties.

The way that correlated trial-to-trial variability between pairs of neurons in the same brain area (termed spike count or noise correlation, r SC) depends on stimulus or task conditions can constrain models of cortical circuits and of the computations performed by networks of neurons ([Cohen and Kohn, 2011][1]). In visual cortex, r SC tends not to depend on stimulus properties ([Kohn and Smith, 2005][2]; [Huang and Lisberger, 2009][3]) but does depend on cognitive factors like visual attention ([Cohen and Maunsell, 2009][4]; [Mitchell et al., 2009][5]). However, neurons across visual areas respond to any visual stimulus or contribute to any perceptual decision, and the way that information from multiple areas is combined to guide perception is unknown. To gain insight into these issues, we recorded simultaneously from neurons in two areas of visual cortex (primary visual cortex, V1, and the middle temporal area, MT) while rhesus monkeys viewed different visual stimuli in different attention conditions. We found that correlations between neurons in different areas depend on stimulus and attention conditions in very different ways than do correlations within an area. Correlations across, but not within, areas depend on stimulus direction and the presence of a second stimulus, and attention has opposite effects on correlations within and across areas. This observed pattern of cross-area correlations is predicted by a normalization model where MT units sum V1 inputs that are passed through a divisive nonlinearity. Together, our results provide insight into how neurons in different areas interact and constrain models of the neural computations performed across cortical areas. SIGNIFICANCE STATEMENT Correlations in the responses of pairs of neurons within the same cortical area have been a subject of growing interest in systems neuroscience. However, correlated variability between different cortical areas is likely just as important. We recorded simultaneously from neurons in primary visual cortex and the middle temporal area while rhesus monkeys viewed different visual stimuli in different attention conditions. We found that correlations between neurons in different areas depend on stimulus and attention conditions in very different ways than do correlations within an area. The observed pattern of cross-area correlations was predicted by a simple normalization model. Our results provide insight into how neurons in different areas interact and constrain models of the neural computations performed across cortical areas. [1]: #ref-8 [2]: #ref-16 [3]: #ref-15 [4]: #ref-9 [5]: #ref-23

Visual attention dramatically improves individuals’ ability to see and modulates the responses of neurons in every known visual and oculomotor area, but whether such modulations can account for perceptual improvements is unclear. We measured the relationship between populations of visual neurons, oculomotor neurons and behavior during detection and discrimination tasks. We found that neither of the two prominent hypothesized neuronal mechanisms underlying attention (which concern changes in information coding and the way sensory information is read out) provide a satisfying account of the observed behavioral improvements. Instead, our results are more consistent with the hypothesis that attention reshapes the representation of attended stimuli to more effectively influence behavior. Our results suggest a path toward understanding the neural underpinnings of perception and cognition in health and disease by analyzing neuronal responses in ways that are constrained by behavior and interactions between brain areas. Ruff and Cohen find that the prominent hypotheses about how attention improves perception do not account for behavioral improvements. Instead, their results suggest that attention reshapes sensory representations so the relevant information guides behavior.

Understanding how cognitive processes affect the responses of sensory neurons may clarify the relationship between neuronal population activity and behavior. However, tools for analyzing neuronal activity have not kept up with technological advances in recording from large neuronal populations. Here, we describe prevalent hypotheses of how cognitive processes affect sensory neurons, driven largely by a model based on the activity of single neurons or pools of neurons as the units of computation. We then use simple simulations to expand this model to a new conceptual framework that focuses on subspaces of population activity as the relevant units of computation, uses comparisons between brain areas or to behavior to guide analyses of these subspaces, and suggests that population activity is optimized to decode the large variety of stimuli and tasks that animals encounter in natural behavior. This framework provides new ways of understanding the ever-growing quantity of recorded population activity data.

The way that humans and animals perceive the lightness of an object depends on its physical luminance as well as its surrounding context. While neuronal responses throughout the visual pathway are modulated by context, the relationship between neuronal responses and lightness perception is poorly understood. We searched for a neuronal mechanism of lightness by recording responses of neuronal populations in monkey primary visual cortex (V1) and area V4 to stimuli that produce a lightness illusion in humans, in which the lightness of a disk depends on the context in which it is embedded. We found that the way individual units encode the luminance (or equivalently for our stimuli, contrast) of the disk and its context is extremely heterogeneous. This motivated us to ask whether the population representation in either V1 or V4 satisfies three criteria: 1) disk luminance is represented with high fidelity, 2) the context surrounding the disk is also represented, and 3) the representations of disk luminance and context interact to create a representation of lightness that depends on these factors in a manner consistent with human psychophysical judgments of disk lightness. We found that populations of units in both V1 and V4 fulfill the first two criteria, but that we cannot conclude that the two types of information in either area interact in a manner that clearly predicts human psychophysical measurements: the interpretation of our population measurements depends on how subsequent areas read out lightness from the population responses.New & NoteworthyA core question in visual neuroscience is how the brain extracts stable representations of object properties from the retinal image. We searched for a neuronal mechanism of lightness perception by determining whether the responses of neuronal populations in primary visual cortex and area V4 could account for a lightness illusion measured using human psychophysics. Our results suggest that comparing psychophysics with population recordings will yield insight into neuronal mechanisms underlying a variety of perceptual phenomena.

Models of divisive normalization can explain the trial-averaged responses of neurons in sensory, association, and motor areas under a wide range of conditions, including how visual attention changes the gains of neurons in visual cortex. Attention, like other modulatory processes, is also associated with changes in the extent to which pairs of neurons share trial-to-trial variability. We showed recently that in addition to decreasing correlations between similarly tuned neurons within the same visual area, attention increases correlations between neurons in primary visual cortex (V1) and the middle temporal area (MT) and that an extension of a classic normalization model can account for this correlation increase. One of the benefits of having a descriptive model that can account for many physiological observations is that it can be used to probe the mechanisms underlying processes such as attention. Here, we use electrical microstimulation in V1 paired with recording in MT to provide causal evidence that the relationship between V1 and MT activity is nonlinear and is well described by divisive normalization. We then use the normalization model and recording and microstimulation experiments to show that the attention dependence of V1-MT correlations is better explained by a mechanism in which attention changes the weights of connections between V1 and MT than by a mechanism that modulates responses in either area. Our study shows that normalization can explain interactions between neurons in different areas and provides a framework for using multiarea recording and stimulation to probe the neural mechanisms underlying neuronal computations.

Recent studies have shown that cognitive factors such as spatial and feature-based attention, learning, and task-switching can change the extent to which the trial-to-trial variability in the responses of neurons in sensory cortex is shared between pairs of neurons (for review, see Cohen and Kohn, 2011). Global cognitive factors related to concentration, motivation, effort, arousal, or alertness also affect performance on perceptual tasks and the responses of individual neurons in many cortical areas (Spitzer et al., 1988; Spitzer and Richmond, 1991; Motter, 1993; Bichot et al., 2001; Hasegawa et al., 2004; Boudreau et al., 2006; Niwa et al., 2012). The question of how global cognitive factors affect correlated response variability is important because these factors likely vary both across and within all psychophysical and physiological studies. Furthermore, global cognitive factors might provide a convenient platform for studying the neuronal mechanisms underlying how cognitive factors affect correlated variability because they can be manipulated easily without training complex perceptual tasks. We recorded simultaneously from groups of neurons in visual area V4 while rhesus monkeys performed a contrast discrimination task whose difficulty changed in blocks of trials. We found that correlated variability decreased when the task was more difficult, even when the visual stimuli were far outside the receptive fields of the recorded neurons. Our results suggest that studying global cognitive factors might provide a general framework for studying how cognitive factors affect the responses of neurons throughout sensory cortex.

Visual attention enhances the responses of visual neurons that encode the attended location. Several recent studies showed that attention also decreases correlations between fluctuations in the responses of pairs of neurons (termed spike count correlation or rSC). The previous results are consistent with two hypotheses. Attention–related changes in rate and rSC might be linked (perhaps through a common mechanism), so that attention always decreases rSC. Alternately, attention might either increase or decrease rSC, possibly depending on the role the neurons play in the behavioral task. We recorded simultaneously from dozens of neurons in area V4 while monkeys performed a discrimination task. We found strong evidence in favor of the second hypothesis, showing that attention can flexibly increase or decrease correlations, depending on whether the neurons provide evidence for the same or opposite perceptual decisions. These results place important constraints on models of the neuronal mechanisms underlying cognitive factors.

Face recognition is a complex cognitive process that requires distinguishable neuronal representations of individual faces. Previous functional magnetic resonance imaging (fMRI) studies using the “fMRI-adaptation” technique have suggested the existence of face-identity representations in face-selective regions, including the fusiform face area (FFA). Here, we present face-identity adaptation findings that are not well explained in terms of face-identity representations. We performed blood-oxygen level–dependent (BOLD) fMRI measurements, while participants viewed familiar faces that were shown repeatedly throughout the experiment. We found decreased activation for repeated faces in face-selective regions, as expected based on previous studies. However, we found similar effects in regions that are not face-selective, including the parahippocampal place area (PPA) and early visual cortex (EVC). These effects were present for exact-image (same view and lighting) as well as different-image (different view and/or lighting) repetition, but more widespread for exact-image repetition. Given the known functional properties of PPA and EVC, it appears unlikely that they contain domain-specific face-identity representations. Alternative interpretations include general attentional effects and carryover of activation from connected regions. These results remind us that fMRI stimulus-change effects can have a range of causes and do not provide conclusive evidence for a neuronal representation of the changed stimulus property.

SummaryInferior temporal (IT) object representations have been intensively studied in monkeys and humans, but representations of the same particular objects have never been compared between the species. Moreover, IT's role in categorization is not well understood. Here, we presented monkeys and humans with the same images of real-world objects and measured the IT response pattern elicited by each image. In order to relate the representations between the species and to computational models, we compare response-pattern dissimilarity matrices. IT response patterns form category clusters, which match between man and monkey. The clusters correspond to animate and inanimate objects; within the animate objects, faces and bodies form subclusters. Within each category, IT distinguishes individual exemplars, and the within-category exemplar similarities also match between the species. Our findings suggest that primate IT across species may host a common code, which combines a categorical and a continuous representation of objects.

UNLABELLED Visual attention, which improves perception of attended locations or objects, has long been known to affect many aspects of the responses of neuronal populations in visual cortex. There are two nonmutually exclusive hypotheses concerning the neuronal mechanisms that underlie these perceptual improvements. The first hypothesis, that attention improves the information encoded by a population of neurons in a particular cortical area, has considerable physiological support. The second hypothesis is that attention improves perception by selectively communicating relevant visual information. This idea has been tested primarily by measuring interactions between neurons on very short timescales, which are mathematically nearly independent of neuronal interactions on longer timescales. We tested the hypothesis that attention changes the way visual information is communicated between cortical areas on longer timescales by recording simultaneously from neurons in primary visual cortex (V1) and the middle temporal area (MT) in rhesus monkeys. We used two independent and complementary approaches. Our correlative experiment showed that attention increases the trial-to-trial response variability that is shared between the two areas. In our causal experiment, we electrically microstimulated V1 and found that attention increased the effect of stimulation on MT responses. Together, our results suggest that attention affects both the way visual stimuli are encoded within a cortical area and the extent to which visual information is communicated between areas on behaviorally relevant timescales. SIGNIFICANCE STATEMENT Visual attention dramatically improves the perception of attended stimuli. Attention has long been thought to act by selecting relevant visual information for further processing. It has been hypothesized that this selection is accomplished by increasing communication between neurons that encode attended information in different cortical areas. We recorded simultaneously from neurons in primary visual cortex and the middle temporal area while rhesus monkeys performed an attention task. We found that attention increased shared variability between neurons in the two areas and that attention increased the effect of microstimulation in V1 on the firing rates of MT neurons. Our results provide support for the hypothesis that attention increases communication between neurons in different brain areas on behaviorally relevant timescales.

Neurons in sensory cortical areas are tuned to multiple dimensions, or features, of their sensory space. Understanding how single neurons represent multiple features is of great interest for determining the informative dimensions of the neurons' response, the decoding algorithms appropriate for extracting this information from the neuronal population, and for determining where specific transformations occur along the visual hierarchy. Despite the established role of cortical area MT in judgments of motion and depth, it is not known how individual neurons jointly encode the two dimensions. We investigated the joint tuning of individual MT neurons for two visual features: direction of motion and binocular disparity, an important depth cue. We found that a separable, multiplicative combination of tuning for the two features can account for more than 90% of the variance in the joint tuning function for over 91% of MT neurons. These results suggest 1) that each feature can be read out independently from MT by simply averaging across the population without regard to the other feature and 2) that the inseparable representations seen in subsequent areas, such as MST, must be computed beyond MT. Intriguingly, we found that the remaining nonseparable component of the joint tuning function often manifested as small but systematic changes in the neurons' preferences for one feature as the other one was varied. We believe this reflects the local columnar organization of tuning for direction and binocular disparity in MT, indicating that joint tuning may provide a new tool with which to probe functional architecture.

Objective. A traditional goal of neural recording with extracellular electrodes is to isolate action potential waveforms of an individual neuron. Recently, in brain–computer interfaces (BCIs), it has been recognized that threshold crossing events of the voltage waveform also convey rich information. To date, the threshold for detecting threshold crossings has been selected to preserve single-neuron isolation. However, the optimal threshold for single-neuron identification is not necessarily the optimal threshold for information extraction. Here we introduce a procedure to determine the best threshold for extracting information from extracellular recordings. We apply this procedure in two distinct contexts: the encoding of kinematic parameters from neural activity in primary motor cortex (M1), and visual stimulus parameters from neural activity in primary visual cortex (V1). Approach. We record extracellularly from multi-electrode arrays implanted in M1 or V1 in monkeys. Then, we systematically sweep the voltage detection threshold and quantify the information conveyed by the corresponding threshold crossings. Main Results. The optimal threshold depends on the desired information. In M1, velocity is optimally encoded at higher thresholds than speed; in both cases the optimal thresholds are lower than are typically used in BCI applications. In V1, information about the orientation of a visual stimulus is optimally encoded at higher thresholds than is visual contrast. A conceptual model explains these results as a consequence of cortical topography. Significance. How neural signals are processed impacts the information that can be extracted from them. Both the type and quality of information contained in threshold crossings depend on the threshold setting. There is more information available in these signals than is typically extracted. Adjusting the detection threshold to the parameter of interest in a BCI context should improve our ability to decode motor intent, and thus enhance BCI control. Further, by sweeping the detection threshold, one can gain insights into the topographic organization of the nearby neural tissue.

Human inferior temporal cortex contains category-selective visual regions, including the fusiform face area (FFA) and the parahippocampal place area (PPA). These regions are defined by their greater category-average activation to the preferred category (faces and places, respectively) relative to nonpreferred categories. The approach of investigating category-average activation has left unclear to what extent category selectivity holds for individual object images. Here we investigate single-image activation profiles to address (1) whether each image from the preferred category elicits greater activation than any image outside the preferred category (categorical ranking), (2) whether there are activation differences within and outside the preferred category (gradedness), and (3) whether the activation profile falls off continuously across the category boundary or exhibits a discontinuity at the boundary (category step). We used functional magnetic resonance imaging to measure the activation elicited in the FFA and PPA by each of 96 object images from a wide range of categories, including faces and places, but also humans and animals, and natural and manmade objects. Results suggest that responses in FFA and PPA exhibit almost perfect categorical ranking, are graded within and outside the preferred category, and exhibit a category step. The gradedness within the preferred category was more pronounced in FFA; the category step was more pronounced in PPA. These findings support the idea that these regions have category-specific functions, but are also consistent with a distributed object representation emphasizing categories while still distinguishing individual images.

Voxel-Based Morphometry (VBM) has been used for several years to study differences in brain structure between populations. Recently, a longitudinal version of VBM has been used to show changes in gray matter associated with relatively short periods of training. In the present study we use fMRI and three different standard implementations of longitudinal VBM: SPM2, FSL, and SPM5 to assess functional and structural changes associated with a simple learning task. Behavioral and fMRI data clearly showed a significant learning effect. However, initially positive VBM results were found to be inconsistent across minor perturbations of the analysis technique and ultimately proved to be artifactual. When alignment biases were controlled for and recommended statistical procedures were used, no significant changes in grey matter density were found. This work, initially intended to show structural and functional changes with learning, rather demonstrates some of the potential pitfalls of existing longitudinal VBM methods and prescribes that these tools be applied and interpreted with extreme caution.

Perceptual decision making typically entails the processing of sensory signals, the formation of a decision, and the planning and execution of a motor response. Although recent studies in monkeys and humans have revealed possible neural mechanisms for perceptual decision making, much less is known about how the decision is subsequently transformed into a motor action and whether or not the decision is represented at an abstract level, i.e., independently of the specific motor response. To address this issue, we used functional MRI to monitor changes in brain activity while human subjects discriminated the direction of motion in random-dot visual stimuli that varied in coherence and responded with either button presses or saccadic eye movements. We hypothesized that areas representing decision variables should respond more to high- than to low-coherence stimuli independent of the motor system used to express a decision. Four areas were found that fulfilled this condition: left posterior dorsolateral prefrontal cortex (DLPFC), left posterior cingulate cortex, left inferior parietal lobule, and left fusifom/parahippocampal gyrus. We previously found that, when subjects made categorical decisions about degraded face and house stimuli, left posterior DLPFC showed a greater response to high- relative to low-coherence stimuli. Furthermore, the left posterior DLPFC appears to perform a comparison of signals from sensory processing areas during perceptual decision making. These data suggest that the involvement of left posterior DLPFC in perceptual decision making transcends both task and response specificity, thereby enabling a flexible link among sensory evidence, decision, and action.