Search

Generalist models for cellular segmentation, like Cellpose, provide good out-of-the-box results for many types of images. However, such models do not allow users to adapt the segmentation style to their specific needs and may perform sub-optimally for test images that are very different from the training images. Here we introduce Cellpose 2.0, a new package which includes an ensemble of diverse pretrained models as well as a human-in-the-loop pipeline for quickly prototyping new specialist models. We show that specialist models pretrained on the Cellpose dataset can achieve state-of-the-art segmentation on new image categories with very little user-provided training data. Models trained on 500-1000 segmented regions-of-interest (ROIs) performed nearly as well as models trained on entire datasets with up to 200,000 ROIs. A human-in-the-loop approach further reduced the required user annotations to 100-200 ROIs, while maintaining state-of-the-art segmentation performance. This approach enables a new generation of specialist segmentation models that can be trained on new image types with only 1-2 hours of user effort. We provide software tools including an annotation GUI, a model zoo and a human-in-the-loop pipeline to facilitate the adoption of Cellpose 2.0.

Spike sorting is the computational process of extracting the firing times of single neurons from recordings of local electrical fields. This is an important but hard problem in neuroscience, complicated by the non-stationarity of the recordings and the dense overlap in electrical fields between nearby neurons. To solve the spike sorting problem, we have continuously developed over the past eight years a framework known as Kilosort. This paper describes the various algorithmic steps introduced in different versions of Kilosort. We also report the development of Kilosort4, a new version with substantially improved performance due to new clustering algorithms inspired by graph-based approaches. To test the performance of Kilosort, we developed a realistic simulation framework which uses densely sampled electrical fields from real experiments to generate non-stationary spike waveforms and realistic noise. We find that nearly all versions of Kilosort outperform other algorithms on a variety of simulated conditions, and Kilosort4 performs best in all cases, correctly identifying even neurons with low amplitudes and small spatial extents in high drift conditions.

Recent studies in mice have shown that orofacial behaviors drive a large fraction of neural activity across the brain. To understand the nature and function of these signals, we need better computational models to characterize the behaviors and relate them to neural activity. Here we developed Facemap, a framework consisting of a keypoint tracking algorithm and a deep neural network encoder for predicting neural activity. We used the Facemap keypoints as input for the deep neural network to predict the activity of ∼50,000 simultaneously-recorded neurons and in visual cortex we doubled the amount of explained variance compared to previous methods. Our keypoint tracking algorithm was more accurate than existing pose estimation tools, while the inference speed was several times faster, making it a powerful tool for closed-loop behavioral experiments. The Facemap tracker was easy to adapt to data from new labs, requiring as few as 10 annotated frames for near-optimal performance. We used Facemap to find that the neuronal activity clusters which were highly driven by behaviors were more spatially spread-out across cortex. We also found that the deep keypoint features inferred by the model had time-asymmetrical state dynamics that were not apparent in the raw keypoint data. In summary, Facemap provides a stepping stone towards understanding the function of the brainwide neural signals and their relation to behavior.

Neuromatch Academy (NMA) designed and ran a fully online 3-week Computational Neuroscience Summer School for 1757 students with 191 teaching assistants (TAs) working in virtual inverted (or flipped) classrooms and on small group projects. Fourteen languages, active community management, and low cost allowed for an unprecedented level of inclusivity and universal accessibility.

A surprising finding of recent studies in mouse is the dominance of widespread movement-related activity throughout the brain, including in early sensory areas. In awake subjects, failing to account for movement risks misattributing movement-related activity to other (e.g., sensory or cognitive) processes. In this article, we (1) review task designs for separating task-related and movement-related activity, (2) review three “case studies” in whichnotconsidering movement would have resulted in critically different interpretations of neuronal function, and (3) discuss functional couplings that may prevent us from ever fully isolating sensory, motor, and cognitive-related activity. Our main thesis is that neural signals related to movement are ubiquitous, and therefore ought to be considered first and foremost when attempting to correlate neuronal activity with task-related processes.

Neuromatch Academy (https://academy.neuromatch.io; (van Viegen et al., 2021)) was designed as an online summer school to cover the basics of computational neuroscience in three weeks. The materials cover dominant and emerging computational neuroscience tools, how they complement one another, and specifically focus on how they can help us to better understand how the brain functions. An original component of the materials is its focus on modeling choices, i.e. how do we choose the right approach, how do we build models, and how can we evaluate models to determine if they provide real (meaningful) insight. This meta-modeling component of the instructional materials asks what questions can be answered by different techniques, and how to apply them meaningfully to get insight about brain function.

Advances in microscopy hold great promise for allowing quantitative and precise measurement of morphological and molecular phenomena at the single-cell level in bacteria; however, the potential of this approach is ultimately limited by the availability of methods to faithfully segment cells independent of their morphological or optical characteristics. Here, we present Omnipose, a deep neural network image-segmentation algorithm. Unique network outputs such as the gradient of the distance field allow Omnipose to accurately segment cells on which current algorithms, including its predecessor, Cellpose, produce errors. We show that Omnipose achieves unprecedented segmentation performance on mixed bacterial cultures, antibiotic-treated cells and cells of elongated or branched morphology. Furthermore, the benefits of Omnipose extend to non-bacterial subjects, varied imaging modalities and three-dimensional objects. Finally, we demonstrate the utility of Omnipose in the characterization of extreme morphological phenotypes that arise during interbacterial antagonism. Our results distinguish Omnipose as a powerful tool for characterizing diverse and arbitrarily shaped cell types from imaging data. This work was supported by the National Institutes of Health (AI080609 to J.D.M., GM128191 to P.A.W., R01-GM128191 to T.W.L. and T32-GM008268 to K.J.C.). L.R. and N.S. were funded by the European Research Council under the European Union’s Horizon 2020 Research and Innovation Programme (grant agreement no. 852201), the Spanish Ministry of Economy, Industry and Competitiveness to the EMBL partnership, the Centro de Excelencia Severo Ochoa (CEX2020-001049-S, MCIN/AEI /10.13039/501100011033), the CERCA Programme/Generalitat de Catalunya and the Spanish Ministry of Economy, Industry and Competitiveness Excelencia award PID2020-115189GB-I00. C.S. was funded by the Howard Hughes Medical Institute at the Janelia Research Campus. J.D.M. is an HHMI Investigator.

Advances in microscopy hold great promise for allowing quantitative and precise readouts of morphological and molecular phenomena at the single cell level in bacteria. However, the potential of this approach is ultimately limited by the availability of methods to perform unbiased cell segmentation, defined as the ability to faithfully identify cells independent of their morphology or optical characteristics. In this study, we present a new algorithm, Omnipose, which accurately segments samples that present significant challenges to current algorithms, including mixed bacterial cultures, antibiotic-treated cells, and cells of extended or branched morphology. We show that Omnipose achieves generality and performance beyond leading algorithms and its predecessor, Cellpose, by virtue of unique neural network outputs such as the gradient of the distance field. Finally, we demonstrate the utility of Omnipose in the characterization of extreme morphological phenotypes that arise during interbacterial antagonism and on the segmentation of non-bacterial objects. Our results distinguish Omnipose as a uniquely powerful tool for answering diverse questions in bacterial cell biology.

Neuromatch Academy (https://neuromatch.io/academy) was designed as an online summer school to cover the basics of computational neuroscience in three weeks. The materials cover dominant and emerging computational neuroscience tools, how they complement one another, and specifically focus on how they can help us to better understand how the brain functions. An original component of the materials is its focus on modeling choices, i.e. how do we choose the right approach, how do we build models, and how can we evaluate models to determine if they provide real (meaningful) insight. This meta-modeling component of the instructional materials asks what questions can be answered by different techniques, and how to apply them meaningfully to get insight about brain function.

Individual neurons in visual cortex provide the brain with unreliable estimates of visual features. It is not known whether the single-neuron variability is correlated across large neural populations, thus impairing the global encoding of stimuli. We recorded simultaneously from up to 50,000 neurons in mouse primary visual cortex (V1) and in higher order visual areas and measured stimulus discrimination thresholds of 0.35° and 0.37°, respectively, in an orientation decoding task. These neural thresholds were almost 100 times smaller than the behavioral discrimination thresholds reported in mice. This discrepancy could not be explained by stimulus properties or arousal states. Furthermore, behavioral variability during a sensory discrimination task could not be explained by neural variability in V1. Instead, behavior-related neural activity arose dynamically across a network of non-sensory brain areas. These results imply that perceptual discrimination in mice is limited by downstream decoders, not by neural noise in sensory representations.

Many biological applications require the segmentation of cell bodies, membranes and nuclei from microscopy images. Deep learning has enabled great progress on this problem, but current methods are specialized for images that have large training datasets. Here we introduce a generalist, deep learning-based segmentation method called Cellpose, which can precisely segment cells from a wide range of image types and does not require model retraining or parameter adjustments. We trained Cellpose on a new dataset of highly-varied images of cells, containing over 70,000 segmented objects. We also demonstrate a 3D extension of Cellpose which reuses the 2D model and does not require 3D-labelled data. To support community contributions to the training data, we developed software for manual labelling and for curation of the automated results, with optional direct upload to our data repository. Periodically retraining the model on the community-contributed data will ensure that Cellpose improves constantly.

Single neurons in visual cortex provide unreliable measurements of visual features due to their high trial-to-trial variability. It is not known if this “noise” extends its effects over large neural populations to impair the global encoding of stimuli. We recorded simultaneously from ∼20,000 neurons in mouse primary visual cortex (V1) and found that the neural populations had discrimination thresholds of ∼0.34° in an orientation decoding task. These thresholds were nearly 100 times smaller than those reported behaviorally in mice. The discrepancy between neural and behavioral discrimination could not be explained by the types of stimuli we used, by behavioral states or by the sequential nature of perceptual learning tasks. Furthermore, higher-order visual areas lateral to V1 could be decoded equally well. These results imply that the limits of sensory perception in mice are not set by neural noise in sensory cortex, but by the limitations of downstream decoders.

A neuronal population encodes information most efficiently when its activity is uncorrelated and high-dimensional, and most robustly when its activity is correlated and lower-dimensional. Here, we analyzed the correlation structure of natural image coding, in large visual cortical populations recorded from awake mice. Evoked population activity was high dimensional, with correlations obeying an unexpected power-law: the nth principal component variance scaled as 1/n. This was not inherited from the 1/f spectrum of natural images, because it persisted after stimulus whitening. We proved mathematically that the variance spectrum must decay at least this fast if a population code is smooth, i.e. if small changes in input cannot dominate population activity. The theory also predicts larger power-law exponents for lower-dimensional stimulus ensembles, which we validated experimentally. These results suggest that coding smoothness represents a fundamental constraint governing correlations in neural population codes.

Sensory cortices are active in the absence of external sensory stimuli. To understand the nature of this ongoing activity, we used two-photon calcium imaging to record from over 10,000 neurons in the visual cortex of mice awake in darkness while monitoring their behavior videographically. Ongoing population activity was multidimensional, exhibiting at least 100 significant dimensions, some of which were related to the spontaneous behaviors of the mice. The largest single dimension was correlated with the running speed and pupil area, while a 16-dimensional summary of orofacial behaviors could predict ∼45% of the explainable neural variance. Electrophysiological recordings with 8 simultaneous Neuropixels probes revealed a similar encoding of high-dimensional orofacial behaviors across multiple forebrain regions. Representation of motor variables continued uninterrupted during visual stimulus presentation, occupying dimensions nearly orthogonal to the stimulus responses. Our results show that a multidimensional representation of motor state is encoded across the forebrain, and is integrated with visual input by neuronal populations in primary visual cortex.

Vertical drift (Z-drift) is a major confound for neural recordings during behavior. The brain floats in liquid, and movements of tens of microns can easily occur, even in head-fixed animals. In the mouse, for instance, postural changes such as locomotion can cause vertical brain movements of up to 20 microns. This displacement creates an apparent change in the activity of neurons recorded with either electrode arrays or two-photon calcium imaging. Here, we present methods to estimate and correct the drift in both optical and electrical recordings. We demonstrate three methods to recover Z-drift in 2-photon calcium imaging. (1) Alignment to a densely-scanned reference volume (z-stack). (2) Estimation from a non-functional channel- such as tdTomato expressed in a neuronal subclass. (3) Estimation from changes in the shape of identified cells in functional recordings. We validate methods 2 and 3 by comparing to method 1, which provides ground truth. We then develop correction methods that remove the effects of Z-drift, and show that correlations of neuronal activity with running are significantly decreased. Finally, we develop a convenient online module for drift correction that eliminates Z-drift at sub-micron resolution. Z-drift also affects electrophysiological recordings. The amplitude and shape of extracellular action potentials changes when the electrode moves relative to the brain, and neurons may even disappear altogether from the set of recorded channels. Fortunately, new electrodes such as Neuropixels have dense arrays of channels, with inter-site spacings as low as 20um. We found that we could estimate the drift in extracellular recordings with linear electrodes by tracking neuronal waveform shifts, and corrected for it by spatially interpolating the raw data prior to spike sorting. In summary, the algorithms presented here provide effective methods to remove Z-drift, a major confound for neural recordings during behavioral experiments. We provide the code as part of the Suite2p and Kilosort pipelines.

Calcium imaging is a powerful method to record the activity of neural populations in many species, but inferring spike times from calcium signals is a challenging problem. We compared multiple approaches using multiple datasets with ground truth electrophysiology and found that simple non-negative deconvolution (NND) outperformed all other algorithms on out-of-sample test data. We introduce a novel benchmark applicable to recordings without electrophysiological ground truth, based on the correlation of responses to two stimulus repeats, and used this to show that unconstrained NND also outperformed the other algorithms when run on "zoomed out" datasets of ∼10,000 cell recordings from the visual cortex of mice of either sex. Finally, we show that NND-based methods match the performance of a supervised method based on convolutional neural networks while avoiding some of the biases of such methods, and at much faster running times. We therefore recommend that spikes be inferred from calcium traces using simple NND because of its simplicity, efficiency, and accuracy.

Calcium imaging is a powerful method to record the activity of neural populations, but inferring spike times from calcium signals is a challenging problem. We compared multiple approaches using multiple datasets with ground truth electrophysiology, and found that simple non-negative deconvolution (NND) outperformed all other algorithms. We introduce a novel benchmark applicable to recordings without electrophysiological ground truth, based on the correlation of responses to two stimulus repeats, and used this to show that unconstrained NND also outperformed the other algorithms when run on “zoomed out” datasets of ~10,000 cell recordings. Finally, we show that NND-based methods match the performance of a supervised method based on convolutional neural networks, while avoiding some of the biases of such methods, and at much faster running times. We therefore recommend that spikes be inferred from calcium traces using simple NND, due to its simplicity, efficiency and accuracy.

Two-photon microscopy of calcium-dependent sensors has enabled unprecedented recordings from vast populations of neurons. While the sensors and microscopes have matured over several generations of development, computational methods to process the resulting movies remain inefficient and can give results that are hard to interpret. Here we introduce Suite2p: a fast, accurate and complete pipeline that registers raw movies, detects active cells, extracts their calcium traces and infers their spike times. Suite2p runs on standard workstations, operates faster than real time, and recovers ~2 times more cells than the previous state-of-the-art method. Its low computational load allows routine detection of ~10,000 cells simultaneously with standard two-photon resonant-scanning microscopes. Recordings at this scale promise to reveal the fine structure of activity in large populations of neurons or large populations of subcellular structures such as synaptic boutons.

Cortical networks exhibit intrinsic dynamics that drive coordinated, large-scale fluctuations across neuronal populations and create noise correlations that impact sensory coding. To investigate the network-level mechanisms that underlie these dynamics, we developed novel computational techniques to fit a deterministic spiking network model directly to multi-neuron recordings from different rodent species, sensory modalities, and behavioral states. The model generated correlated variability without external noise and accurately reproduced the diverse activity patterns in our recordings. Analysis of the model parameters suggested that differences in noise correlations across recordings were due primarily to differences in the strength of feedback inhibition. Further analysis of our recordings confirmed that putative inhibitory neurons were indeed more active during desynchronized cortical states with weak noise correlations. Our results demonstrate that network models with intrinsically-generated variability can accurately reproduce the activity patterns observed in multi-neuron recordings and suggest that inhibition modulates the interactions between intrinsic dynamics and sensory inputs to control the strength of noise correlations. DOI: http://dx.doi.org/10.7554/eLife.19695.001, eLife digest Our brains contain billions of neurons, which are continually producing electrical signals to relay information around the brain. Yet most of our knowledge of how the brain works comes from studying the activity of one neuron at a time. Recently, studies of multiple neurons have shown that they tend to be active together in short bursts called “up” states, which are followed by periods in which they are less active called “down” states. When we are sleeping or under a general anesthetic, the neurons may be completely silent during down states, but when we are awake the difference in activity between the two states is usually less extreme. However, it is still not clear how the neurons generate these patterns of activity. To address this question, Stringer et al. studied the activity of neurons in the brains of awake and anesthetized rats, mice and gerbils. The experiments recorded electrical activity from many neurons at the same time and found a wide range of different activity patterns. A computational model based on these data suggests that differences in the degree to which some neurons suppress the activity of other neurons may account for this variety. Increasing the strength of these inhibitory signals in the model decreased the fluctuations in electrical activity across entire areas of the brain. Further analysis of the experimental data supported the model’s predictions by showing that inhibitory neurons – which act to reduce electrical activity in other neurons – were more active when there were fewer fluctuations in activity across the brain. The next step following on from this work would be to develop ways to build computer models that can mimic the activity of many more neurons at the same time. The models could then be used to interpret the electrical activity produced by many different kinds of neuron. This will enable researchers to test more sophisticated hypotheses about how the brain works. DOI: http://dx.doi.org/10.7554/eLife.19695.002

Equilibrium formally can be represented as an ensemble of uncoupled systems undergoing unbiased dynamics in which detailed balance is maintained. Many nonequilibrium processes can be described by suitable subsets of the equilibrium ensemble. Here, we employ the “weighted ensemble” (WE) simulation protocol [Huber and Kim, Biophys. J.1996, 70, 97–110] to generate equilibrium trajectory ensembles and extract nonequilibrium subsets for computing kinetic quantities. States do not need to be chosen in advance. The procedure formally allows estimation of kinetic rates between arbitrary states chosen after the simulation, along with their equilibrium populations. We also describe a related history-dependent matrix procedure for estimating equilibrium and nonequilibrium observables when phase space has been divided into arbitrary non-Markovian regions, whether in WE or ordinary simulation. In this proof-of-principle study, these methods are successfully applied and validated on two molecular systems: explicitly solvated methane association and the implicitly solvated Ala4 peptide. We comment on challenges remaining in WE calculations.

The “weighted ensemble” (WE) method, originally designed for non-equilibrium path sampling, can also be applied to equilibrium sampling [J. Chem. Phys. 133: 014110 (2010)]. WE is a parallel method with multiple trajectories coupled periodically through configuration space in a statistically rigorous way. We demonstrate the first applications of equilibrium WE to molecular systems. Because “ordinary” dynamics trajectories are employed, the approach can simultaneously yield rate constants for transitions among arbitrary states.

Neuron activity across the brain How is it that groups of neurons dispersed through the brain interact to generate complex behaviors? Three papers in this issue present brain-scale studies of neuronal activity and dynamics (see the Perspective by Huk and Hart). Allen et al. found that in thirsty mice, there is widespread neural activity related to stimuli that elicit licking and drinking. Individual neurons encoded task-specific responses, but every brain area contained neurons with different types of response. Optogenetic stimulation of thirst-sensing neurons in one area of the brain reinstated drinking and neuronal activity across the brain that previously signaled thirst. Gründemann et al. investigated the activity of mouse basal amygdala neurons in relation to behavior during different tasks. Two ensembles of neurons showed orthogonal activity during exploratory and nonexploratory behaviors, possibly reflecting different levels of anxiety experienced in these areas. Stringer et al. analyzed spontaneous neuronal firing, finding that neurons in the primary visual cortex encoded both visual information and motor activity related to facial movements. The variability of neuronal responses to visual stimuli in the primary visual area is mainly related to arousal and reflects the encoding of latent behavioral states. Science , this issue p. eaav3932 , p. eaav8736 , p. eaav7893 ; see also p. 236