Search

We present the first gesture recognition system implemented end-to-end on event-based hardware, using a TrueNorth neurosynaptic processor to recognize hand gestures in real-time at low power from events streamed live by a Dynamic Vision Sensor (DVS). The biologically inspired DVS transmits data only when a pixel detects a change, unlike traditional frame-based cameras which sample every pixel at a fixed frame rate. This sparse, asynchronous data representation lets event-based cameras operate at much lower power than frame-based cameras. However, much of the energy efficiency is lost if, as in previous work, the event stream is interpreted by conventional synchronous processors. Here, for the first time, we process a live DVS event stream using TrueNorth, a natively event-based processor with 1 million spiking neurons. Configured here as a convolutional neural network (CNN), the TrueNorth chip identifies the onset of a gesture with a latency of 105 ms while consuming less than 200 mW. The CNN achieves 96.5% out-of-sample accuracy on a newly collected DVS dataset (DvsGesture) comprising 11 hand gesture categories from 29 subjects under 3 illumination conditions.

Deep networks are now able to achieve human-level performance on a broad spectrum of recognition tasks. Independently, neuromorphic computing has now demonstrated unprecedented energy-efficiency through a new chip architecture based on spiking neurons, low precision synapses, and a scalable communication network. Here, we demonstrate that neuromorphic computing, despite its novel architectural primitives, can implement deep convolution networks that i) approach state-of-the-art classification accuracy across 8 standard datasets, encompassing vision and speech, ii) perform inference while preserving the hardware's underlying energy-efficiency and high throughput, running on the aforementioned datasets at between 1200 and 2600 frames per second and using between 25 and 275 mW (effectively > 6000 frames / sec / W) and iii) can be specified and trained using backpropagation with the same ease-of-use as contemporary deep learning. For the first time, the algorithmic power of deep learning can be merged with the efficiency of neuromorphic processors, bringing the promise of embedded, intelligent, brain-inspired computing one step closer., 7 pages, 6 figures

Mental imagery occurs “when a representation of the type created during the initial phases of perception is present but the stimulus is not actually being perceived.” How does the capability to perform mental imagery arise? Extending the idea that imagery arises from learned associations, we propose that mental rotation, a specific form of imagery, could arise through the mechanism of sequence learning–that is, by learning to regenerate the sequence of mental images perceived while passively observing a rotating object. To demonstrate the feasibility of this proposal, we constructed a simulated nervous system and embedded it within a behaving humanoid robot. By observing a rotating object, the system learns the sequence of neural activity patterns generated by the visual system in response to the object. After learning, it can internally regenerate a similar sequence of neural activations upon briefly viewing the static object. This system learns to perform a mental rotation task in which the subject must determine whether two objects are identical despite differences in orientation. As with human subjects, the time taken to respond is proportional to the angular difference between the two stimuli. Moreover, as reported in humans, the system fills in intermediate angles during the task, and this putative mental rotation activates the same pathways that are activated when the system views physical rotation. This work supports the proposal that mental rotation arises through sequence learning and the idea that mental imagery aids perception through learned associations, and suggests testable predictions for biological experiments.

Effective visual object recognition requires mechanisms to bind object features (e.g. color, shape and motion) while distinguishing distinct objects. Synchronously active neuronal circuits among reentrantly connected cortical areas may provide a basis for visual binding. To assess the potential of this mechanism, we have constructed a mobile brain-based device, Darwin VIII, which is guided by simulated analogues of cortical and sub-cortical areas required for visual processing, decision-making, reward and motor responses. These simulated areas are reentrantly connected and each area contains neuronal units representing both the mean activity level and the relative timing of the activity of groups of neurons. Darwin VIII learns to discriminate among multiple objects with shared visual features and associates 'target' objects with innately preferred auditory cues. We observed the co-activation of globally distributed neuronal circuits that corresponded to distinct objects in Darwin VIII's visual field. These circuits, which are constrained by a reentrant neuroanatomy and modulated by behavior and synaptic plasticity, are necessary for successful discrimination. By situating Darwin VIII in a rich real-world environment involving continual changes in the size and location of visual stimuli due to self-generated movement, and by recording its behavioral and neuronal responses in detail, we were able to show that reentrant connectivity and dynamic synchronization provide an effective mechanism for binding the features of visual objects.

Introduction Within the past few decades, the nature of consciousness has become a central issue in neuroscience, and it is increasingly the focus of both theoretical and empirical work. Studying consciousness is vital to developing an understanding of human perception and behavior, of our relationships with one another, and of our relationships with other potentially conscious animals. Although the study of consciousness through the construction of artificial models is a recent innovation, the advantages of such an approach are clear. First, models allow us to investigate consciousness in ways that are currently not feasible using human subjects or other animals. Second, an artifact that exhibits the necessary and sufficient properties of consciousness may conceivably be the forerunner of a new and very useful class of neuromorphic robots. A model of consciousness must take into account current theories of its biological bases. Although the field of artificial consciousness is a new one, it is striking how little attention has been given to modeling mechanisms. Instead, great – and perhaps undue – emphasis has been placed on purely phenomenological models. Many of these models are strongly reductionist in aim and fail to specify neural mechanisms.

In order to respond appropriately to environmental stimuli, organisms must integrate over time spatiotemporal signals that reflect object motion and self-movement. One possible mechanism to achieve this spatiotemporal transformation is to delay or lag neural responses. This paper reviews our recent modeling work testing the sufficiency of delayed responses in the nervous system in two different behavioral tasks: (1) Categorizing spatiotemporal tactile cues with thalamic ''lag'' cells and downstream coincidence detectors, and (2) Predictive motor control was achieved by the cerebellum through a delayed eligibility trace rule at cerebellar synapses. Since the timing of these neural signals must closely match real-world dynamics, we tested these ideas using the brain based device (BBD) approach in which a simulated nervous system is embodied in a robotic device. In both tasks, biologically inspired neural simulations with delayed neural responses were critical for successful behavior by the device.

The cerebellum is known to be critical for accurate adaptive control and motor learning. We propose here a mechanism by which the cerebellum may replace reflex control with predictive control. This mechanism is embedded in a learning rule (the delayed eligibility trace rule) in which synapses onto a Purkinje cell or onto a cell in the deep cerebellar nuclei become eligible for plasticity only after a fixed delay from the onset of suprathreshold presynaptic activity. To investigate the proposal that the cerebellum is a general-purpose predictive controller guided by a delayed eligibility trace rule, a computer model based on the anatomy and dynamics of the cerebellum was constructed. It contained components simulating cerebellar cortex and deep cerebellar nuclei, and it received input from a middle temporal visual area and the inferior olive. The model was incorporated in a real-world brain-based device (BBD) built on a Segway robotic platform that learned to traverse curved paths. The BBD learned which visual motion cues predicted impending collisions and used this experience to avoid path boundaries. During learning, the BBD adapted its velocity and turning rate to successfully traverse various curved paths. By examining neuronal activity and synaptic changes during this behavior, we found that the cerebellar circuit selectively responded to motion cues in specific receptive fields of simulated middle temporal visual areas. The system described here prompts several hypotheses about the relationship between perception and motor control and may be useful in the development of general-purpose motor learning systems for machines.

The primary visual cortex (V1) in primates is known to perform edge analysis. A new neural network model of V1 is proposed that integrates three of the most prominent features of V1 architecture-complex-cells, long-range horizontal connections formed by Hebbian learning, and feature maps. The utility of the model is demonstrated on the problem of extracting edges from gray-scale photographs. This biologically based model outperforms the Canny edge operator with hysteresis when tested on a variety of gray-scale photographs with the local edge coherence metric of Kitchen and Rosenfeld (1981).

The common octopus, or Octopus vulgaris, has the largest nervous system of any invertebrate, and has been shown to possess learning and memory capabilities that in many ways rival those of some vertebrates [1]. Nevertheless, the neural architecture of this cephalopod mollusk differs markedly from that of any vertebrate. Investigating the differences and similarities between the neural architecture—or connectome—of the octopus and mammals, such as the mouse, may lead to deep insights into the computational principles underlying animal cognition. The octopus brain provides some unique advantages for anatomical research, since its axons are generally thick and unmyelinated, allowing traditional staining methods, such as Golgi, to be used effectively. With this in mind, we first imaged the brain using the Knife-Edge Scanning Microscope [2], a custom serial sectioning microscope that can image large blocks of tissue (1 cm3) at sub-micrometer resolution. We imaged large portions of the octopus subesophageal mass (SUB) and the optic lobe (OL) which were stained using Golgi. In order to extract the geometry of the neuronal morphology, we used our Maximum Intensity Projection (MIP)-based tracing algorithm [3]. The imaging results are shown in Figure 1(a-d), and tracing results are shown in 1(e). Although quite preliminary, to our knowledge this is the first time large volumes of the octopus brain have been imaged at sub-micrometer resolution, allowing us to resolve many of the processes that make up the neural network. We expect that this pilot study and the more detailed investigations to follow will allow fruitful comparisons of the neural circuitries of individual octopuses with different ecological life histories, as well as of animals that have been exposed to a variety of neurodegenerative insults. Moreover, such explorations will engender a greater understanding of how functional neural architecture is altered by learning in invertebrates such as the octopus and vertebrates such as the mouse. In sum, this approach should contribute greatly to our understanding of the computational architecture of invertebrates and ultimately provide insights into the differences between invertebrate and vertebrate cognitive capabilities. Figure 1 Octopus subesophageal mass (SUB) and optic lobe (OL) imaged with the KESM (a–d), and tracing results (e). Scale (block width): (a) 1.44 mm, (b) 0.72 mm, (c) 1.44 mm, (d-f) 76.8 μm. Voxel resolution: 0.6 μm x 0.7 μm x 1.0 ...

Patterns of activity in brains are commonly composed of temporal sequences of periods with steady-state firing rates lasting several hundred milliseconds separated by sharp transitions during movement [1], perception [2], and remembering [3]. Although network models involving mean-firing-rate neurons have been used to generate sequential neural activity [1], spiking networks with such capability require further exploration. We describe how large-scale Winner-Take-All (WTA) spiking networks can be coupled together and trained to generate such sequential neural activity. These networks are composed of conductance-based excitatory and inhibitory spiking neurons [4]. Model synapses were subject to short-term synaptic plasticity and spike-timing dependent plasticity (STDP). Each network was a Center-Annular-Surround (CAS) network, a variant of center-surround networks that we have found to effectively generate WTA dynamics in large-scale networks of spiking neurons. Two CAS networks were coupled together reentrantly to form a network capable of sequence learning and recall. We found that networks of this sort can be trained to respond to a sequence of sensory cues by generating temporally ordered patterns of neuronal activity. The patterns consist of brief steady states separated by sharp transitions that resemble those observed experimentally. After training, synaptic changes resulting from STDP acting on connections between the coupled networks formed a link between temporally adjacent patterns of neural activity within the sequence. Figure ​Figure11 reflects an analysis of spiking data from this simulated network trained with a repeating sequence of eight cues. Each row in the figure plots the match score over time to one of the eight patterns of neural activity corresponding to one of the cues. White is a perfect match, while black indicates a complete mismatch. The last training repeat is from t = 24 to t = 32. Subsequently, external cues were removed and network activity continued, cycling through all eight patterns until another input cue was presented. Then the activity pattern changed to the activity pattern appropriate to the cue presented (see at t = 37), continuing the sequence when the cue was removed. The system was robust with respect to various initial conditions. Figure 1 After training, a large-scale network of approximately 4,000 spiking neurons transitions between activity patterns, in the absence of external cues, reflecting the learned sequence We also used the present model to control specific motor sequences in a robotic device. The population activity pattern in this modeled nervous system has similarities to that observed in primate frontal cortex during multi-segmented limb movements.