Search

Appearance-based gaze estimation from RGB images provides relatively unconstrained gaze tracking from commonly available hardware. The accuracy of subject-independent models is limited partly by small intra-subject and large inter-subject variations in appearance, and partly by a latent subject-dependent bias. To improve estimation accuracy, we have previously proposed a gaze decomposition method that decomposes the gaze angle into the sum of a subject-independent gaze estimate from the image and a subject-dependent bias. Estimating the bias from images outperforms previously proposed calibration algorithms, unless the amount of calibration data is prohibitively large. This paper extends that work with a more complete characterization of the interplay between the complexity of the calibration dataset and estimation accuracy. In particular, we analyze the effect of the number of gaze targets, the number of images used per gaze target and the number of head positions in calibration data using a new NISLGaze dataset, which is well suited for analyzing these effects as it includes more diversity in head positions and orientations for each subject than other datasets. A better understanding of these factors enables low complexity high performance calibration. Our results indicate that using only a single gaze target and single head position is sufficient to achieve high quality calibration. However, it is useful to include variability in head orientation as the subject is gazing at the target. Our proposed estimator based on these studies (GEDDNet) outperforms state-of-the-art methods by more than 6.3%. One of the surprising findings of our work is that the same estimator yields the best performance both with and without calibration. This is convenient, as the estimator works well "straight out of the box," but can be improved if needed by calibration. However, this seems to violate the conventional wisdom that train and test conditions must be matched. To better understand the reasons, we provide a new theoretical analysis that specifies the conditions under which this can be expected. The dataset is available at http://nislgaze.ust.hk. Source code is available at https://github.com/HKUST-NISL/GEDDnet.

Stroke is the leading cause of adult disability. Robot-assisted rehabilitation systems show great promise for motor recovery after a stroke. In this work, we present a gazecontrolled robotic system for upper limb rehabilitation. Subjects perform a painting task in virtual reality. We designed a novel and challenging painting task to encourage motivation and engagement, as these are critical factors in treatment efficacy. Because the robotic system can be programmed to provide varying amounts of assistance or resistance to the subject, it can be applied to a wide range of patients at different phases of recovery. We describe here the system configured in two modes: resistive control and hierarchical control. The former is designed for later stages of recovery, where the patient's impaired limb has recovered some function. It can be configured to provide varying degrees of resistance by adjusting the properties of an admittance controller. The latter targets patients in more acute phases, where the impaired limb is less responsive. It provides a combination of assistive and corrective control. We pilot tested our system on 10 able-bodied subjects. Our results show that the system can provide varying degrees of resistive control, and that the integration of high level control modulated by gaze can improve engagement. These results suggest that the system may provide a more engaging environment for a wide range of rehabilitative therapies than currently available.

Gaze-based interfaces are especially useful for people with disabilities involving the upper limbs or hands. Typically, users select from a number of options (e.g. letters or commands) displayed on a screen by gazing at the desired option. However, in some applications, e.g. gaze-based driving, it may be dangerous to direct gaze away from the environment towards a separate display. In addition, a purely gaze based interface can present a high cognitive load to users, as gaze is not normally used for selection and/or control, but rather for other purposes, such as information gathering. To address these issues, this paper presents a cost-effective multi-modal system for gaze based driving which combines appearance-based gaze estimates derived from webcam images with push button inputs that trigger command execution. This system uses an intuitive "direct interface", where users determine the direction of motion by gazing in the corresponding direction in the environment. We have implemented the system for both wheelchair control and robotic teleoperation. The use of our system should provide substantial benefits for patients with severe motor disabilities, such as ALS, by providing them with a more natural and affordable method of wheelchair control. We compare the performance of our system to the more conventional and common "indirect" system where gaze is used to select commands from a separate display, showing that our system enables faster and more efficient navigation.

Animals utilize a variety of active sensing mechanisms to perceive the world around them. Echolocating bats are an excellent model for the study of active auditory localization. The big brown bat (Eptesicus fuscus), for instance, employs active head roll movements during sonar prey tracking. The function of head rolls in sound source localization is not well understood. Here, we propose an echolocation model with multi-axis head rotation to investigate the effect of active head roll movements on sound localization performance. The model autonomously learns to align the bat's head direction towards the target. We show that a model with active head roll movements better localizes targets than a model without head rolls. Furthermore, we demonstrate that active head rolls also reduce the time required for localization in elevation. Finally, our model offers key insights to sound localization cues used by echolocating bats employing active head movements during echolocation., Competing Interests: The authors have declared that no competing interests exist.

Vision-based autonomous driving through imitation learning mimics the behavior of human drivers by mapping driver view images to driving actions. This article shows that performance can be enhanced via the use of eye gaze. Previous research has shown that observing an expert's gaze patterns can be beneficial for novice human learners. We show here that neural networks can also benefit. We trained a conditional generative adversarial network to estimate human gaze maps accurately from driver-view images. We describe two approaches to integrating gaze information into imitation networks: eye gaze as an additional input and gaze modulated dropout. Both significantly enhance generalization to unseen environments in comparison with a baseline vanilla network without gaze, but gaze-modulated dropout performs better. We evaluated performance quantitatively on both single images and in closed-loop tests, showing that gaze modulated dropout yields the lowest prediction error, the highest success rate in overtaking cars, the longest distance between infractions, lowest epistemic uncertainty, and improved data efficiency. Using Grad-CAM, we show that gaze modulated dropout enables the network to concentrate on task-relevant areas of the image.

Eye typing is a hands-free method of human computer interaction, which is especially useful for people with upper limb disabilities. Users select a desired key by gazing at it in an image of a keyboard for a fixed dwell time. There is a tradeoff in selecting the dwell time; shorter dwell times lead to errors due to unintentional selections, while longer dwell times lead to a slow input speed. We propose to speed up eye typing while maintaining low error by dynamically adjusting the dwell time for each letter based on the past input history. More likely letters are assigned shorter dwell times. Our method is based on a probabilistic generative model of gaze, which enables us to assign dwell times using a principled model that requires only a few free parameters. We evaluate our model on both able-bodied subjects and subjects with a spinal cord injury (SCI). Compared to the standard dwell time method, we find consistent increases in typing speed in both cases. e.g., 41.8% faster typing for able-bodied subjects on a transcription task and 49.5% faster typing for SCI subjects in a chatbot task. We observed more inter-subject variability for SCI subjects.

Previous research has shown that exposure to Japanese gardens reduces physiological measures of stress, e.g. heart rate, in both healthy subjects and dementia patients. However, the correlation between subjects' physiological responses and their visual behavior while viewing the garden has not yet been investigated. To address this, we developed a system to collect simultaneous measurements of eye gaze and three physiological indicators of autonomic nervous system activity: electrocardiogram, blood volume pulse, and galvanic skin response. We recorded healthy subjects' physiological/behavioral responses when they viewed two environments (an empty courtyard and a Japanese garden) in two ways (directly or as a projected 2D photograph). Similar to past work, we found that differences in subject's physiological responses to the two environments when viewed directly, but not as a photograph. We also found differences in their behavioral responses. We quantified subject's behavioral responses using several gaze metrics commonly considered to be measures of engagement of focus: average fixation duration, saccade amplitude, spatial entropy and gaze transition entropy. We found decrease in gaze transition entropy, the only metric that accounts for both the spatial and temporal properties of gaze, to have a weak positive correlation with decrease in heart rate. This suggests a relationship between engagement/focus and relaxation. Finally, we found gender differences: females' gaze patterns were more spatially distributed and had higher transition entropy than males., Competing Interests: The author(s) declare(s) that the contents of the article are in agreement with the ethics described in http://biblio.unibe.ch/portale/elibrary/BOP/jemr/ethics.html and that there is no conflict of interest regarding the publication of this paper.

The development of vision during the first months of life is an active process that comprises the learning of appropriate neural representations and the learning of accurate eye movements. While it has long been suspected that the two learning processes are coupled, there is still no widely accepted theoretical framework describing this joint development. Here, we propose a computational model of the development of active binocular vision to fill this gap. The model is based on a formulation of the active efficient coding theory, which proposes that eye movements as well as stimulus encoding are jointly adapted to maximize the overall coding efficiency. Under healthy conditions, the model self-calibrates to perform accurate vergence and accommodation eye movements. It exploits disparity cues to deduce the direction of defocus, which leads to coordinated vergence and accommodation responses. In a simulated anisometropic case, where the refraction power of the two eyes differs, an amblyopia-like state develops in which the foveal region of one eye is suppressed due to inputs from the other eye. After correcting for refractive errors, the model can only reach healthy performance levels if receptive fields are still plastic, in line with findings on a critical period for binocular vision development. Overall, our model offers a unifying conceptual framework for understanding the development of binocular vision., Competing Interests: The authors declare no competing interest., (Copyright © 2020 the Author(s). Published by PNAS.)

We present a model for the autonomous and simultaneous learning of active binocular and motion vision. The model is based on the Active Efficient Coding (AEC) framework, a recent generalization of classic efficient coding theories to active perception. The model learns how to efficiently encode the incoming visual signals generated by an object moving in 3-D through sparse coding. Simultaneously, it learns how to produce eye movements that further improve the efficiency of the sensory coding. This learning is driven by an intrinsic motivation to maximize the system's coding efficiency. We test our approach on the humanoid robot iCub using simulations. The model demonstrates self-calibration of accurate object fixation and tracking of moving objects. Our results show that the model keeps improving until it hits physical constraints such as camera or motor resolution, or limits on its internal coding capacity. Furthermore, we show that the emerging sensory tuning properties are in line with results on disparity, motion, and motion-in-depth tuning in the visual cortex of mammals. The model suggests that vergence and tracking eye movements can be viewed as fundamentally having the same objective of maximizing the coding efficiency of the visual system and that they can be learned and calibrated jointly through AEC.

We propose a biologically inspired model that enables a humanoid robot to learn how to track its end effector by integrating visual and proprioceptive cues as it interacts with the environment. A key novel feature of this model is the incorporation of sensorimotor prediction, where the robot predicts the sensory consequences of its current body motion as measured by proprioceptive feedback. The robot develops the ability to perform smooth pursuit-like eye movements to track its hand, both in the presence and absence of visual input, and to track exteroceptive visual motions. Our framework makes a number of advances over past work. First, our model does not require a fiducial marker to indicate the robot hand explicitly. Second, it does not require the forward kinematics of the robot arm to be known. Third, it does not depend upon pre-defined visual feature descriptors. These are learned during interaction with the environment. We demonstrate that the use of prediction in multisensory integration enables the agent to incorporate the information from proprioceptive and visual cues better. The proposed model has properties that are qualitatively similar to the characteristics of human eye-hand coordination.

Recent research has shown that design principles inherent in a Japanese style garden can reduce measures of stress in both healthy and dementia patients. However, it was not clear how subjects' visual interaction with the scene affected their physiological responses. To address that, we developed a novel non-invasive system to collect synchronized measurements of eye gaze and physiological indicators of sympathetic neural activity: the electrocardiogram, the blood volume pulse and the galvanic skin response, as subjects view a garden environment. We characterized the visual engagement of subjects using the average fixation duration, the saccade amplitude and the gaze transition entropy. We find a statistically significant positive correlation between gaze transition entropy and mean heart rate change. Our results suggest that the visual engagement of subjects with their environment may influence their physiological responses to it: more engagement may lead to more relaxation. Our results also highlight the importance of taking into account the detailed spatio-temporal characteristics of the gaze trajectory.

Previous studies have demonstrated that exposure to a Japanese garden is a non-pharmacological measure to improve the behavioral symptoms of elderly people with dementia, and that Japanese gardens are significantly more effective than other environments. However, it is not clear whether Japanese gardens have similar effects in the young. To address this open question, we measured the physiological responses of university students when viewing a Japanese garden, and compared them to the same students' responses when viewing a control space. We measured three physiological indicators of autonomous nervous system (ANS) activity: the electrocardiograph (ECG), the blood volume pulse (BVP) and the galvanic skin response (GSR). Our results suggest that the Japanese garden does not have as calming an effect on younger subjects as observed previously in elderly subjects. However, students did respond more positively to the Japanese garden than to an unstructured space. Ambient temperature was found to be a critical factor affecting heart rate and heart rate variability, but not other measures.

Face recognition plays an import role in our daily lives. However, computer face recognition performance degrades dramatically with the presence of variations in illumination, head pose and occlusion. In contrast, the human brain can recognize target faces over a much wider range of conditions. In this paper, we investigate target face detection through electroencephalography (EEG). We address the problem of single-trial target-face detection in a rapid serial visual presentation (RSVP) paradigm. Whereas most previous approaches used support vector machines (SVMs), we use a convolutional neural network (CNN) to classify EEG signals when subjects view target and non-target face stimuli. The CNN outperforms the SVM algorithm, which is commonly used for event-related-potential (ERP) detection. We also compare the difference in performance when using animal stimuli. The proposed system can be potentially used in rapid face recognition system.

This paper investigates two types of eye movements: vergence and saccades. Vergence eye movements are responsible for bringing the images of the two eyes into correspondence, whereas saccades drive gaze to interesting regions in the scene. Control of both vergence and saccades develops during early infancy. To date, these two types of eye movements have been studied separately. Here, we propose a computational model of an active vision system that integrates these two types of eye movements. We hypothesize that incorporating a saccade strategy driven by bottom-up attention will benefit the development of vergence control. The integrated system is based on the active efficient coding framework, which describes the joint development of sensory-processing and eye movement control to jointly optimize the coding efficiency of the sensory system. In the integrated system, we propose a binocular saliency model to drive saccades based on learned binocular feature extractors, which simultaneously encode both depth and texture information. Saliency in our model also depends on the current fixation point. This extends prior work, which focused on monocular images and saliency measures that are independent of the current fixation. Our results show that the proposed saliency-driven saccades lead to better vergence performance and faster learning in the overall system than random saccades. Faster learning is significant because it indicates that the system actively selects inputs for the most effective learning. This work suggests that saliency-driven saccades provide a scaffold for the development of vergence control during infancy.

Eye tracking systems are typically divided into two categories: remote and mobile. Remote systems, where the eye tracker is located near the object being viewed by the subject, have the advantage of being less intrusive, but are typically used for tracking gaze points on fixed two dimensional (2D) computer screens. Mobile systems such as eye tracking glasses, where the eye tracker are attached to the subject, are more intrusive, but are better suited for cases where subjects are viewing objects in the three dimensional (3D) environment. In this paper, we describe how remote gaze tracking systems developed for 2D computer screens can be used to track gaze points in a 3D environment. The system is non-intrusive. It compensates for small head movements by the user, so that the head need not be stabilized by a chin rest or bite bar. The system maps the 3D gaze points of the user onto 2D images from a scene camera and is also located remotely from the subject. Measurement results from this system indicate that it is able to estimate gaze points in the scene camera to within one degree over a wide range of head positions.

This paper addresses the problem of estimating gaze location in the 3D environment using a remote eye tracker. Instead of relying only on data provided by the eye tracker, we investigate how to integrate gaze direction with the point-cloud-based representation of the scene provided by a Kinect sensor. The algorithm first combines the gaze vectors for the two eyes provided by the eye tracker into a single gaze vector emanating from a point in between the two eyes. The gaze target in the three dimensional environment is then identified by finding the point in the 3D point cloud that is closest to the gaze vector. Our experimental results demonstrate that the estimate of the gaze target location provided by this method is significantly better than that provided when considering gaze information alone. It is also better than two other methods for integrating point cloud information: (1) finding the 3D point closest to the gaze location as estimated by triangulating the gaze vectors from the two eyes, and (2) finding the 3D point with smallest average distance to the two gaze vectors considered individually. The proposed method has an average error of 1.7 cm in a workspace of 25 × 23 × 24 cm located at a distance of 60 cm from the user.

This paper describes novel event-based spatio-temporal features called time-surfaces and how they can be used to create a hierarchical event-based pattern recognition architecture. Unlike existing hierarchical architectures for pattern recognition, the presented model relies on a time oriented approach to extract spatio-temporal features from the asynchronously acquired dynamics of a visual scene. These dynamics are acquired using biologically inspired frameless asynchronous event-driven vision sensors. Similarly to cortical structures, subsequent layers in our hierarchy extract increasingly abstract features using increasingly large spatio-temporal windows. The central concept is to use the rich temporal information provided by events to create contexts in the form of time-surfaces which represent the recent temporal activity within a local spatial neighborhood. We demonstrate that this concept can robustly be used at all stages of an event-based hierarchical model. First layer feature units operate on groups of pixels, while subsequent layer feature units operate on the output of lower level feature units. We report results on a previously published 36 class character recognition task and a four class canonical dynamic card pip task, achieving near 100 percent accuracy on each. We introduce a new seven class moving face recognition task, achieving 79 percent accuracy.This paper describes novel event-based spatio-temporal features called time-surfaces and how they can be used to create a hierarchical event-based pattern recognition architecture. Unlike existing hierarchical architectures for pattern recognition, the presented model relies on a time oriented approach to extract spatio-temporal features from the asynchronously acquired dynamics of a visual scene. These dynamics are acquired using biologically inspired frameless asynchronous event-driven vision sensors. Similarly to cortical structures, subsequent layers in our hierarchy extract increasingly abstract features using increasingly large spatio-temporal windows. The central concept is to use the rich temporal information provided by events to create contexts in the form of time-surfaces which represent the recent temporal activity within a local spatial neighborhood. We demonstrate that this concept can robustly be used at all stages of an event-based hierarchical model. First layer feature units operate on groups of pixels, while subsequent layer feature units operate on the output of lower level feature units. We report results on a previously published 36 class character recognition task and a four class canonical dynamic card pip task, achieving near 100 percent accuracy on each. We introduce a new seven class moving face recognition task, achieving 79 percent accuracy.

Optokinetic nystagmus (OKN) is an involuntary eye movement responsible for stabilizing retinal images in the presence of relative motion between an observer and the environment. Fully understanding the development of OKN requires a neurally plausible computational model that accounts for the neural development and the behavior. To date, work in this area has been limited. We propose a neurally plausible framework for the joint development of disparity and motion tuning in the visual cortex and of optokinetic and vergence eye-movement behavior. To our knowledge, this framework is the first developmental model to describe the emergence of OKN in a behaving organism. Unlike past models, which were based on scalar models of overall activity in different neural areas, our framework models the development of the detailed connectivity both from the retinal input to the visual cortex and from the visual cortex to the motor neurons. This framework accounts for the importance of the development of normal vergence control and binocular vision in achieving normal monocular OKN behaviors. Because the model includes behavior, we can simulate the same perturbations as past experiments, such as artificially induced strabismus. The proposed model agrees both qualitatively and quantitatively with a number of findings from the literature on both binocular vision and the optokinetic reflex. Finally, our model makes quantitative predictions about OKN behavior using the same methods used to characterize OKN in the experimental literature.

We describe a hybrid brain computer interface that integrates gaze information from an eye tracker with brain activity information measured by electroencephalography (EEG). Users explicitly control the end effector of a robot arm to move in one of four directions using motor imagery to perform a pick and place task. Measurements of the natural eye gaze behavior of subjects is used to infer the instantaneous intent of the users based on the past gaze trajectory. This information is integrated with the output of the EEG classifier and contextual information about the environment probabilistically using Bayesian inference. Our experiments demonstrate that subjects can achieve 100% task completion within three minutes and that the integration of EEG and gaze information significantly improves performance over either cue in isolation.

We describe a two stage hidden Markov model based algorithm for inferring target identity in a 2D cursor control task where subjects are instructed to use a joystick to steer a cursor towards a target while avoiding obstacles. The first stage of the model converts a regularly sampled gaze trajectory into a sequence of fixations. The second stage then makes a determination of the end target in the cursor control task based on this sequence of fixations. In contrast to prior work, this two stage model allows for more accurate modelling of the natural eye gaze behavior of subjects, which in turn leads to increased accuracy and speed of target identification. This work demonstrates the importance of accurate gaze modelling, and paves the way for more natural and reliable hybrid Brain Computer Interfaces.

Primary visual cortical complex cells are thought to serve as invariant feature detectors and to provide input to higher cortical areas. We propose a single model for learning the connectivity required by complex cells that integrates two factors that have been hypothesized to play a role in the development of invariant feature detectors: temporal slowness and sparsity. This model, the generative adaptive subspace self-organizing map (GASSOM), extends Kohonen's adaptive subspace self-organizing map (ASSOM) with a generative model of the input. Each observation is assumed to be generated by one among many nodes in the network, each being associated with a different subspace in the space of all observations. The generating nodes evolve according to a first-order Markov chain and generate inputs that lie close to the associated subspace. This model differs from prior approaches in that temporal slowness is not an externally imposed criterion to be maximized during learning but, rather, an emergent property of the model structure as it seeks a good model of the input statistics. Unlike the ASSOM, the GASSOM does not require an explicit segmentation of the input training vectors into separate episodes. This enables us to apply this model to an unlabeled naturalistic image sequence generated by a realistic eye movement model. We show that the emergence of temporal slowness within the model improves the invariance of feature detectors trained on this input.

Changing disparity (CD) and interocular velocity difference (IOVD) are two possible mechanisms for stereomotion perception. We propose two neurally plausible models for the representation of motion-in-depth (MID) via the CD and IOVD mechanisms. These models create distributed representations of MID velocity as the responses from a population of neurons selective to different MID velocity. Estimates of perceived MID velocity can be computed from the population response. They can be applied directly to binocular image sequences commonly used to characterize MID perception in psychophysical experiments. Contrary to common assumptions, we find that the CD and IOVD mechanisms cannot be distinguished easily by random dot stereograms that disrupt correlations between the two eyes or through time. We also demonstrate that the assumed spatial connectivity between the units in these models can be learned through exposure to natural binocular stimuli. Our experiments with these developmental models of MID selectivity suggest that neurons selective to MID are more likely to develop via the CD mechanism than the IOVD mechanism., (Copyright © 2014 Elsevier B.V. All rights reserved.)

The efficient coding hypothesis posits that sensory systems of animals strive to encode sensory signals efficiently by taking into account the redundancies in them. This principle has been very successful in explaining response properties of visual sensory neurons as adaptations to the statistics of natural images. Recently, we have begun to extend the efficient coding hypothesis to active perception through a form of intrinsically motivated learning: a sensory model learns an efficient code for the sensory signals while a reinforcement learner generates movements of the sense organs to improve the encoding of the signals. To this end, it receives an intrinsically generated reinforcement signal indicating how well the sensory model encodes the data. This approach has been tested in the context of binocular vison, leading to the autonomous development of disparity tuning and vergence control. Here we systematically investigate the robustness of the new approach in the context of a binocular vision system implemented on a robot. Robustness is an important aspect that reflects the ability of the system to deal with unmodeled disturbances or events, such as insults to the system that displace the stereo cameras. To demonstrate the robustness of our method and its ability to self-calibrate, we introduce various perturbations and test if and how the system recovers from them. We find that (1) the system can fully recover from a perturbation that can be compensated through the system's motor degrees of freedom, (2) performance degrades gracefully if the system cannot use its motor degrees of freedom to compensate for the perturbation, and (3) recovery from a perturbation is improved if both the sensory encoding and the behavior policy can adapt to the perturbation. Overall, this work demonstrates that our intrinsically motivated learning approach for efficient coding in active perception gives rise to a self-calibrating perceptual system of high robustness.

We present a simple optimization criterion that leads to autonomous development of a sensorimotor feedback loop driven by the neural representation of the depth in the mammalian visual cortex. Our test bed is an active stereo vision system where the vergence angle between the two eyes is controlled by the output of a population of disparity-selective neurons. By finding a policy that maximizes the total response across the neuron population, the system eventually tracks a target as it moves in depth. We characterized the tracking performance of the resulting policy using objects moving both sinusoidally and randomly in depth. Surprisingly, the system can even learn how to track based on stimuli it cannot track: even though the closed loop 3 dB tracking bandwidth of the system is 0.3 Hz, correct tracking policies are learned for input stimuli moving as fast as 0.75 Hz.

Changing disparity is a possible cue for stereomotion perception. We propose the changing disparity energy model, a physiologically plausible model for neurons tuned to changing disparity. This model combines the disparity and motion energy models commonly used to model cortical neuron outputs. The model outputs are consistent with psychophysical experiments indicating that stereomotion speed discrimination thresholds for dynamic random dot stereograms are higher than for random dot stereograms. Thus, these experimental results are not necessarily strong evidence for the existence of an inter-ocular velocity difference cue. The model also predicts a relationship between the speed discrimination threshold ratio and the dot density.

In this paper, we propose an algorithm for disparity estimation from disparity energy neurons that seeks to maintain simplicity and biological plausibility, while also being based upon a formulation that enables us to interpret the model outputs probabilistically. We use the Bayes factor from statistical hypothesis testing to show that, in contradiction to the implicit assumption of many previously proposed biologically plausible models, a larger response from a disparity energy neuron does not imply more evidence for the hypothesis that the input disparity is close to the preferred disparity of the neuron. However, we find that the normalized response can be interpreted as evidence, and that information from different orientation channels can be combined by pooling the normalized responses. Based on this insight, we propose an algorithm for disparity estimation constructed out of biologically plausible operations. Our experimental results on real stereograms show that the algorithm outperforms a previously proposed coarse-to-fine model. In addition, because its outputs can be interpreted probabilistically, the model also enables us to identify occluded pixels or pixels with incorrect disparity estimates.

To support large numbers of model neurons, neuromorphic vision systems are increasingly adopting a distributed architecture, where different arrays of neurons are located on different chips or processors. Spike-based protocols are used to communicate activity between processors. The spike activity in the arrays depends on the input statistics as well as internal parameters such as time constants and gains. In this paper, we investigate strategies for automatically adapting these parameters to maintain a constant firing rate in response to changes in the input statistics. We find that under the constraint of maintaining a fixed firing rate, a strategy based upon updating the gain alone performs as well as an optimal strategy where both the gain and the time constant are allowed to vary. We discuss how to choose the time constant and propose an adaptive gain control mechanism whose operation is robust to changes in the input statistics. Our experimental results on a mobile robotic platform validate the analysis and efficacy of the proposed strategy.

The relative depth of objects causes small shifts in the left and right retinal positions of these objects, called binocular disparity. This letter describes an electronic implementation of a single binocularly tuned complex cell based on the binocular energy model, which has been proposed to model disparity-tuned complex cells in the mammalian primary visual cortex. Our system consists of two silicon retinas representing the left and right eyes, two silicon chips containing retinotopic arrays of spiking neurons with monocular Gabor-type spatial receptive fields, and logic circuits that combine the spike outputs to compute a disparity-selective complex cell response. The tuned disparity can be adjusted electronically by introducing either position or phase shifts between the monocular receptive field profiles. Mismatch between the monocular receptive field profiles caused by transistor mismatch can degrade the relative responses of neurons tuned to different disparities. In our system, the relative responses between neurons tuned by phase encoding are better matched than neurons tuned by position encoding. Our numerical sensitivity analysis indicates that the relative responses of phase-encoded neurons that are least sensitive to the receptive field parameters vary the most in our system. We conjecture that this robustness may be one reason for the existence of phase-encoded disparity-tuned neurons in biological neural systems.