Search

Appearance-based gaze estimation from RGB images provides relatively unconstrained gaze tracking from commonly available hardware. The accuracy of subject-independent models is limited partly by small intra-subject and large inter-subject variations in appearance, and partly by a latent subject-dependent bias. To improve estimation accuracy, we have previously proposed a gaze decomposition method that decomposes the gaze angle into the sum of a subject-independent gaze estimate from the image and a subject-dependent bias. Estimating the bias from images outperforms previously proposed calibration algorithms, unless the amount of calibration data is prohibitively large. This paper extends that work with a more complete characterization of the interplay between the complexity of the calibration dataset and estimation accuracy. In particular, we analyze the effect of the number of gaze targets, the number of images used per gaze target and the number of head positions in calibration data using a new NISLGaze dataset, which is well suited for analyzing these effects as it includes more diversity in head positions and orientations for each subject than other datasets. A better understanding of these factors enables low complexity high performance calibration. Our results indicate that using only a single gaze target and single head position is sufficient to achieve high quality calibration. However, it is useful to include variability in head orientation as the subject is gazing at the target. Our proposed estimator based on these studies (GEDDNet) outperforms state-of-the-art methods by more than $6.3\%$6.3%. One of the surprising findings of our work is that the same estimator yields the best performance both with and without calibration. This is convenient, as the estimator works well ”straight out of the box,” but can be improved if needed by calibration. However, this seems to violate the conventional wisdom that train and test conditions must be matched. To better understand the reasons, we provide a new theoretical analysis that specifies the conditions under which this can be expected. The dataset is available at http://nislgaze.ust.hk. Source code is available at https://github.com/HKUST-NISL/GEDDnet.

Vision-based autonomous driving through imitation learning mimics the behavior of human drivers by mapping driver view images to driving actions. This article shows that performance can be enhanced via the use of eye gaze. Previous research has shown that observing an expert’s gaze patterns can be beneficial for novice human learners. We show here that neural networks can also benefit. We trained a conditional generative adversarial network to estimate human gaze maps accurately from driver-view images. We describe two approaches to integrating gaze information into imitation networks: eye gaze as an additional input and gaze modulated dropout. Both significantly enhance generalization to unseen environments in comparison with a baseline vanilla network without gaze, but gaze-modulated dropout performs better. We evaluated performance quantitatively on both single images and in closed-loop tests, showing that gaze modulated dropout yields the lowest prediction error, the highest success rate in overtaking cars, the longest distance between infractions, lowest epistemic uncertainty, and improved data efficiency. Using Grad-CAM, we show that gaze modulated dropout enables the network to concentrate on task-relevant areas of the image. [ABSTRACT FROM AUTHOR]

An event-based camera has independent pixels that quickly respond to relative brightness changes and report them asynchronously with high temporal resolution (microsecond level). We present an event-by-event processing framework for sensing the velocity of a moving object and tracking it with an active event-based camera. This paper describes an event-by-event approach for estimating both local velocity and global velocity. Instead of accumulating events over a fixed period of time, we directly estimate the normal velocity of the pixel at which a new event occurs and update the estimate of the global velocity given the new estimate. Our proposed global velocity estimation algorithm is based on the Kalman filter which also provides a confidence measurement for the estimated velocity. Our active tracking system monitors the confidence of the velocity estimate and triggers a tracking command once the confidence reaches a certain threshold. We evaluated our velocity estimation algorithms and the active tracking system in our recently proposed simulator, AESIM. Our results demonstrate that our system can successfully track a moving object with low latency.

We present an algorithm that enables a robot to learn the visuomotor transformation from its joint angle space to visual space. The learned transformation can accurately predict location and shape of robot end effector's image projection. This paper extends past work by approximating the end effector by a planar region, rather than a point, in 3-D space, and through its use of spatially and temporally local, rather than global, measures of image consistency. Our robotic experiments demonstrate that the proposed algorithm can learn location and shape of the image region corresponding to the end effector, and how it deforms as the arm moves randomly in front of the camera. Our approach does not require that the end effector be identified with a specific marker. We also demonstrate that the region corresponding to the end effector can adapt to changes in the end effector shape.

We have developed a grid network that broadcasts spikes (all-or-none events) in a multichip neuromorphic system by relaying them from chip to chip. The grid is expandable because, unlike a bus, its capacity does not decrease as more chips are added. The multiple relays do not increase latency because the grid's cycle time is shorter than the bus. We describe an asynchronous relay implementation that automatically assigns chip addresses to indicate the source of spikes, encoded as word-serial address-events. This design, which is integrated on each chip, connects neurons at corresponding locations on each of the chips (pointwise connectivity) and supports oblivious, targeted, and excluded delivery of spikes. Results from two chips fabricated in 0.25-μm technology are presented, showing word-rates up to 45.4 M events/s. [ABSTRACT FROM AUTHOR]

Neurons in the mammalian primary visual cortex are selective along multiple stimulus dimensions, including retinal position, spatial frequency, and orientation. Neurons tuned to different stimulus features but the same retinal position are grouped into retinotopic arrays of hypercolumns. This paper describes a neuromorphic implementation of orientation hypercolwnns, which consists of a single silicon retina feeding multiple chips, each of which contains an array of neurons tuned to the same orientation and spatial frequency, but different retinal locations. All chips operate in continuous time, and communicate with each other using spikes transmitted by the address-event representation protocol. This system is modular in the sense that orientation coverage can be increased simply by adding more chips, and expandable in the sense that its output can be used to construct neurons tuned to other stimulus dimensions. We present measured results from the system, demonstrating neuronal selectivity along position, spatial frequency and orientation. We also demonstrate that the system supports recurrent feedback between neurons within one hypercolumn, even though they reside on different chips. The measured results from the system are in excellent concordance with theoretical predictions. [ABSTRACT FROM AUTHOR]

Reaction-diffusion dynamics have been proposed to explain pattern-formation behavior: in a variety of systems, e.g., chemical and biological. This paper describes a cellular neural network chip that exhibits spatially organized patterns of activity, which are formed by reaction diffusion. The chip contains four 32 × 32 cell arrays of transistors which are locally coupled and operate in weak inversion. Typical patterns consist of alternating regions of high- and low-drain currents with a preferred width, but no preferred orientation. Experimental, theoretical, and simulation results for this chip are in complete concordance, demonstrating that conventional very large-scale integration technology can be an ideal substrate for studying the spatio-temporal dynamics and applications of reaction-diffusion. The chip, which was fabricated in a 0.5-pm process, settles to steady-state patterns within several hundred microseconds and dissipates 10.55 mW. [ABSTRACT FROM AUTHOR]

This paper describes the electronic implementation of a four-layer cellular neural network architecture implementing two components of a functional model of neurons in the visual cortex: linear orientation selective filtering and half wave rectification. Separate ON and OFF layers represent the positive and negative outputs of two-phase quadrature Gabor-type filters, whose orientation and spatial-frequency tunings are electronically adjustable. To enable the construction of a multichip network to extract different orientations in parallel, the chip includes an address event representation (AER) transceiver that accepts and produces two-dimensional images that are rate encoded as spike trains. It also includes routing circuitry that facilitates point-to-point signal fan in and fan out. We present measured results from a 32 × 64 pixel prototype, which was fabricated in the TSMC0.25-μm process on a 3.84 by 2.54 mm die. Quiescent power dissipation is 3 mW and is determined primarily by the spike activity on the AER bus. Settling times are on the order of a few milliseconds. In comparison with a two-layer network implementing the same filters, this network results in a more symmetric circuit design with lower quiescent power dissipation, albeit at the expense of twice as many transistors. [ABSTRACT FROM AUTHOR]

This May 2004 issue of the journal IEEE Transactions on Circuits and System has been organized into sections that reflect some of the major themes that have arisen over recent years. Each section contains articles detailing advances in theory, application, design, and implementation consistent with the broad multidisciplinary scope of cellular neural/nonlinear network (CNN). The first section highlights the current state of the art in single-layer CNNs, where the state of each cell is a scalar. The second section highlights recent developments in multilayer CNNs, which can exhibit a richer range of dynamical behaviors than the single-layer CNN. The final section contains articles that pave the way for the future development of CNN technology and active wave computing.