Search

We present a novel adaptive multimodal intensity-event algorithm to optimize an overall objective of object tracking under bit rate constraints for a host-chip architecture. The chip is a computationally resource-constrained device acquiring high-resolution intensity frames and events, while the host is capable of performing computationally expensive tasks. We develop a joint intensity-neuromorphic event rate-distortion compression framework with a quadtree (QT)-based compression of intensity and events scheme. The goal of this compression framework is to optimally allocate bits to the intensity frames and neuromorphic events based on the minimum distortion at a given communication channel capacity. The data acquisition on the chip is driven by the presence of objects of interest in the scene as detected by an object detector. The most informative intensity and event data are communicated to the host under rate constraints so that the best possible tracking performance is obtained. The detection and tracking of objects in the scene are done on the distorted data at the host. Intensity and events are jointly used in a fusion framework to enhance the quality of the distorted images, in order to improve the object detection and tracking performance. The performance assessment of the overall system is done in terms of the multiple object tracking accuracy (MOTA) score. Compared with using intensity modality only, there is an improvement in MOTA using both these modalities in different scenarios.

The Tauc–Lorentz–Urbach (TLU) dispersion model allows us to build a dielectric function from only a few parameters. However, this dielectric function is non-analytic and presents some mathematical drawbacks. As a consequence of this issue, the model becomes inaccurate. In the present work, we will adopt a procedure to conveniently transform the TLU model into a self-consistent dispersion model. The transformation involves the integration of the original TLU imaginary dielectric function ϵ2 by using a Lorentzian-type function of semi-width, Γ. This novel model is analytic and obeys the other necessary mathematical requirements of the optical constants of solid-state materials. The main difference with the non-analytic TLU model occurs at values of the photon energy near or lower than that of the bandgap energy (within the Urbach absorption region). In particular, this new model allows us to reliably extend the optical characterization of amorphous-semiconductor thin films within the limit to zero photon energy. To the best of our knowledge, this is the first time that the analytic TLU model has been successfully used to accurately determine the optical constants of unhydrogenated a-Si films using only their normal-incidence transmission spectra.

We present an easy and flexible calibration approach for small field of view Deflectometry systems to be used in VR headsets. Our method replaces all required calibration targets and mirrors with a small (phone) screen to perform intrinsic and extrinsic camera calibration as well as extrinsic calibration of the whole setup.

Optical coherence tomography (OCT) is an optical technique which allows for volumetric visualization of the internal structures of translucent materials. Additional information can be gained by measuring the rate of signal attenuation in depth. Techniques have been developed to estimate the rate of attenuation on a voxel by voxel basis. This depth resolved attenuation analysis gives insight into tissue structure and organization in a spatially resolved way. However, the presence of speckle in the OCT measurement causes the attenuation coefficient image to contain unrealistic fluctuations and makes the reliability of these images at the voxel level poor. While the distribution of speckle in OCT images has appeared in literature, the resulting voxelwise corruption of the attenuation analysis has not. In this work, the estimated depth resolved attenuation coefficient from OCT data with speckle is shown to be approximately exponentially distributed. After this, a prior distribution for the depth resolved attenuation coefficient is derived for a simple system using statistical mechanics. Finally, given a set of depth resolved estimates which were made from OCT data in the presence of speckle, a posterior probability distribution for the true voxelwise attenuation coefficient is derived and a Bayesian voxelwise estimator for the coefficient is given. These results are demonstrated in simulation and validated experimentally.

Many visual and robotics tasks in real-world scenarios rely on robust handling of high speed motion and high dynamic range (HDR) with effectively high spatial resolution and low noise. Such stringent requirements, however, cannot be directly satisfied by a single imager or imaging modality, rather by multi-modal sensors with complementary advantages. In this paper, we address high performance imaging by exploring the synergy between traditional frame-based sensors with high spatial resolution and low sensor noise, and emerging event-based sensors with high speed and high dynamic range. We introduce a novel computational framework, termed Guided Event Filtering (GEF), to process these two streams of input data and output a stream of super-resolved yet noise-reduced events. To generate high quality events, GEF first registers the captured noisy events onto the guidance image plane according to our flow model. it then performs joint image filtering that inherits the mutual structure from both inputs. Lastly, GEF re-distributes the filtered event frame in the space-time volume while preserving the statistical characteristics of the original events. When the guidance images under-perform, GEF incorporates an event self-guiding mechanism that resorts to neighbor events for guidance. We demonstrate the benefits of GEF by applying the output high quality events to existing event-based algorithms across diverse application categories, including high speed object tracking, depth estimation, high frame-rate video synthesis, and super resolution/HDR/color image restoration.

Scientific literature contains large volumes of complex, unstructured figures that are compound in nature (i.e. composed of multiple images, graphs, and drawings). Separation of these compound figures is critical for information retrieval from these figures. In this paper, we propose a new strategy for compound figure separation, which decomposes the compound figures into constituent subfigures while preserving the association between the subfigures and their respective caption components. We propose a two-stage framework to address the proposed compound figure separation problem. In particular, the subfigure label detection module detects all subfigure labels in the first stage. Then, in the subfigure detection module, the detected subfigure labels help to detect the subfigures by optimizing the feature selection process and providing the global layout information as extra features. Extensive experiments are conducted to validate the effectiveness and superiority of the proposed framework, which improves the detection precision by 9%.

With several advantages over conventional RGB cameras, event cameras have provided new opportunities for tackling visual tasks under challenging scenarios with fast motion, high dynamic range, and/or power constraint. Yet unlike image/video compression, the performance of event compression algorithm is far from satisfying and practical. The main challenge for compressing events is the unique event data form, i.e., a stream of asynchronously fired event tuples each encoding the 2D spatial location, timestamp, and polarity (denoting an increase or decrease in brightness). Since events only encode temporal variations, they lack spatial structure which is crucial for compression. To address this problem, we propose a novel event compression algorithm based on a quad tree (QT) segmentation map derived from the adjacent intensity images. The QT informs 2D spatial priority within the 3D space-time volume. In the event encoding step, events are first aggregated over time to form polarity-based event histograms. The histograms are then variably sampled via Poisson Disk Sampling prioritized by the QT based segmentation map. Next, differential encoding and run length encoding are employed for encoding the spatial and polarity information of the sampled events, respectively, followed by Huffman encoding to produce the final encoded events. Our Poisson Disk Sampling based Lossy Event Compression (PDS-LEC) algorithm performs rate-distortion based optimal allocation. On average, our algorithm achieves greater than 6x compression compared to the state of the art., 8 main pages