Search

We present a novel adaptive multimodal intensity-event algorithm to optimize an overall objective of object tracking under bit rate constraints for a host-chip architecture. The chip is a computationally resource-constrained device acquiring high-resolution intensity frames and events, while the host is capable of performing computationally expensive tasks. We develop a joint intensity-neuromorphic event rate-distortion compression framework with a quadtree (QT)-based compression of intensity and events scheme. The goal of this compression framework is to optimally allocate bits to the intensity frames and neuromorphic events based on the minimum distortion at a given communication channel capacity. The data acquisition on the chip is driven by the presence of objects of interest in the scene as detected by an object detector. The most informative intensity and event data are communicated to the host under rate constraints so that the best possible tracking performance is obtained. The detection and tracking of objects in the scene are done on the distorted data at the host. Intensity and events are jointly used in a fusion framework to enhance the quality of the distorted images, in order to improve the object detection and tracking performance. The performance assessment of the overall system is done in terms of the multiple object tracking accuracy (MOTA) score. Compared with using intensity modality only, there is an improvement in MOTA using both these modalities in different scenarios.

The poor lateral and depth resolution of state-of-the-art 3D sensors based on the time-of-flight (ToF) principle has limited widespread adoption to a few niche applications. In this work, we introduce a novel sensor concept that provides ToF-based 3D measurements of real world objects and surfaces with depth precision up to 35 $\mu m$ μ m and point cloud densities commensurate with the native sensor resolution of standard CMOS/CCD detectors (up to several megapixels). Such capabilities are realized by combining the best attributes of continuous wave ToF sensing, multi-wavelength interferometry, and heterodyne interferometry into a single approach. We describe multiple embodiments of the approach, each featuring a different sensing modality and associated tradeoffs.

Many visual and robotics tasks in real-world scenarios rely on robust handling of high speed motion and high dynamic range (HDR) with effectively high spatial resolution and low noise. Such stringent requirements, however, cannot be directly satisfied by a single imager or imaging modality, rather by multi-modal sensors with complementary advantages. In this paper, we address high performance imaging by exploring the synergy between traditional frame-based sensors with high spatial resolution and low sensor noise, and emerging event-based sensors with high speed and high dynamic range. We introduce a novel computational framework, termed Guided Event Filtering (GEF), to process these two streams of input data and output a stream of super-resolved yet noise-reduced events. To generate high quality events, GEF first registers the captured noisy events onto the guidance image plane according to our flow model. it then performs joint image filtering that inherits the mutual structure from both inputs. Lastly, GEF re-distributes the filtered event frame in the space-time volume while preserving the statistical characteristics of the original events. When the guidance images under-perform, GEF incorporates an event self-guiding mechanism that resorts to neighbor events for guidance. We demonstrate the benefits of GEF by applying the output high quality events to existing event-based algorithms across diverse application categories, including high speed object tracking, depth estimation, high frame-rate video synthesis, and super resolution/HDR/color image restoration.

Scientific literature contains large volumes of complex, unstructured figures that are compound in nature (i.e. composed of multiple images, graphs, and drawings). Separation of these compound figures is critical for information retrieval from these figures. In this paper, we propose a new strategy for compound figure separation, which decomposes the compound figures into constituent subfigures while preserving the association between the subfigures and their respective caption components. We propose a two-stage framework to address the proposed compound figure separation problem. In particular, the subfigure label detection module detects all subfigure labels in the first stage. Then, in the subfigure detection module, the detected subfigure labels help to detect the subfigures by optimizing the feature selection process and providing the global layout information as extra features. Extensive experiments are conducted to validate the effectiveness and superiority of the proposed framework, which improves the detection precision by 9%.

We present a novel approach for accurate eye tracking as required, e.g., in VR/AR/MR headsets. Our method exploits the retrieved surface normals and dense 3D features extracted from deflectometry measurements to estimate the gazing direction.

This Feature Issue includes 19 articles that highlight advances in the field of Computational Optical Sensing and Imaging. Many of the articles were presented at the 2019 OSA Topical Meeting on Computational Optical Sensing and Imaging held in Munich, Germany, on June 24–27. Articles featured in the issue cover a broad array of topics ranging from imaging through scattering media, imaging round corners and compressive imaging to machine learning for recovery of images.

We present a novel computational imaging system with high resolution and low noise. Our system consists of a traditional video camera which captures high-resolution intensity images, and an event camera which encodes high-speed motion as a stream of asynchronous binary events. To process the hybrid input, we propose a unifying framework that first bridges the two sensing modalities via a noise-robust motion compensation model, and then performs joint image filtering. The filtered output represents the temporal gradient of the captured space-time volume, which can be viewed as motion-compensated event frames with high resolution and low noise. Therefore, the output can be widely applied to many existing event-based algorithms that are highly dependent on spatial resolution and noise robustness. In experimental results performed on both publicly available datasets as well as our contributing RGB-DAVIS dataset, we show systematic performance improvement in applications such as high frame-rate video synthesis, feature/corner detection and tracking, as well as high dynamic range image reconstruction.

Highly accelerated real-time cine MRI using compressed sensing (CS) is a promising approach to achieve high spatio-temporal resolution and clinically acceptable image quality in patients with arrhythmia and/or dyspnea. However, its lengthy image reconstruction time may hinder its clinical translation. The purpose of this study was to develop a neural network for reconstruction of non-Cartesian real-time cine MRI k-space data faster (

Nonlinear unmixing of hyperspectral reflectance data is one of the key problems in quantitative imaging of painted works of art. The approach presented is to interrogate a hyperspectral image cube by first decomposing it into a set of reflectance curves representing pure basis pigments and second to estimate the scattering and absorption coefficients of each pigment in a given pixel to produce estimates of the component fractions. This two-step algorithm uses a deep neural network to qualitatively identify the constituent pigments in any unknown spectrum and, based on the pigment(s) present and Kubelka-Munk theory to estimate the pigment concentration on a per-pixel basis. Using hyperspectral data acquired on a set of mock-up paintings and a well-characterized illuminated folio from the 15th century, the performance of the proposed algorithm is demonstrated for pigment recognition and quantitative estimation of concentration.

Visible hyperspectral imaging (HSI) is a fast and non-invasive imaging method that has been adapted by the field of conservation science to study painted surfaces. By collecting reflectance spectra from a 2D surface, the resulting 3D hyperspectral data cube contains millions of recorded spectra. While processing such large amounts of spectra poses an analytical and computational challenge, it also opens new opportunities to apply powerful methods of multivariate analysis for data evaluation. With the intent of expanding current data treatment of hyperspectral datasets, an innovative approach for data reduction and visualization is presented in this article. It uses a statistical embedding method known as t-distributed stochastic neighbor embedding (t-SNE) to provide a non-linear representation of spectral features in a lower 2D space. The efficiency of the proposed method for painted surfaces from cultural heritage is established through the study of laboratory prepared paint mock-ups, and medieval French illuminated manuscript.

Temporal Video Frame Synthesis (TVFS) aims at synthesizing novel frames at timestamps different from existing frames, which has wide applications in video codec, editing and analysis. In this paper, we propose a high framerate TVFS framework which takes hybrid input data from a low-speed frame-based sensor and a high-speed event-based sensor. Compared to frame-based sensors, event-based sensors report brightness changes at very high speed, which may well provide useful spatio-temoral information for high framerate TVFS. In our framework, we first introduce a differentiable forward model to approximate the physical sensing process, fusing the two different modes of data as well as unifying a variety of TVFS tasks, i.e., interpolation, prediction and motion deblur. We leverage autodifferentiation which propagates the gradients of a loss defined on the measured data back to the latent high framerate video. We show results with better performance compared to state-of-the-art. Second, we develop a deep learning-based strategy to enhance the results from the first step, which we refer as a residual "denoising" process. Our trained "denoiser" is beyond Gaussian denoising and shows properties such as contrast enhancement and motion awareness. We show that our framework is capable of handling challenging scenes including both fast motion and strong occlusions., Comment: 10 pages, 11 figures

The nascent field of indirect imaging is concerned with the recovery of information pertaining to objects that are beyond the line-of-sight (LoS) and hidden from view. Current approaches to indirect imaging are either limited in their ability to recover spatially resolved imagery (resolution of few centimeters at 1-meter standoff) or impose severe restrictions on the imaging geometry. The present work examines two approaches that recover spatial detail on hidden objects by exploiting spatial and spectral correlation in the light scattered by the objects. Experiments have demonstrated the ability to discern sub-millimeter spatial detail, on centimeter sized objects positioned 1-meter behind a wall.

In this paper, we propose an adaptive host-chip system for video acquisition constrained under a given bit rate to optimize object tracking performance. The chip is an imaging instrument with limited computational power consisting of a very high-resolution focal plane array (FPA) that transmits quadtree (QT)-segmented video frames to the host. The host has unlimited computational power for video analysis. We find the optimal QT decomposition to minimize a weighted rate distortion equation using the Viterbi algorithm. The weights are user-defined based on the class of objects to track. Faster R-CNN and a Kalman filter are used to detect and track the objects of interest respectively. We evaluate our architecture’s performance based on the Multiple Object Tracking Accuracy (MOTA).

Pigment identification and mapping gives us insight into an artists' material use, allows us to measure slow chemical changes in painted surfaces, and allows us to detect anachronistic uses of materials that can be associated with either forgeries or past restorations. Earlier work has demonstrated the potential of a dictionary-based reflectance approach for pigment classification. This technique identifies pigments by searching for the pigment combinations that best reproduce the measured reflectance curve. The prospect of pigment classification through modeling is attractive because it can be extended to a layered medium -- potentially opening a route to a depth-resolved pigment classification method. In this work, we investigate a layered pigment classification technique with a fused deep learning and optimization-based Kubelka-Munk framework. First, we discuss the efficacy of the algorithm in a thick, single-layer system. Specifically, we consider the impacts of layer thickness, total pigment concentration, and spectrally similar pigment combinations. Following a thorough discussion of the single layer problem, the system is generalized to multiple layers. Finally, as a concrete example, we use the two-layered system to demonstrate both the impacts of layer thickness and dictionary content on paint localization within the painting. Results of the algorithm are then shown for mock-up paintings for which the ground truth is known.

Recently, time-of-flight (ToF) sensors have emerged as a promising three-dimensional sensing technology that can be manufactured inexpensively in a compact size. However, current state-of-the-art ToF sensors suffer from low spatial resolution due to physical limitations in the fabrication process. In this paper, we analyze the ToF sensor’s output as a complex value coupling the depth and intensity information in a phasor representation. Based on this analysis, we introduce a novel multi-frame superresolution technique that can improve both spatial resolution in intensity and depth images simultaneously. We believe our proposed method can benefit numerous applications where high resolution depth sensing is desirable, such as precision automated navigation and collision avoidance.

In this paper, the problem of automatic nonlinear unmixing of hyperspectral reflectance data using works of art as test cases is described. We use a deep neural network to decompose a given spectrum quantitatively to the abundance values of pure pigments. We show that adding another step to identify the constituent pigments of a given spectrum leads to more accurate unmixing results. Towards this, we use another deep neural network to identify pigments first and integrate this information to different layers of the network used for pigment unmixing. As a test set, the hyperspectral images of a set of mock-up paintings consisting of a broad palette of pigment mixtures, and pure pigment exemplars, were measured. The results of the algorithm on the mock-up test set are reported and analyzed.

The risk of unauthorized remote access of streaming video from networked cameras underlines the need for stronger privacy safeguards. We propose a lens-free coded aperture camera system for human action recognition that is privacy-preserving. While coded aperture systems exist, we believe ours is the first system designed for action recognition without the need for image restoration as an intermediate step. Action recognition is done using a deep network that takes in as input, non-invertible motion features between pairs of frames computed using phase correlation and log-polar transformation. Phase correlation encodes translation while the log polar transformation encodes in-plane rotation and scaling. We show that the translation features are independent of the coded aperture design, as long as its spectral response within the bandwidth has no zeros. Stacking motion features computed on frames at multiple different strides in the video can improve accuracy. Preliminary results on simulated data based on a subset of the UCF and NTU datasets are promising. We also describe our prototype lens-free coded aperture camera system, and results for real captured videos are mixed., Comment: CVCOPS2019

This paper presents an adaptive image sampling algorithm based on Deep Learning (DL). The adaptive sampling mask generation network is jointly trained with an image inpainting network. The sampling rate is controlled in the mask generation network, and a binarization strategy is investigated to make the sampling mask binary. Besides the image sampling and reconstruction application, we show that the proposed adaptive sampling algorithm is able to speed up raster scan processes such as the X-Ray fluorescence (XRF) image scanning process. Recently XRF laboratory-based systems have evolved to lightweight and portable instruments thanks to technological advancements in both X-Ray generation and detection. However, the scanning time of an XRF image is usually long due to the long exposures requires (e.g., $100 \mu s-1ms$ per point). We propose an XRF image inpainting approach to address the issue of long scanning time, thus speeding up the scanning process while still maintaining the possibility to reconstruct a high quality XRF image. The proposed adaptive image sampling algorithm is applied to the RGB image of the scanning target to generate the sampling mask. The XRF scanner is then driven according to the sampling mask to scan a subset of the total image pixels. Finally, we inpaint the scanned XRF image by fusing the RGB image to reconstruct the full scan XRF image. The experiments show that the proposed adaptive sampling algorithm is able to effectively sample the image and achieve a better reconstruction accuracy than that of the existing methods., Comment: journal preprint v3

In this work, we propose using camera arrays coupled with coherent illumination as an effective method of improving spatial resolution in long distance images by a factor of ten and beyond. Recent advances in ptychography have demonstrated that one can image beyond the diffraction limit of the objective lens in a microscope. We demonstrate a similar imaging system to image beyond the diffraction limit in long range imaging. We emulate a camera array with a single camera attached to an $XY$ translation stage. We show that an appropriate phase retrieval based reconstruction algorithm can be used to effectively recover the lost high resolution details from the multiple low resolution acquired images. We analyze the effects of noise, required degree of image overlap, and the effect of increasing synthetic aperture size on the reconstructed image quality. We show that coherent camera arrays have the potential to greatly improve imaging performance. Our simulations show resolution gains of $10\times$ and more are achievable. Furthermore, experimental results from our proof-of-concept systems show resolution gains of $4\times - 7\times$ for real scenes. All experimental data and code is made publicly available on the project webpage. Finally, we introduce and analyze in simulation a new strategy to capture macroscopic Fourier Ptychography images in a single snapshot, albeit using a camera array.

We introduce a system that exploits the screen and front-facing camera of a mobile device to perform three-dimensional deflectometry-based surface measurements. In contrast to current mobile deflectometry systems, our method can capture surfaces with large normal variation and wide field of view (FoV). We achieve this by applying automated multi-view panoramic stitching algorithms to produce a large FoV normal map from a hand-guided capture process without the need for external tracking systems, like robot arms or fiducials. The presented work enables 3D surface measurements of specular objects ’in the wild’ with a system accessible to users with little to no technical imaging experience. We demonstrate high-quality 3D surface measurements without the need for a calibration procedure. We provide experimental results with our prototype Deflectometry system and discuss applications for computer vision tasks such as object detection and recognition.

We present a Bayesian approach for 3D image reconstruction of an extended object imaged with multi-focus microscopy (MFM). MFM simultaneously captures multiple sub-images of different focal planes to provide 3D information of the sample. The naive method to reconstruct the object is to stack the sub-images along the $z$ -axis, but the result suffers from poor resolution in the $z$ -axis. The maximum a posteriori framework provides a way to reconstruct a 3D image according to its observation model and prior knowledge. It jointly estimates the 3D image and the model parameters. Experimental results with synthetic and real experimental data show that it enables the high-quality 3D reconstruction of an extended object from MFM.

Historical paper often contains features embedded in its structure that are invisible under standard viewing conditions. These features (watermarks, laid lines, and chain lines) can provide valuable information about a sheet’s provenance. Standard methods of reproducing watermarks, such as beta-radiography and low-voltage x-rays, are costly and time intensive, and therefore inaccessible to many institutions or individuals. In this work we introduce an inexpensive prototype whose elements are a light table and a consumer-grade photographic camera. For a given document we acquire an image with light emitted by a light table passing through the document and two images of the front and the back side with ambient light. The images are then processed to suppress the printed elements and isolate the watermark. The proposed method is capable of recovering images of watermarks similar to the ones obtained with standard methods while being a non-destructive, rapid, easy to operate, and inexpensive method.

Continuous wave time-of-flight (ToF) cameras have been rapidly gaining widespread adoption in many applications due to their cost effectiveness, simplicity, and compact size. However, the current generation of ToF cameras suffers from low spatial resolution due to physical fabrication limitations. In this paper, we propose an imaging architecture to achieve high spatial resolution ToF imaging using optical multiplexing and compressive sensing (CS). Our approach is based on the observation that, while depth is non-linearly related to ToF pixel measurements, a phasor representation of captured images results in a linear image formation model. We utilize this property to develop a CS-based technique that is used to recover high resolution 3D images. Based on the proposed architecture, we developed a prototype 1-megapixel compressive ToF camera that achieves as much as 4 x improvement in spatial resolution. We believe that our proposed architecture provides a simple and low-cost solution to improve the spatial resolution of ToF and related sensors.

Ptychography is an imaging technique which aims to recover the complex-valued exit wavefront of an object from a set of its diffraction pattern magnitudes. Ptychography is one of the most popular techniques for sub-30 nanometer imaging as it does not suffer from the limitations of typical lens based imaging techniques. The object can be reconstructed from the captured diffraction patterns using iterative phase retrieval algorithms. Over time many algorithms have been proposed for iterative reconstruction of the object based on manually derived update rules. In this paper, we adapt automatic differentiation framework to solve practical and complex ptychographic phase retrieval problems and demonstrate its advantages in terms of speed, accuracy, adaptability and generalizability across different scanning techniques.

Multi-focus microscope (MFM) provides a way to obtain 3D information by simultaneously capturing multiple focal planes. The naive method for MFM reconstruction is to stack the sub-images with alignment. However, the resolution in the z-axis in this method is limited by the number of acquired focal planes. In this work we build on a recent reconstruction algorithm for MFM, using information from multiple frames to improve the reconstruction quality. We propose two multiple-frame MFM image reconstruction algorithms: batch and recursive approaches. In the batch approach, we take multiple MFM frames and jointly estimate the 3D image and the motion for each frame. In the recursive approach, we utilize the reconstructed image from the previous frame. Experimental results show that the proposed algorithms produce a sequence of 3D object reconstruction with high quality that enable reconstruction of dynamic extended objects.

Surface shape scanning techniques, such as laser scanning and photometric stereo, are widespread analytical tools used in the field of cultural heritage. Compared to regular 2D RGB photos, 3D surface scans provide higher fidelity of an object's surface shape which assist conservators, art historians, and archaeologists in understanding how these artworks and artifacts are made and to digitally document them for purposes of conservation. However, current state-of-the-art 3D surface scanning tools used in art conservation are often expensive and bulky-such as light dome structures that are often over 1 m in diameter. In this paper, we introduce mobile shape-from-shifting (SfS): a simple, low-cost and streamlined photometric stereo framework for scanning planar surfaces with a consumer mobile device coupled to a low-cost add-on component. Our free-form mobile SfS framework relaxes the rigorous hardware and other complex requirements inherent to conventional 3D scanning tools. This is achieved by taking a sequence of photos with the on-board camera and flash of a mobile device. The sequence of captures are used to reconstruct high quality normal maps using nearlight photometric stereo algorithms, which are of comparable quality to conventional photometric stereo. We demonstrate 3D surface reconstructions with SfS on different materials and scales. Moreover, the mobile SfS technique can be used "in the wild" so that 3D scans may be performed in their natural environment, eliminating the need for transport to a laboratory setting. With the elegant design and low cost, we believe our Mobile SfS can greatly benefit the conservation community by providing a userfriendly and cost-effective solution for 3D surface scanning.

Fourier ptychography is an imaging technique that overcomes the diffraction limit of conventional cameras with applications in microscopy and long range imaging. Diffraction blur causes resolution loss in both cases. In Fourier ptychography, a coherent light source illuminates an object, which is then imaged from multiple viewpoints. The reconstruction of the object from these set of recordings can be obtained by an iterative phase retrieval algorithm. However, the retrieval process is slow and does not work well under certain conditions. In this paper, we propose a new reconstruction algorithm that is based on convolutional neural networks and demonstrate its advantages in terms of speed and performance.

As x-ray and electron tomography is pushed further into the nanoscale, the limitations of rotation stages become more apparent, leading to challenges in the alignment of the acquired projection images. Here we present an approach for rapid post-acquisition alignment of these projections to obtain high quality three-dimensional images. Our approach is based on a joint estimation of alignment errors, and the object, using an iterative refinement procedure. With simulated data where we know the alignment error of each projection image, our approach shows a residual alignment error that is a factor of a thousand smaller, and it reaches the same error level in the reconstructed image in less than half the number of iterations. We then show its application to experimental data in x-ray and electron nanotomography.

Digital in-line holography serves as a useful encoder for spatial information. This allows three-dimensional reconstruction from a two-dimensional image. This is applicable to the tasks of fast motion capture, particle tracking etc. Sampling high resolution holograms yields a spatiotemporal tradeoff. We spatially subsample holograms to increase temporal resolution. We demonstrate this idea with two subsampling techniques, periodic and uniformly random sampling. The implementation includes an on-chip setup for periodic subsampling and a DMD (Digital Micromirror Device) -based setup for pixel-wise random subsampling. The on-chip setup enables direct increase of up to 20 in camera frame rate. Alternatively, the DMD-based setup encodes temporal information as high-speed mask patterns, and projects these masks within a single exposure (coded exposure). This way, the frame rate is improved to the level of the DMD with a temporal gain of 10. The reconstruction of subsampled data using the aforementioned setups is achieved in two ways. We examine and compare two iterative reconstruction methods. One is an error reduction phase retrieval and the other is sparsity-based compressed sensing algorithm. Both methods show strong capability of reconstructing complex object fields. We present both simulations and real experiments. In the lab, we image and reconstruct structure and movement of static polystyrene microspheres, microscopic moving peranema, macroscopic fast moving fur and glitters.

Can we reconstruct the entire internal shape of a room if all we can directly observe is a small portion of one internal wall, presumably through a window in the room? While conventional wisdom may indicate that this is not possible, motivated by recent work on ‘looking around corners’, we show that one can exploit light echoes to reconstruct the internal shape of hidden rooms. Existing techniques for looking around the corner using transient images model the hidden volume using voxels and try to explain the captured transient response as the sum of the transient responses obtained from individual voxels. Such a technique inherently suffers from challenges with regards to low signal to background ratios (SBR) and has difficulty scaling to larger volumes. In contrast, in this paper, we argue for using a plane-based model for the hidden surfaces. We demonstrate that such a plane-based model results in much higher SBR while simultaneously being amenable to larger spatial scales. We build an experimental prototype composed of a pulsed laser source and a single-photon avalanche detector (SPAD) that can achieve a time resolution of about 30ps and demonstrate high-fidelity reconstructions both of individual planes in a hidden volume and for reconstructing entire polygonal rooms composed of multiple planar walls.

On-chip holographic video is a convenient way to monitor biological samples simultaneously at high spatial resolution and over a wide field-of-view. However, due to the limited readout rate of digital detector arrays, one often faces a tradeoff between the per-frame pixel count and frame rate of the captured video. In this report, we propose a subsampled phase retrieval (SPR) algorithm to overcome the spatial-temporal trade-off in holographic video. Compared to traditional phase retrieval approaches, our SPR algorithm uses over an order of magnitude less pixel measurements while maintaining suitable reconstruction quality. We use an on-chip holographic video setup with pixel sub-sampling to experimentally demonstrate a factor of 5.5 increase in sensor frame rate while monitoring the in vivo movement of Peranema microorganisms.

Light scattering on diffuse rough surfaces was long assumed to destroy geometry and photometry information about hidden (non line of sight) objects making ‘looking around the corner’ (LATC) and ‘non line of sight’ (NLOS) imaging impractical. Recent work pioneered by Kirmani et al. [1], Velten et al. [2] demonstrated that transient information (time of flight information) from these scattered third bounce photons can be exploited to solve LATC and NLOS imaging. In this paper, we quantify the geometric and photometric reconstruction limits of LATC and NLOS imaging for the first time using a classical linear systems approach. The relationship between the albedo of the voxels in a hidden volume to the third bounce measurements at the sensor is a linear system that is determined by the geometry and the illumination source. We study this linear system and employ empirical techniques to find the limits of the information contained in the third bounce photons as a function of various system parameters.

Compressed sensing has been discussed separately in spatial and temporal domains. Compressive holography has been introduced as a method that allows 3D tomographic reconstruction at different depths from a single 2D image. Coded exposure is a temporal compressed sensing method for high speed video acquisition. In this work, we combine compressive holography and coded exposure techniques and extend the discussion to 4D reconstruction in space and time from one coded captured image. In our prototype, digital in-line holography was used for imaging macroscopic, fast moving objects. The pixel-wise temporal modulation was implemented by a digital micromirror device. In this paper we demonstrate $10\times$ temporal super resolution with multiple depths recovery from a single image. Two examples are presented for the purpose of recording subtle vibrations and tracking small particles within 5 ms., Comment: 12 pages, 6 figures

Identifying food types consumed and their calorie composition is one of the central tasks of dietary assessment. Traditional automated image processing methods learn to map images to an existing food database with known caloric composition. However, even when the correct food type is identified, caloric makeup can vary depending on its ingredients, and using true-color images proves insufficient to distinguish within food type variability. In this paper, we show that hyperspectral imaging provides useful information and promise in distinguishing caloric composition within the same food type. We collect data using a hyperspectral camera from Nigerian foods cooked with varying degrees of fat content, and capture images under different intensities of light. We apply Principle Component Analysis (PCA) to reduce the dimensionality, and train a Support Vector Machine (SVM) classifier using a Radial Basis Function kernel and show that applying this technique on hyperspectral images can more readily distinguish calorie composition. Furthermore, compared with methods that only use true-color based features, our method shows that a classifier trained using features from hyperspectral images is significantly more predictive of within-food caloric content, and by fusing results from two classifiers trained separately using hyperspectral and RGB imagery we obtain the greatest predictive power.

We propose a dictionary-based phase retrieval approach for monitoring in vivo biological samples based on lens-free on-chip holographic video. Our results present a temporal increase of 9× with 4 × 4 sub-sampling.

Incoherent holography has recently attracted significant research interest due to its flexibility for a wide variety of light sources. In this paper, we use compressive sensing to reconstruct a three-dimensional volumetric object from its two-dimensional Fresnel incoherent correlation hologram. We show how compressed sensing enables reconstruction without out-of-focus artifacts, when compared to conventional back-propagation recovery. Finally, we analyze the reconstruction guarantees of the proposed approach both numerically and theoretically and compare that with coherent holography.

In this paper, we study the problem of automatic identification of pigments applied to paintings using hyperspectral reflectance data. Here, we cast the problem of pigment identification in a novel way by decomposing the spectrum into pure pigments. The pure pigment exemplars, chosen and prepared in our laboratory based on historic sources and archaeological examples, closely resemble the materials used to make ancient paintings. To validate our algorithm, we created a set of mock-up paintings in our laboratory consisting of a broad palette of mixtures of pure pigments. Our results clearly demonstrate more accurate estimation of pigment composition than purely distance-based methods such as spectral angle mapping (SAM) and spectral correlation mapping (SCM). In addition, we studied hyperspectral imagery acquired of a Roman-Egyptian portrait, excavated from the site of Tebtunis in the Fayum region of Egypt, and dated to about the 2nd century CE. Using ground truth information obtained using Raman spectroscopy, we show qualitatively that our method accurately detects pigment composition for the specific pigments hematite and indigo.

X-Ray fluorescence (XRF) scanning of works of art is becoming an increasingly popular non-destructive analytical method. The high quality XRF spectra is necessary to obtain significant information on both major and minor elements used for characterization and provenance analysis. However, there is a trade-off between the spatial resolution of an XRF scan and the Signal-to-Noise Ratio (SNR) of each pixel's spectrum, due to the limited scanning time. In this paper, we propose an XRF image super-resolution method to address this trade-off, thus obtaining a high spatial resolution XRF scan with high SNR. We use a sparse representation of each pixel using a dictionary trained from the spectrum samples of the image, while imposing a spatial smoothness constraint on the sparse coefficients. We then increase the spatial resolution of the sparse coefficient map using a conventional super-resolution method. Finally the high spatial resolution XRF image is reconstructed by the high spatial resolution sparse coefficient map and the trained spectrum dictionary.

We propose an auto-refocused phase retrieval approach that combines refocusing and phase retrieving properties based on lens-free on-chip in-line holography. We demonstrate a 4D tracking application of imaging/monitoring in vivo biomedical scenes, e.g. Blepharisma.

In this paper, we propose a streamlined framework of robust 3D acquisition for cultural heritage using both photometric stereo and photogrammetric information. An uncalibrated photometric stereo setup is augmented by a synchronized secondary witness camera co-located with a point light source. By recovering the witness camera’s position for each exposure with photogrammetry techniques, we estimate the precise 3D location of the light source relative to the photometric stereo camera. We have shown a significant improvement in both light source position estimation and normal map recovery compared to previous uncalibrated photometric stereo techniques. In addition, with the new configuration we propose, we benefit from improved surface shape recovery by jointly incorporating corrected photometric stereo surface normals and a sparse 3D point cloud from photogrammetry.

Over the last decade, a number of computational imaging (CI) systems have been proposed for tasks such as motion deblurring, defocus deblurring and multispectral imaging. These techniques increase the amount of light reaching the sensor via multiplexing and then undo the deleterious effects of multiplexing by appropriate reconstruction algorithms. Given the widespread appeal and the considerable enthusiasm generated by these techniques, a detailed performance analysis of the benefits conferred by this approach is important. Unfortunately, a detailed analysis of CI has proven to be a challenging problem because performance depends equally on three components: (1) the optical multiplexing, (2) the noise characteristics of the sensor, and (3) the reconstruction algorithm which typically uses signal priors. A few recent papers [12], [30], [49] have performed analysis taking multiplexing and noise characteristics into account. However, analysis of CI systems under state-of-the-art reconstruction algorithms, most of which exploit signal prior models, has proven to be unwieldy. In this paper, we present a comprehensive analysis framework incorporating all three components. In order to perform this analysis, we model the signal priors using a Gaussian Mixture Model (GMM). A GMM prior confers two unique characteristics. First, GMM satisfies the universal approximation property which says that any prior density function can be approximated to any fidelity using a GMM with appropriate number of mixtures. Second, a GMM prior lends itself to analytical tractability allowing us to derive simple expressions for the `minimum mean square error' (MMSE) which we use as a metric to characterize the performance of CI systems. We use our framework to analyze several previously proposed CI techniques (focal sweep, flutter shutter, parabolic exposure, etc.), giving conclusive answer to the question: `How much performance gain is due to use of a signal prior and how much is due to multiplexing? Our analysis also clearly shows that multiplexing provides significant performance gains above and beyond the gains obtained due to use of signal priors.

In this paper, we consider the problem of on-chip temporal compressive sensing for video reconstruction at high frame-rates without the need of any additional optical components. We devise an optimization scheme in order to achieve adequate spatio-temporal sampling of subsequent frames under maximal capturing speed, based on the bandwidth constraints of a sensor. We test this optimization strategy on a commercially available camera and propose a set of reconstruction steps that can achieve reasonable performance but, at the same time, accommodate high-resolution video reconstruction under realistic time requirements. Our analysis constitutes a set of first steps bringing high-speed compressive video capture within the realm of commercial availability.

We propose quantifying color in crowd-sourced images from mobile phones to monitor built heritage over time. Time-lapse color movies in CIE color space can provide information on a large range of deterioration mechanisms, including soiling, biofilm growth, weathering and vandalism. Citizen science can create large-scale geographical coverage of sites difficult to obtain any other way. We show that the color accuracy of current phones is sufficient for this purpose and demonstrate image registration, color calibration and change detection using mobile phone cameras. For accurate color, a calibration target of known, stable colors need to be in the image field of view.

In this paper, we propose a near-light illumination model for image relighting and 3D shape recovery. Classic methods such as used by the popular RTI software from Cultural Heritage Imaging assume that lighting is infinitely far away from the scene. However, this constraint is impossible to achieve in practice: light sources cannot be too far away from the scene due to space and power constraints. This causes non-uniform illumination artifacts due to variations in the distance between the light source and points in the scene. We correct this effect to provide much more uniformly lit images that yield more appealing image for relighting applications. Furthermore, we use our near-light model for more accurate photometric stereo calculations of surface normals, eliminating the “potato-chip” shaped surface reconstruction error that results from violating the far-light assumption. We verify our model with both free-form capture using hand-held flash as the illumination source, and capture using LED lights mounted on a dome shaped surface.

Multiple images are routinely used to improve imaging performance in both incoherent and coherent imaging systems. We will discuss some recent algorithmic advances and the applications they enable. Article not available.

We propose a reconstruction method using compressed sensing to accurately recover an image from a single hologram despite a large amount of unmeasured intensities, which allows us to overcome dynamic range limitations in the sensor.