Search

In this paper, we present a prototype pseudo-direct time-of-flight (ToF) CMOS image sensor, achieving high distance accuracy, precision, and robustness to multipath interference. An indirect ToF (iToF)-based image sensor, which enables high spatial resolution, is used to acquire temporal compressed signals in the charge domain. Whole received light waveforms, like those acquired with conventional direct ToF (dToF) image sensors, can be obtained after image reconstruction based on compressive sensing. Therefore, this method has the advantages of both dToF and iToF depth image sensors, such as high resolution, high accuracy, immunity to multipath interference, and the absence of motion artifacts. Additionally, two approaches to refine the depth resolution are explained: (1) the introduction of a sub-time window; and (2) oversampling in image reconstruction and quadratic fitting in the depth calculation. Experimental results show the separation of two reflections 40 cm apart under multipath interference conditions and a significant improvement in distance precision down to around 1 cm. Point cloud map videos demonstrate the improvements in depth resolution and accuracy. These results suggest that the proposed method could be a promising approach for virtually implementing dToF imaging suitable for challenging environments with multipath interference.

This paper presents a time-of-flight image sensor based on 8-Tap P-N junction demodulator (PND) pixels, which is designed for hybrid-type short-pulse (SP)-based ToF measurements under strong ambient light. The 8-tap demodulator implemented with multiple p-n junctions used for modulating the electric potential to transfer photoelectrons to eight charge-sensing nodes and charge drains has an advantage of high-speed demodulation in large photosensitive areas. The ToF image sensor implemented using 0.11 µm CIS technology, consisting of an 120 (H) × 60 (V) image array of the 8-tap PND pixels, successfully works with eight consecutive time-gating windows with the gating width of 10 ns and demonstrates for the first time that long-range (>10 m) ToF measurements under high ambient light are realized using single-frame signals only, which is essential for motion-artifact-free ToF measurements. This paper also presents an improved depth-adaptive time-gating-number assignment (DATA) technique for extending the depth range while having ambient-light canceling capability and a nonlinearity error correction technique. By applying these techniques to the implemented image sensor chip, hybrid-type single-frame ToF measurements with depth precision of maximally 16.4 cm (1.4% of the maximum range) and the maximum non-linearity error of 0.6% for the full-scale depth range of 1.0–11.5 m and operations under direct-sunlight-level ambient light (80 klux) have been realized. The depth linearity achieved in this work is 2.5 times better than that of the state-of-the-art 4-tap hybrid-type ToF image sensor.

An ultra-high-speed computational CMOS image sensor with a burst frame rate of 303 megaframes per second, which is the fastest among the solid-state image sensors, to our knowledge, is demonstrated. This image sensor is compatible with ordinary single-aperture lenses and can operate in dual modes, such as single-event filming mode or multi-exposure imaging mode, by reconfiguring the number of exposure cycles. To realize this frame rate, the charge modulator drivers were adequately designed to suppress the peak driving current taking advantage of the operational constraint of the multi-tap charge modulator. The pixel array is composed of macropixels with 2 × 2 4-tap subpixels. Because temporal compressive sensing is performed in the charge domain without any analog circuit, ultrafast frame rates, small pixel size, low noise, and low power consumption are achieved. In the experiments, single-event imaging of plasma emission in laser processing and multi-exposure transient imaging of light reflections to extend the depth range and to decompose multiple reflections for time-of-flight (TOF) depth imaging with a compression ratio of 8× were demonstrated. Time-resolved images similar to those obtained by the direct-type TOF were reproduced in a single shot, while the charge modulator for the indirect TOF was utilized.

Multi-path interference causes depth errors in indirect time-of-flight (ToF) cameras. In this paper, resolving multi-path interference caused by surface reflections using a multi-tap macro-pixel computational CMOS image sensor is demonstrated. The imaging area is implemented by an array of macro-pixels composed of four subpixels embodied by a four-tap lateral electric field charge modulator (LEFM). This sensor can simultaneously acquire 16 images for different temporal shutters. This method can reproduce more than 16 images based on compressive sensing with multi-frequency shutters and sub-clock shifting. In simulations, an object was placed 16 m away from the sensor, and the depth of an interference object was varied from 1 to 32 m in 1 m steps. The two reflections were separated in two stages: coarse estimation based on a compressive sensing solver and refinement by a nonlinear search to investigate the potential of our sensor. Relative standard deviation (precision) and relative mean error (accuracy) were evaluated under the influence of photon shot noise. The proposed method was verified using a prototype multi-tap macro-pixel computational CMOS image sensor in single-path and dual-path situations. In the experiment, an acrylic plate was placed 1 m or 2 m and a mirror 9.3 m from the sensor.

This paper presents a high-linearity high-resolution time-of-flight (ToF) linear-array digital image sensor using a time-domain negative feedback technique. A coarse ToF measurement loop uses a 5-bit digital-to-time converter (DTC) and a delayed gating-pulse generator for time-domain feedback to find the zero of the difference between ToF and the digital estimate of the gating-pulse delay while maintaining a constant operating point of the analog readout circuits. A fine ToF measurement uses a delta-sigma modulation (DSM) loop using the time-domain feedback with a bit-stream signal form. Because of the self-contained property of the DSM for low distortion and noise exploited by the oversampling signal processing, the proposed technique provides high-linearity and high-range resolution in the fine ToF measurement. A prototype ToF sensor of 16.8 × 16.8 μm2 two-tap pixels and fabricated in a 0.11 μm (1P4M) CMOS image sensors (CIS) process achieves +0.9%/−0.47% maximum nonlinearity error and a resolution of 0.24 mm (median) for the measurement range of 0–1.05 m. The ToF sensor produces an 11-bit fully digital output with a ToF measurement time of 22.4 ms.

Intense Poisson noise drastically degrades image quality when only a few or when a single photon hits each pixel. Multiaperture systems are able to provide multiple images of the same scene, which are acquired simultaneously. After registration and cropping, the partial scene information contained in each aperture image should be the same, while the noise will be different in each one. A similar case arises in multitap systems, which are widely used in Time-of-Flight imaging (ToF), where several integration channels per pixel exist and where several sequential acquisitions are needed to generate a depth image. In this case, raw images might be different from each other, but still, since they are images of the same scene, information redundancy can be exploited to filter out the noise. In this work, we propose two different ways of joint processing of low-light multiaperture images. One of them is an extension of bilateral filtering to the multiaperture case, while the other relies on the compressive sensing theory and aims to recover a noiseless image from fewer measurements than the total number of pixels in the original noisy images. Experimental results show that both methods exhibit very close performance, which is much higher than those of previous methods. Additionally, we show that bilateral filtering can also be applied to the raw images of multitap ToF systems, leading to a significant error reduction in the final depth image.

An 8-tap CMOS lock-in pixel image sensor that has seven carrier-capturing and a draining time window was developed for short-pulse time-of-flight (TOF) measurements. The proposed pixel for the short-pulse TOF measurements has seven consecutive time-gating windows, each of which has the width of 6 ns, which is advantageous for high-resolution range imaging, particularly for relatively longer distances (>5 m) and under high ambient light operations. In order to enhance the depth resolution, a technique for the depth-adaptive time-gating-number assignment (DATA) for the short-pulse TOF measurement is proposed. A prototype of the 8-tap CMOS lock-in pixel image sensor is implemented with a 1POLY 4METAL 0.11-μm CIS process. The maximum non-linearity error of 1.56%FS for the range of 1−6.4 m and the depth resolution of 6.4 mm was obtained at 6.2 m using the DATA technique.

This paper presents a CMOS time-of-flight (ToF) range image sensor using high-speed lock-in pixels with background light canceling capability. The proposed lock-in pixel uses MOS gate-induced lateral electric field control of depleted potential of pinned photodiode for implementing a multiple-tap charge modulator while achieving a high-speed charge transfer for high-time resolution. A TOF image sensor with 320 x 240 effective pixels is implemented using a 0.11-μm CMOS image sensor process. The TOF sensor has a range resolution of less than 12 mm without background light and 20 mm under background line for the range from 0.8 to 1.8 m and integration time of 50 ms. The effectiveness of in-pixel background light canceling with a three-tap output pixel is demonstrated.

This paper introduces an optical vehicle-to-vehicle (V2V) communication system based on an optical wireless communication technology using an LED transmitter and a camera receiver, which employs a special CMOS image sensor, i.e, an optical communication image sensor (OCI). The OCI has a “communication pixel (CPx)” that can promptly respond to light intensity variations and an output circuit of a “flag image” in which only high-intensity light sources, such as LEDs, have emerged. The OCI that employs these two technologies provides capabilities for a 10-Mb/s optical signal reception and real-time LED detection to the camera receiver. The optical V2V communication system consisting of the LED transmitters mounted on a leading vehicle and the camera receiver mounted on a following vehicle is constructed, and various experiments are conducted under real driving and outdoor lighting conditions. Due to the LED detection method using the flag image, the camera receiver correctly detects LEDs, in real time, in challenging outdoor conditions. Furthermore, between two vehicles, various vehicle internal data (such as speed) and image data (320 × 240, color) are transmitted successfully, and the 13.0-fps image data reception is achieved while driving outside.

In extremely low-light conditions, random telegraph signal (RTS) noise and dark current white defects become visible. In this paper, a multi-aperture imaging system and selective averaging method which removes the RTS noise and the dark current white defects by minimizing the synthetic sensor noise at every pixel is proposed. In the multi-aperture imaging system, a very small synthetic F-number which is much smaller than 1.0 is achieved by increasing optical gain with multiple lenses. It is verified by simulation that the effective noise normalized by optical gain in the peak of noise histogram is reduced from 1.38e⁻ to 0.48 e⁻ in a 3 × 3-aperture system using low-noise CMOS image sensors based on folding-integration and cyclic column ADCs. In the experiment, a prototype 3 × 3-aperture camera, where each aperture has 200 × 200 pixels and an imaging lens with a focal length of 3.0 mm and F-number of 3.0, is developed. Under a low-light condition, in which the maximum average signal is 11e⁻ per aperture, the RTS and dark current white defects are removed and the peak signal-to-noise ratio (PSNR) of the image is increased by 6.3 dB.

In this paper, a wide dynamic range (WDR) CMOS image sensor (CIS) with a charge splitting gate (SG) and two storage diodes (SDs) is presented. By using single-gate on/off control with the SG, photocurrent path to the first (SD1) or second storage diodes (SD2) is switched alternatively and periodically during exposure and signal electrons generated in a photodiode (PD) are transferred to and accumulated in the SD1 or SD2. By setting a large ratio of the off-time to on-time of the SG, two different sensitivity signals, which are originated by the same photodiode, are generated and a WDR image signal is obtained. This technique has a distinct advantage on mitigating the problem of motion artifact in WDR imaging with high and low sensitivity signals and flexible dynamic control of the dynamic range. An experimental WDR CMOS image sensor with 280 (H) × 406 (V)-pixel array consisting of 14 sub-arrays, each of which have 20 (H) × 406 (V) pixels, was implemented and tested. For the SG on/off-time ratio of 30 and 279, the DR of 93 dB and 104 dB, respectively, was demonstrated. The effect of the proposed WDR imaging operation on the reduced motion artifact was experimentally confirmed.

We demonstrate the multiband color filtering of a standard RGB color and a complementary CMY color by a plasmonic color filter, composed of concentric corrugated metallic thin film rings. The surface plasmon resonance is excited by the periodic corrugation, and the coupled light is transmitted through the central subwavelength aperture. Color selectivity is achieved not only in the visible but also in the near-infrared (NIR) region. Therefore, simultaneous imaging with visible and NIR can be realized by the integration of plasmonic color filters with sensors. We investigate the angle of incidence dependence of the transmission color selectivity and the color purity of the fabricated plasmonic color filter array.

A lensless camera is an ultra-thin computational-imaging system. Existing lensless cameras are based on the axial arrangement of an image sensor and a coding mask, and therefore, the back side of the image sensor cannot be captured. In this paper, we propose a lensless camera with a novel design that can capture the front and back sides simultaneously. The proposed camera is composed of multiple coded image sensors, which are complementary-metal-oxide-semiconductor (CMOS) image sensors in which air holes are randomly made at some pixels by drilling processing. When the sensors are placed facing each other, the object-side sensor works as a coding mask and the other works as a sparsified image sensor. The captured image is a sparse coded image, which can be decoded computationally by using compressive sensing-based image reconstruction. We verified the feasibility of the proposed lensless camera by simulations and experiments. The proposed thin lensless camera realized super-field-of-view imaging without lenses or coding masks and therefore can be used for rich information sensing in confined spaces. This work also suggests a new direction in the design of CMOS image sensors in the era of computational imaging.

In this paper, we report on the development of a monolithic active pixel sensor for X-ray imaging using 0.2 µm fully depleted silicon-on-insulator (SOI)-based technology to support next generation astronomical satellite missions. Detail regarding low-noise dual-gain SOI based pixels with a charge sensitive amplifier and pinned depleted diode sensor structure is presented. The proposed multi-well sensor structure underneath the fully-depleted SOI allows the design of a detector with low node capacitance and high charge collection efficiency. Configurations for achieving very high charge-to-voltage conversion gain of 52 µV/e− and 187 µV/e− are demonstrated. Furthermore, in-pixel dual gain selection is used for low-noise and wide dynamic range X-ray energy detection. A technique to improve the noise performance by removing correlated system noise leads to an improvement in the spectroscopic performance of the measured X-ray energy. Taken together, the implemented chip has low dark current (44.8 pA/cm2 at −30 °C), improved noise performance (8.5 e− rms for high gain and 11.7 e− rms for low gain), and better energy resolution of 2.89% (171 eV FWHM) at 5.9 keV using 55Fe and 1.67% (234 eV FWHM) at 13.95 keV using 241Am.

A probabilistic method to remove the random telegraph signal (RTS) noise and to increase the signal level is proposed, and was verified by simulation based on measured real sensor noise. Although semi-photon-counting-level (SPCL) ultra-low noise complementary-metal-oxide-semiconductor (CMOS) image sensors (CISs) with high conversion gain pixels have emerged, they still suffer from huge RTS noise, which is inherent to the CISs. The proposed method utilizes a multi-aperture (MA) camera that is composed of multiple sets of an SPCL CIS and a moderately fast and compact imaging lens to emulate a very fast single lens. Due to the redundancy of the MA camera, the RTS noise is removed by the maximum likelihood estimation where noise characteristics are modeled by the probability density distribution. In the proposed method, the photon shot noise is also relatively reduced because of the averaging effect, where the pixel values of all the multiple apertures are considered. An extremely low-light condition that the maximum number of electrons per aperture was the only 2 e − was simulated. PSNRs of a test image for simple averaging, selective averaging (our previous method), and the proposed method were 11.92 dB, 11.61 dB, and 13.14 dB, respectively. The selective averaging, which can remove RTS noise, was worse than the simple averaging because it ignores the pixels with RTS noise and photon shot noise was less improved. The simulation results showed that the proposed method provided the best noise reduction performance.

The photometric stereo method enables estimation of surface normals from images that have been captured using different but known lighting directions. The classical photometric stereo method requires at least three images to determine the normals in a given scene. However, this method cannot be applied to dynamic scenes because it is assumed that the scene remains static while the required images are captured. In this work, we present a dynamic photometric stereo method for estimation of the surface normals in a dynamic scene. We use a multi-tap complementary metal-oxide-semiconductor (CMOS) image sensor to capture the input images required for the proposed photometric stereo method. This image sensor can divide the electrons from the photodiode from a single pixel into the different taps of the exposures and can thus capture multiple images under different lighting conditions with almost identical timing. We implemented a camera lighting system and created a software application to enable estimation of the normal map in real time. We also evaluated the accuracy of the estimated surface normals and demonstrated that our proposed method can estimate the surface normals of dynamic scenes.

Raman imaging eliminates the need for staining procedures, providing label-free imaging to study biological samples. Recent developments in stimulated Raman scattering (SRS) have achieved fast acquisition speed and hyperspectral imaging. However, there has been a problem of lack of detectors suitable for MHz modulation rate parallel detection, detecting multiple small SRS signals while eliminating extremely strong offset due to direct laser light. In this paper, we present a complementary metal-oxide semiconductor (CMOS) image sensor using high-speed lock-in pixels for stimulated Raman scattering that is capable of obtaining the difference of Stokes-on and Stokes-off signal at modulation frequency of 20 MHz in the pixel before reading out. The generated small SRS signal is extracted and amplified in a pixel using a high-speed and large area lateral electric field charge modulator (LEFM) employing two-step ion implantation and an in-pixel pair of low-pass filter, a sample and hold circuit and a switched capacitor integrator using a fully differential amplifier. A prototype chip is fabricated using 0.11 μm CMOS image sensor technology process. SRS spectra and images of stearic acid and 3T3-L1 samples are successfully obtained. The outcomes suggest that hyperspectral and multi-focus SRS imaging at video rate is viable after slight modifications to the pixel architecture and the acquisition system.

A complementary metal-oxide semiconductor (CMOS) lock-in pixel to observe stimulated Raman scattering (SRS) using a high speed lateral electric field modulator (LEFM) for photo-generated charges and in-pixel readout circuits is presented. An effective SRS signal generated after the SRS process is very small and needs to be extracted from an extremely large offset due to a probing laser signal. In order to suppress the offset components while amplifying high-frequency modulated small SRS signal components, the lock-in pixel uses a high-speed LEFM for demodulating the SRS signal, resistor-capacitor low-pass filter (RC-LPF) and switched-capacitor (SC) integrator with a fully CMOS differential amplifier. AC (modulated) components remained in the RC-LPF outputs are eliminated by the phase-adjusted sampling with the SC integrator and the demodulated DC (unmodulated) components due to the SRS signal are integrated over many samples in the SC integrator. In order to suppress further the residual offset and the low frequency noise (1/f noise) components, a double modulation technique is introduced in the SRS signal measurements, where the phase of high-frequency modulated laser beam before irradiation of a specimen is modulated at an intermediate frequency and the demodulation is done at the lock-in pixel output. A prototype chip for characterizing the SRS lock-in pixel is implemented and a successful operation is demonstrated. The reduction effects of residual offset and 1/f noise components are confirmed by the measurements. A ratio of the detected small SRS to offset a signal of less than 10−5 is experimentally demonstrated, and the SRS spectrum of a Benzonitrile sample is successfully observed.

An optical wireless communication (OWC) system based on a light-emitting-diode (LED) transmitter and a camera receiver has been developed for use in the automotive area. The automotive OWC system will require Mb/s-class data rates and the ability to quickly detect LEDs from an image. The key to achieving this is improvements to the capabilities of the image sensor mounted on the camera receiver. In this paper, we report on a novel OWC system equipped with an optical communication image sensor (OCI), which is newly developed using CMOS technology. To obtain higher transmission rates, the OCI employs a specialized “communication pixel (CPx)” capable of responding promptly to optical intensity variations. Furthermore, a new quick LED detection technique, based on a 1-bit flag image which only reacts to high-intensity objects, is formulated. The communication pixels, ordinary image pixels, and associated circuits (including 1-bit flag image output circuits) are then integrated into the OCI. This paper describes the design, fabrication, and capabilities of the OCI, as well as the development of the LED and image sensor based OWC system, which boasts a 20-Mb/s/pixel data rate without LED detection and a 15-Mb/s/pixel data rate with a 16.6-ms real-time LED detection.

Significance: We present a motion-resistant three-wavelength spatial frequency domain imaging (SFDI) system with ambient light suppression using an 8-tap complementary metal-oxide semiconductor (CMOS) image sensor (CIS) developed at Shizuoka University. The system addresses limitations in conventional SFDI systems, enabling reliable measurements in challenging imaging scenarios that are closer to real-world conditions. Aim: Our study demonstrates a three-wavelength SFDI system based on an 8-tap CIS. We demonstrate and evaluate the system's capability of mitigating motion artifacts and ambient light bias through tissue phantom reflectance experiments and in vivo volar forearm experiments. Approach: We incorporated the Hilbert transform to reduce the required number of projected patterns per wavelength from three to two per spatial frequency. The 8-tap image sensor has eight charge storage diodes per pixel; therefore, simultaneous image acquisition of eight images based on multi-exposure is possible. Taking advantage of this feature, the sensor simultaneously acquires images for planar illumination, sinusoidal pattern projection at three wavelengths, and ambient light. The ambient light bias is eliminated by subtracting the ambient light image from the others. Motion artifacts are suppressed by reducing the exposure and projection time for each pattern while maintaining sufficient signal levels by repeating the exposure. The system is compared to a conventional SFDI system in tissue phantom experiments and then in vivo measurements of human volar forearms. Results: The 8-tap image sensor-based SFDI system achieved an acquisition rate of 9.4 frame sets per second, with three repeated exposures during each accumulation period. The diffuse reflectance maps of three different tissue phantoms using the conventional SFDI system and the 8-tap image sensor-based SFDI system showed good agreement except for high scattering phantoms. For the in vivo volar forearm measurements, our system successfully measured total hemoglobin concentration, tissue oxygen saturation, and reduced scattering coefficient maps of the subject during motion (16.5 cm/s) and under ambient light (28.9 lx), exhibiting fewer motion artifacts compared with the conventional SFDI. Conclusions: We demonstrated the potential for motion-resistant three-wavelength SFDI system with ambient light suppression using an 8-tap CIS. [ABSTRACT FROM AUTHOR]