Search

Mobile captured images can be aligned using their gyroscope sensors. Optical image stabilizer (OIS) terminates this possibility by adjusting the images during the capturing. In this work, we propose a deep network that compensates for the motions caused by the OIS, such that the gyroscopes can be used for image alignment on the OIS cameras. To achieve this, we first record both videos and gyroscope readings with an OIS camera as training data. Then, we convert gyroscope readings into motion fields. Second, we propose an Essential Mixtures motion model for rolling shutter cameras, where an array of rotations within a frame are extracted as the ground-truth guidance. Third, we train a convolutional neural network with gyroscope motions as input to compensate for the OIS motion. Once finished, the compensation network can be applied for other scenes, where the image alignment is purely based on gyroscopes with no need for images contents, delivering strong robustness. Experiments show that our results are comparable with that of non-OIS cameras, and outperform image-based alignment results with a relatively large margin. Code and dataset is available at: https://github.com/lhaippp/DeepOIS.

Visible light positioning (VLP) is a promising technology since it can provide high accuracy indoor localization based on the existing lighting infrastructure. However, existing approaches often require dense LED distributions and persistent line-of-sight (LOS) between transmitter and receiver. What's more, sensors are imperfect, and their measurements are prone to errors. Through multi sensors fusion, we can compensate the deficiencies of stand-alone sensors and provide more reliable pose estimations. In this work, we propose a loosely-coupled multi-sensor fusion method based on VLP and Simultaneous Localization and Mapping (SLAM), using light detection and ranging (LiDAR), odometry, and rolling shutter camera. Our multi-sensor localizer can provide accurate and robust robot localization and navigation in LED shortage/outage situations. The experimental results show that our proposed scheme can provide an average accuracy of 2.5 cm with around 42 ms average positioning latency.

Recent studies have been proposed to extract speech from the captured video of objects vibrating by sound waves. Among them, from the viewpoint of equipment cost, the method of extracting speech from the video captured by rolling-shutter cameras, which are widely used in consumer digital single-lens reflex cameras, has been attracting attention. The conventional method with the rolling-shutter video uses a grayscale video for processing based on phase images. However, a grayscale video has a smaller dynamic range than an RGB video, and thus the speech extraction accuracy of the conventional method degrades. Therefore, this paper proposes a speech extraction method based on RGB-intensity gradients on an RGB video to improve speech extraction accuracy. The proposed method extracts the speech by calculating the similarity of R, G, and B intensity gradients, and using these three intensity gradients expands the dynamic range. The experimental results on the quality and intelligibility of the extracted speech show our proposed method outperforms the conventional method.

International audience; In this article we study the adaptation of the concept of homography to Rolling Shutter (RS) images. This extension has never been clearly adressed despite the many roles played by the homography matrix in multi-view geometry. We first show that a direct point-to-point relationship on a RS pair can be expressed as a set of 3 to 8 atomic 3x3 matrices depending on the kinematic model used for the instantaneous-motion during image acquisition. We call this group of matrices the RS Homography. We then propose linear solvers for the computation of these matrices using point correspondences. Finally, we derive linear and closed form solutions for two famous problems in computer vision in the case of RS images: image stitching and plane-based relative pose computation. Extensive experiments with both synthetic and real data from public benchmarks show that the proposed methods outperform state-of-art techniques.

Many methods have been proposed to process stereo matching for rolling shutter image pairs, they treat all pixels from an image pair in an identical way and require additional estimation approaches to estimate motion states of the camera. However, pixels from a rolling shutter image pair naturally have diverse baseline lengths, and motion estimation methods are unstable for one instantaneous image pair input. In this letter, we present a rolling shutter stereo depth estimation pipeline, which can robustly estimate motion states and depth maps by alternating the estimation of the depth maps and refining the motion states from coarse image levels to fine image levels. What is more, we design a novel cost volume building method for rolling shutter image pairs, which adapts depth candidates to the change of baseline lengths for all pixels. We further demonstrate the usability of the proposed method by constructing a new platform, building an outdoor evaluation dataset, and comparing it with baseline methods.

This paper describes a camera simulation framework for validating machine vision algorithms under general airborne camera imperfections. Lens distortion, image delay, rolling shutter, motion blur, interlacing, vignetting, image noise, and light level are modelled. This is the first simulation that considers all temporal distortions jointly, along with static lens distortions in an online manner. Several innovations are proposed including a motion tracking system allowing the camera to follow the flight log with eligible derivatives. A reverse pipeline, relating each pixel in the output image to pixels in the ideal input image, is developed. It is shown that the inverse lens distortion model and the inverse temporal distortion models are decoupled in this way. A short-time pixel displacement model is proposed to solve for temporal distortions (i.e. delay, rolling shutter, motion blur, and interlacing). Evaluation is done by several means including regenerating an airborne dataset, regenerating the camera path on a calibration pattern, and evaluating the ability of the time displacement model to predict other frames. Qualitative evaluations are also made.

Due to the delay in the row-wise exposure and the lack of stable support when a photographer holds a CMOS camera, video jitter and rolling shutter distortion are closely coupled degradations in the captured videos. However, previous methods have rarely considered both phenomena and usually treat them separately, with stabilization approaches that are unable to handle the rolling shutter effect and rolling shutter removal algorithms that are incapable of addressing motion shake. To tackle this problem, we propose a novel method that simultaneously stabilizes and rectifies a rolling shutter shaky video. The key issue is to estimate both inter-frame motion and intra-frame motion. Specifically, for each pair of adjacent frames, we first estimate a set of spatially variant inter-frame motions using a neighbor-motion-aware local motion model, where the classical mesh-based model is improved by introducing a new constraint to enhance the neighbor motion consistency. Then, different from other 2D rolling shutter removal methods that assume the pixels in the same row have a single intra-frame motion, we build a novel mesh-based intra-frame motion calculation model to cope with the depth variation in a mesh row and obtain more faithful estimation results. Finally, temporal and spatial motion constraints and an adaptive weight assignment strategy are considered together to generate the optimal warping transformations for different motion situations. Experimental results demonstrate the effectiveness and superiority of the proposed method when compared with other state-of-the-art methods.

The traditional visual SLAM systems take the monocular or stereo camera as input sensor, with complex map initialization and map point triangulation steps needed for 3D map reconstruction, which are easy to fail, computationally complex and can cause noisy measurements. The emergence of RGB-D camera which provides RGB image together with depth information breaks this situation. While a number of RGB-D SLAM systems have been proposed in recent years, the current classification research on RGB-D SLAM is very lacking, and their advantages and shortcomings remain unclear regarding different applications and perturbations, such as illumination transformation, noise and rolling shutter effect of sensors. In this paper, we mainly introduced the basic concept and structure of the RGB-D SLAM system, and then introduced the differences between the various RGB-D SLAM systems in the three aspects of tracking, mapping, and loop detection, and we make a classification study on different RGB-D SLAM algorithms according to the three aspect. Furthermore, we discuss some advanced topics and open problems of RGB-D SLAM, hoping that it will help for future exploring. In the end, we conducted a large number of evaluation experiments on multiple RGB-D SLAM systems, and analyzed their advantages and disadvantages, as well as performance differences in different application scenarios, and provided references for researchers and developers.

Since the rolling shutter (RS) camera successively exposes each scanline, accurately reconstructing scene depth from an RS stereo image pair remains a great challenge. Directly applying the deep-learning-based depth estimation methods tailored for the global shutter (GS) stereo images leads to undesirable RS depth results due to inherent flaws in the network structure. In this letter, we fill this gap by developing an end-to-end RS-stereo-aware plane sweep network to improve the accuracy of the classic GS-based algorithm ( i.e. DPSNet) in estimating the RS depth map. Specifically, we derive the RS-stereo-aware plane sweep model and further produce a more accurate and efficient cost volume through the effective incorporation of this model within DPSNet. Furthermore, to enable learning-based approaches to address the depth estimation problem in the context of RS stereo images, we contribute the first RS stereo dataset, CARLA-RSS. Experimental results demonstrate that our proposed pipeline achieves state-of-the-art performance.

Camera sensors rely on global or rolling shutter functions to expose an image. This fixed function approach severely limits the sensors' ability to capture high-dynamic-range (HDR) scenes and resolve high-speed dynamics. Spatially varying pixel exposures have been introduced as a powerful computational photography approach to optically encode irradiance on a sensor and computationally recover additional information of a scene, but existing approaches rely on heuristic coding schemes and bulky spatial light modulators to optically implement these exposure functions. Here, we introduce neural sensors as a methodology to optimize per-pixel shutter functions jointly with a differentiable image processing method, such as a neural network, in an end-to-end fashion. Moreover, we demonstrate how to leverage emerging programmable and re-configurable sensor-processors to implement the optimized exposure functions directly on the sensor. Our system takes specific limitations of the sensor into account to optimize physically feasible optical codes and we evaluate its performance for snapshot HDR and high-speed compressive imaging both in simulation and experimentally with real scenes.

We present minimal, non-iterative solutions to the absolute pose problem for images from rolling shutter cameras. The absolute pose problem is a key problem in computer vision and rolling shutter is present in a vast majority of today's digital cameras. We discuss several camera motion models and propose two feasible rolling shutter camera models for a polynomial solver. In previous work a linearized camera model was used that required an initial estimate of the camera orientation. We show how to simplify the system of equations and make this solver faster. Furthermore, we present a first solution of the non-linearized camera orientation model using the Cayley parameterization. The new solver does not require any initial camera orientation estimate and therefore serves as a standalone solution to the rolling shutter camera pose problem from six 2D-to-3D correspondences. We show that our algorithms outperform P3P followed by a non-linear refinement using a rolling shutter model.

A method has been developed for calibrating the time intervals of video recorders using a LED panel known as “SEXTA”, which utilizes Global Positioning System (GPS) signal as time reference. This method directly measures the frame intervals frame by frame down to a time resolution of 2 ms of the recordings by video recorders equipped with CCD or CMOS sensors in global shutter or rolling shutter modes respectively. The method has been validated by using a Digital Single Lens Reflex (DSLR) camera and a previous studied dashboard camera. Frame by frame analysis of the calibrated video recording of SEXTA can reveal characteristics of sensors, such as type of sensors (CMOS vs CCD), exposure time, the direction and speed of rolling shutter for CMOS sensors as well as the common characteristics of dashboard cameras such as frame skipping.

The typical optical camera communication (OCC) modulation scheme is based on binary intensity modulation. To increase the transmission data rate, multi-level modulation format is highly desirable. In this work, we bring forward and demonstrate a rolling shutter 4-level pulse amplitude modulation (PAM4) demodulation scheme for OCC systems using pixel-per-symbol labeling neural network (PPSL-NN) for the first time up to the authors’ knowledge. A bit-rate distance product of 28.8 kbit/s • m per color is achieved. The proposed scheme is to calculate and re-sample the pixel-per-symbol (PPS) to make sure the same number of pixels in each PAM4 symbol is corresponding to a label for the neural network. Experiment results reveal that the proposed scheme can efficiently demodulate high speed PAM4 signal in the rolling shutter OCC pattern.

The vast majority of modern consumer cameras employ a rolling shutter (RS) mechanism which has a price and electronic advantage to global shutter (GS). However, in geometric computer vision applications such as visual simultaneous localization and mapping (VSLAM), performances of accuracy and robustness are usually deteriorated due to the rolling shutter effect when using the RS cameras. This paper introduced the Wuhan University Rolling Shutter Visual-Inertial (WHU-RSVI) synthetic dataset for evaluating VSLAM and VI-SLAM (visual-inertial SLAM) methods in which RS cameras or IMU data are typically used. The proposed synthetic dataset contains RS images, time-synchronized GS images, inertial measurement unit (IMU) measurements, and accurate ground truth. It provides camera images with $640\times 480 $ resolution at 30 Hz and IMU measurements from 90 Hz to 14400 Hz. The cubic B-spline curves are used to model the motion of trajectories. Based on the known trajectories, an image of each pose can be rendered, and the corresponding IMU measurement model is then established. The dataset provides realistic images and IMU measurements by modeling the sensor noise in RGB and IMU data. Two trajectories with three sequences of different motion speeds (i. e., slow, medium and fast corresponding to different rolling shutter effects) are contained in the proposed dataset. Herein, the proposed dataset can be applied to compare the impact of different rolling shutter effects on a specific method.

Optical camera communication (OCC) traditionally uses the bright and dark stripes of rolling shutter pattern captured by the complementary metal-oxide-semiconductor (CMOS) image-sensor to realize communication. For the reason that these stripes are distributed on plane space of the image, the conventional OCC is based on image space-domain. In this paper, different from the convention, we first introduce a Modulation and Image Frequency-Domain Recognition (M-IFDR) method of OCC. In this approach, LEDs are modulated to assign different frequency characteristics instead of transmitting data. Taking discrete Fourier transform (DFT) of the rolling shutter pattern to obtain the spectrum image, we can recognize the LED according to the characteristics of the spectrum image based on image frequency-domain. The experiment results show that the proposed M-IFDR method has good performance, with at least 426 recognizable LEDs and the recognition distance reaching 3m. Therefore, the proposed M-IFDR method can be considered as one of the effective methods of OCC.

In this paper, a positioning system framework based on a single light emitting diode (LED) lamp is introduced. The LED lamp is used to transmit the positioning information, and the commercial mobile phone is used as the receiver. A fringe image carrying information is acquired by the rolling shutter effect of the complementary metal oxide semiconductor (CMOS) image sensor. Based on this, a piecewise fitting method is proposed to instead of the overall fitting method, which improves the decoding success rate. We also propose a method of LED light source recognition, which detects the edge of the light source based on brightness weakening, the Sobel operator and the least-squares fitting. In addition, the imaging principle of the camera in horizontal and tilted scenarios is analyzed. Accurate indoor positioning is achieved by the imaging principle of the camera lens and the attitude information of the mobile phone. The experimental results show that the system established in this paper can achieve centimeter-level positioning accuracy whether the mobile phone is placed horizontally or obliquely.

Structure from small motion has become an important topic in 3D computer vision as a method for estimating depth, since capturing the input is so user-friendly. However, major limitations exist with respect to the form of depth uncertainty, due to the narrow baseline and the rolling shutter effect. In this paper, we present a dense 3D reconstruction method from small motion clips using commercial hand-held cameras, which typically cause the undesired rolling shutter artifact. To address these problems, we introduce a novel small motion bundle adjustment that effectively compensates for the rolling shutter effect. Moreover, we propose a pipeline for a fine-scale dense 3D reconstruction that models the rolling shutter effect by utilizing both sparse 3D points and the camera trajectory from narrow-baseline images. In this reconstruction, the sparse 3D points are propagated to obtain an initial depth hypothesis using a geometry guidance term. Then, the depth information on each pixel is obtained by sweeping the plane around each depth search space near the hypothesis. The proposed framework shows accurate dense reconstruction results suitable for various sought-after applications. Both qualitative and quantitative evaluations show that our method consistently generates better depth maps compared to state-of-the-art methods.

Joint rolling shutter correction and deblurring (RSCD) techniques are critical for the prevalent CMOS cameras. However, current approaches are still based on conventional energy optimization and are developed for static scenes. To enable learning-based approaches to address real-world RSCD problem, we contribute the first dataset, BS-RSCD, which includes both ego-motion and object-motion in dynamic scenes. Real distorted and blurry videos with corresponding ground truth are recorded simultaneously via a beam-splitter-based acquisition system. Since direct application of existing individual rolling shutter correction (RSC) or global shutter deblurring (GSD) methods on RSCD leads to undesirable results due to inherent flaws in the network architecture, we further present the first learning-based model (JCD) for RSCD. The key idea is that we adopt bi-directional warping streams for displacement compensation, while also preserving the non-warped deblurring stream for details restoration. The experimental results demonstrate that JCD achieves state-of-the-art performance on the realistic RSCD dataset (BS-RSCD) and the synthetic RSC dataset (Fastec-RS). The dataset and code are available at https://github.com/zzh-tech/RSCD., Comment: To be published in CVPR 2021

Physical adversarial examples for camera-based computer vision have so far been achieved through visible artifacts -- a sticker on a Stop sign, colorful borders around eyeglasses or a 3D printed object with a colorful texture. An implicit assumption here is that the perturbations must be visible so that a camera can sense them. By contrast, we contribute a procedure to generate, for the first time, physical adversarial examples that are invisible to human eyes. Rather than modifying the victim object with visible artifacts, we modify light that illuminates the object. We demonstrate how an attacker can craft a modulated light signal that adversarially illuminates a scene and causes targeted misclassifications on a state-of-the-art ImageNet deep learning model. Concretely, we exploit the radiometric rolling shutter effect in commodity cameras to create precise striping patterns that appear on images. To human eyes, it appears like the object is illuminated, but the camera creates an image with stripes that will cause ML models to output the attacker-desired classification. We conduct a range of simulation and physical experiments with LEDs, demonstrating targeted attack rates up to 84%.

Most computer vision techniques rely on cameras which uniformly sample the 2D image plane. However, there exists a class of applications for which the standard uniform 2D sampling of the image plane is sub-optimal. This class consists of applications where the scene points of interest occupy the image plane sparsely (e.g., marker-based motion capture), and thus most pixels of the 2D camera sensor would be wasted. Recently, diffractive optics were used in conjunction with sparse (e.g., line) sensors to achieve high-speed capture of such sparse scenes. One such approach, called “Diffraction Line Imaging”, relies on the use of diffraction gratings to spread the point-spread-function (PSF) of scene points from a point to a color-coded shape (e.g., a horizontal line) whose intersection with a line sensor enables point positioning. In this paper, we extend this approach for arbitrary diffractive optical elements and arbitrary sampling of the sensor plane using a convolution-based image formation model. Sparse scenes are then recovered by formulating a convolutional coding inverse problem that can resolve mixtures of diffraction PSFs without the use of multiple sensors, extending the application of diffraction-based imaging to a new class of significantly denser scenes. For the case of a single-axis diffraction grating, we provide an approach to determine the minimal required sensor sub-sampling for accurate scene recovery. Compared to methods that use a speckle PSF from a narrow-band source or a diffuser-based PSF with a rolling shutter sensor, our approach uses spectrally-coded PSFs from broad-band sources and allows arbitrary sensor sampling, respectively. We demonstrate that the presented combination of the imaging approach and scene recovery method is well suited for high-speed marker based motion capture and particle image velocimetry (PIV) over long periods.

Electric Network Frequency (ENF) fluctuates over time around the nominal value because of power load changes. Such fluctuations have a subtle influence on the artificial light emitted by sources that are supplied by the electric grid. Captured by camera sensors, this information can be used in digital forensic investigations to timestamp, geolocate or authenticate video recordings. However, extracting an ENF signal of sufficient quality from digital videos is still challenging, especially in presence of rolling shutter mechanisms and scene movements that interfere with the ENF extraction. In this paper, we present a novel method for the extraction of ENF signals in video recordings by exploiting parametric and non-parametric spectral estimation, and combining them to limit ENF distortions and increase frequency resolution. The experimental results obtained evaluating our technique using a publicly available dataset demonstrate its effectiveness in timestamping video recordings with a high level of accuracy.

We propose a camera-based positioning system using visible light sources as beacons and a smartphone camera as the receiver, taking advantage of the rolling shutter effect. The sources are modulated using On-Off Keying (OOK) and Manchester encoding, transmitting an unique identifier (ID). An image processing algorithm is proposed for the ID decoding and the fixture localization. For the position estimation, an arbitrary orientation of the receiver on the 3D coordinate space is considered. The system was validated using a setup 1m long, 1m wide and with 2.7m high, containing four light fixtures, and yielding an average position error of 7.4 cm.

During data collection from the images using rolling shutter effect in the optical camera communication, interference from the surrounding light source reduce the system performances. On the other hand, thresholding problem creates after getting the normalized intensity from the image when number of sources appear in the camera’s field of view. Therefore, we applied deep learning approach for removing the interfering light sources and use synchronous thresholding method for data correction. We also observed the performance of signal-error-rate of the system in different condition in Python environment.

Abstract Recently, large progress was made in the development towards low-cost PIV (Particle Image Velocimetry) for industrial and educational applications. This paper presents the use of two low-cost action cameras for stereoscopic planar PIV. A continuous wave laser or alternatively an LED was used for illumination and pulsed by a frequency generator. A slight detuning of the light pulsation and camera frame rate minimizes systematic errors by the rolling shutter effect and allows for the synchronization of both cameras by postprocessing without the need of hardware synchronization. The setup was successfully qualified on a rotating particle pattern in a planar and stereoscopic configuration as well as on the jet of an aquarium pump. Since action cameras are intended to be used at outdoor activities, they are small, very robust and work autarkic. In conjunction with the synchronization and image pre-processing scheme presented herein, those cameras enable stereoscopic PIV in harsh environments and even on moving experiments. Graphic abstract

Underwater 3D laser scanners are an essential type of sensor used by unmanned underwater vehicles (UUVs) for operations such as navigation, inspection, and object recognition and manipulation. Scanners that acquire 3D data by sweeping a laser plane across the scene can provide very high lateral resolution. However, their data may suffer from rolling shutter effect if the change of pose of the robot with respect to the scanned target during the sweep is not negligible. In order to compensate for motion-related distortions without the need for point cloud postprocessing, the 6-DoF pose at which the scanner acquires each line needs to be accurately known. In the underwater domain, autonomous vehicles are often equipped with a high-end inertial navigation system (INS) that provides reliable navigation data. Nonetheless, the relative pose of the 3D scanner with respect to the inertial reference frame of the robot is not usually known a priori. Therefore, this paper uses an ego-motion-based calibration algorithm to calibrate the extrinsic parameters of the visual-inertial sensor pair. Simulations are performed to quantify how miscalibration affects motion-related distortion. The method is also evaluated experimentally in laboratory conditions This research was founded by the GIRONA1000 project (grant number DPI2017-86372-C3-2-R, by the Spanish Ministry of Science), the EUMR project (grant number H2020-INFRAIA-2017-1-twostage-731103, by the European Commission), by the doctoral grant of the University of Girona IFUdG2019 and by the Secretaria d’Universitats i Recerca del Departament d’Economia i Coneixement de la Generalitat de Catalunya under grant 2019FI_B_00812

International audience; A smartphone is a low-cost pocket wireless multichannel multiphysical data acquisition system: the use of such a device for noise and vibration analysis is a challenging task. To what extent is it possible to carry out relevant analysis from it? The Survishno conference, held in Lyon in July 2019, proposed a contest to participants based on this subject. Two challenges were proposed, wherein each a mute video showing an object moving/excited at different frequencies was provided. Due to the frequencies set and the video sampling characteristics, special effects occurred and are visible on both videos. From the first video, participants were asked to estimate the Instantaneous Angular Speed (IAS) of a rotating fan. From the second video, they were asked to perform the modal analysis of a cantilever beam. This paper gathers the interesting ideas proposed by the contestants and proposes a global method to solve these two problems. One major point of the paper might be the advantageous use of the rolling shutter effect, a well-known artefact of smartphone videos, to perform advanced mechanical analyses: the consideration of the unavoidable slight phase shift between the acquisition of each pixel opens up the possibility to perform a dynamic analysis at frequencies that are much higher than the video frame rate.

Compressive lensless imagers enable novel applications in an extremely compact device, requiring only a phase or amplitude mask placed close to the sensor. They have been demonstrated for 2D and 3D microscopy, single-shot video, and single-shot hyperspectral imaging; in each of these cases, a compressive-sensing-based inverse problem is solved in order to recover a 3D data-cube from a 2D measurement. Typically, this is accomplished using convex optimization and hand-picked priors. Alternatively, deep learning-based reconstruction methods offer the promise of better priors, but require many thousands of ground truth training pairs, which can be difficult or impossible to acquire. In this work, we propose the use of untrained networks for compressive image recovery. Our approach does not require any labeled training data, but instead uses the measurement itself to update the network weights. We demonstrate our untrained approach on lensless compressive 2D imaging as well as single-shot high-speed video recovery using the camera's rolling shutter, and single-shot hyperspectral imaging. We provide simulation and experimental verification, showing that our method results in improved image quality over existing methods., Comment: 17 pages, 8 figures

We propose a slight modification to the rolling shutter by shuffling the scanline mechanism to significantly improve its sampling ability to recover high speed videos from a single image using compressive reconstruction algorithms.

We propose and demonstrate using Z-score averaging neural-network (Z-NN) and background-content-removal (BCR) to mitigate the inter-symbol-interference (ISI) in camera-communication (OCC). 950-bit/s over 3-m free-space transmission is achieved.

We propose and demonstrate a light-panel and rolling-shutter-effect (RSE) camera-based visible light communication (VLC) system using Z-score normalization, red/green/blue (RGB) color channel separation, and 1-D artificial neural network (ANN). The proposed scheme can mitigate the high inter-symbol interference (ISI) generated by the RSE pattern due to the low pixel-per-bit and high noise-ratio (NR) of the display contents.

Optical camera communication (OCC) traditionally uses the bright and dark stripes of a rolling shutter pattern captured by the complementary metal–oxide–semiconductor image sensor to realize communication. We assign a unique identification [LED’s unique identification (LED-ID)] to each LED and introduce an LED-ID detection and recognition method of OCC using discrete Fourier transform (DFT). It is different from the traditional LED modulation and demodulation method. In this approach, LED-IDs are modulated to possess different frequency characteristics instead of transmitting data. Taking the DFT of the rolling shutter pattern to obtain a spectrum image, we can recognize the LED-ID according to the characteristics of the spectrum image. The experiment results show that the proposed method has good performance, with at least 426 recognizable LED-IDs and a recognition distance reaching 3 m. Therefore, the proposed method can be considered as one of the effective methods for OCC.

We present an algorithm for estimating consistent dense depth maps and camera poses from a monocular video. We integrate a learning-based depth prior, in the form of a convolutional neural network trained for single-image depth estimation, with geometric optimization, to estimate a smooth camera trajectory as well as detailed and stable depth reconstruction. Our algorithm combines two complementary techniques: (1) flexible deformation-splines for low-frequency large-scale alignment and (2) geometry-aware depth filtering for high-frequency alignment of fine depth details. In contrast to prior approaches, our method does not require camera poses as input and achieves robust reconstruction for challenging hand-held cell phone captures containing a significant amount of noise, shake, motion blur, and rolling shutter deformations. Our method quantitatively outperforms state-of-the-arts on the Sintel benchmark for both depth and pose estimations and attains favorable qualitative results across diverse wild datasets., Project website: https://robust-cvd.github.io/

Rolling-shutter (RS) optical camera communication (OCC) increases the symbol rate much faster than the frame rate of the image sensor. However, transmission distance extension while maintaining sufficient throughput is a technical issue to resolve for indoor OCC. In this study, to increase line-of-sight (LOS) transmission distance and to improve maximum data rate further, RS OCC based on symbol decision using dual-camera spatial luminance distribution is demonstrated. Even at long distance where conventional pixel value threshold is not available for symbol decision and distance extension, two-step thresholds based on spatial luminance distribution are effective for the decision and extension. In addition, ensemble averaging in spatial luminance distribution with dual cameras increases the symbol rate further. In RS OCC using dual-camera spatial luminance distribution, LOS transmission distance increases up to 2.4 meters under error-free conditions. Throughput degradation is suppressed within 36 percent.

In this paper, we propose a novel algorithm for joint calibration of a rolling shutter camera and inertial sensors. Specifically, the proposed method performs online calibration to estimate the camera intrinsics, the readout time, the extrinsics and the time offset between the camera and the IMU, as well as IMU’s bias. By employing a multi-state constrained invariant Kalman filter, the proposed method processes the measurements of IMU and rolling shutter camera in a decoupled way, which greatly reduces the computational complexity of Jacobian matrices and further improves the robustness and accuracy. Experiments on public datasets demonstrate that the proposed approach can accurately calibrate the considered parameters and perform better than state-of-the-art methods in terms of accuracy and robustness.

LED based positioning technology has recently become a popular choice due to the low cost, convenience for maintenance and strong adaptability to the environment. In this paper, we introduce an illuminance-distance analysis method with a rolling shutter camera, for a low-cost indoor positioning system to improve robustness in the case of insufficient LEDs. Firstly, the binary threshold method is used to decode the light signals. Then the gray value information of image is extracted to analyze the depth of LEDs, using the illuminance attenuation theorem. It provides potential solution to the position of devices with the camera by combining depth information, image and camera intrinsic. Our proposed method reduced the minimum number of LEDs observed by the camera from three to two, using only the camera. We evaluated the proposed method with real scene data and demonstrated the effectiveness of illumination-distance model based localization.

This paper proposes a new frame recovery method for screen to camera communication (SCC). SCC is a signal transmission system using a display as transmitter and a camera as receiver. In case of using a CMOS sensor camera, a captured video is distorted by rolling shutter effect which causes bit errors. Against the rolling shutter distortion, we propose a model of distorted video frame and a frame recovery method. In order to recover the frame, we insert known symbols to a transmitted frame. The frame is recovered by solving an optimization problem using the inserted known symbols. To evaluate the proposed method performance, we conduct an experiment using commercial devices. The result shows that the proposed method can accurately recover the transmitted frame and improve bit error rate compared to a simply binarization method and an adaptive binarization method.

In this paper, we present a novel linear algorithm to estimate the 6 DoF relative pose from consecutive frames of stereo rolling shutter (RS) cameras. Our method is derived based on the assumption that stereo cameras undergo motion with constant velocity around the center of the baseline, which needs 9 pairs of correspondences on both left and right consecutive frames. The stereo RS images enable the recovery of depth maps from the semi-global matching (SGM) algorithm. With the estimated camera motion and depth map, we can correct the RS images to get the undistorted images without any scene structure assumption. Experiments on both simulated points and synthetic RS images demonstrate the effectiveness of our algorithm in relative pose estimation., Accepted by International Conference on Image Processing (ICIP 2020)

We present a novel network for rolling shutter effect correction. Our network takes two consecutive rolling shutter images and estimates the corresponding global shutter image of the latest frame. The dense displacement field from a rolling shutter image to its corresponding global shutter image is estimated via a motion estimation network. The learned feature representation of a rolling shutter image is then warped, via the displacement field, to its global shutter representation by a differentiable forward warping block. An image decoder recovers the global shutter image based on the warped feature representation. Our network can be trained end-to-end and only requires the global shutter image for supervision. Since there is no public dataset available, we also propose two large datasets: the Carla-RS dataset and the Fastec-RS dataset. Experimental results demonstrate that our network outperforms the state-of-the-art methods. We make both our code and datasets available at https://github.com/ethliup/DeepUnrollNet.

Remote Photoplethysmography (rPPG) is a fast-growing technique of vital sign estimation by analyzing video of a person. Several major phenomena affecting rPPG signals have been studied (e.g. video compression, distance from person to camera, skin tone, head motions). However, to develop a highly accurate rPPG method, new, minor, factors should be investigated. First considered factor is irregular frame rate of video recordings. Despite of PPG signal transformation by frame rate irregularity, no significant distortion of PPG signal spectra was found in the experiments. Second factor is rolling shutter effect which generates tiny phase shift of the same PPG signal in different parts of the frame caused by progressive scanning. In particular conditions effect of this artifact could be of the same order of magnitude as physiologically caused phase shifts. Third factor is a size of temporal windows, which could significantly influence the estimated error of vital sign evaluation. It follows that one should account difference in size of processing windows when comparing rPPG methods. Short series of experiments were conducted to estimate importance of these phenomena and to determine necessity of their further comprehensive study.

Spatially varying exposures via rolling shutter is an efficient way to capture differently exposed images for high dynamic range (HDR) scenes. Neither camera movement nor moving objects is an issue for such a captured method. However, a possible issue is that the resolution of captured images is reduced. In this paper, we introduce a novel cross image cubic interpolator for the spatially varying exposures via the rolling shutter. Both intra correlation among pixels with the same exposure and inter correlation among pixels with the different exposures are utilized by the proposed interpolator. Experimental results show that quality of upsampled images is significantly improved.

Camera-based visible light communication (CVLC) is a versatile and potential optical wireless communication using rapidly developing smartphone cameras as the receiver. In CVLC, the provision of both flicker-free illumination and high data rate has been a challenging issue. To some extent, however, this issue can be resolved by utilizing a rolling shutter property that is inherited from complementary metal-oxide semiconductor sensors adopted in all smartphone cameras. We propose a practical and versatile wide receiver orientation in CVLC on the basis of this rolling shutter property. That is, unlike previous studies focused on either a line-of-sight (LOS) or non-line-of-sight (NLOS) link only, an orientation independent rolling shutter based CVLC is presented. Based on a unique method of exploiting diffuse reflection of illumination, a wide orientation of the receiver is adequately supported in an indoor environment, thus making the CVLC data reception more practical and convenient. A special transmitter for this wide orientation is designed to ensure that adequate diffuse reflection is present in all directions, regardless of whether a LOS or NLOS link is established. In addition, adaptive algorithms are proposed to ensure an efficient and reliable wide orientation CVLC. Experiments are conducted and analyzed for practical orientations of the receiver in an indoor CVLC environment. While providing sufficient illumination and various orientations, it is found that the proposed scheme achieves a data rate of 6.72 kbps at a distance of up to 50 cm in a small-scale indoor experiment.

The last two decades have witnessed significant advances in the field of indoor positioning systems, which have led to technologies with high location precision. However, there still lacks robust meter-level indoor positioning approaches which only rely on in-building communication infrastructures and smartphones. We present STARLIT, a system that enables a single LED beacon to localize smartphones to within sub-meter. As the smartphone camera contains millions of pixels, we create a virtual sensor array with the camera to measure the received signal strength (RSS) of the LED beacon. Different from the existing camera-based approaches, which need to capture images of the LED within a short light-to-camera distance, we utilize the reflection light from the floor. By exploiting the rolling shutter mechanism in the smartphone cameras, we propose a solution to separate the signal layer from the image background and noise. Given the measured RSSs, we establish an equation set with the Lambertian model and the camera projection model to solve the location of the smartphone. We have implemented STARLIT and evaluated its performance in an office room. Our experiments demonstrate that the STARLIT can achieve a median error of 23cm and an 80-percentile error of 55cm.