Search

Most of Optical Camera Communication (OCC) systems benefit from the rolling shutter mechanism of Complementary Metal-Oxide Semiconductor (CMOS) cameras to record the brightness evolution of the Light-Emitting Diode (LED) through dark and bright strips within images. While this technique enhances the maximum achievable data rate, the main difficulty lies in the demodulation of the signal extracted from images, subject to blooming effect. Thus, two main approaches were proposed to deal with this issue, using adaptive thresholds whose value evolves according to amplitude changes or detecting signal variations with the first-order derivative. As the second method is more robust, a new demodulation method based on the detection of the zeros of the first-order derivative of the extracted signal was proposed in this paper. Obtained results clearly show an improvement in the extracted signal demodulation compared to other methods, achieving a raw Bit Error Rate (BER) of 10−3 around 50 cm in a Line-Of-Sight scenario, and increasing the maximum communication distance by 43.5%, reaching 330 cm in the case of a Non-Line-Of-Sight transmission. [ABSTRACT FROM AUTHOR]

Most of Optical Camera Communication (OCC) systems benefit from the rolling shutter mechanism of Complementary Metal-Oxide Semiconductor (CMOS) cameras to record the brightness evolution of the Light-Emitting Diode (LED) through dark and bright strips within images. While this technique enhances the maximum achievable data rate, the main difficulty lies in the demodulation of the signal extracted from images, subject to blooming effect. Thus, two main approaches were proposed to deal with this issue, using adaptive thresholds whose value evolves according to amplitude changes or detecting signal variations with the first-order derivative. As the second method is more robust, a new demodulation method based on the detection of the zeros of the first-order derivative of the extracted signal was proposed in this paper. Obtained results clearly show an improvement in the extracted signal demodulation compared to other methods, achieving a raw Bit Error Rate (BER) of 10−3 around 50 cm in a Line-Of-Sight scenario, and increasing the maximum communication distance by 43.5%, reaching 330 cm in the case of a Non-Line-Of-Sight transmission.

The potential of VLP is increasing with the rise of indoor mobile machine applications. In this paper, a 3D indoor VLP system based on machine learning and optical camera communications is presented. The system uses electronically controlled LED luminaires as reference points and a rolling shutter CMOS sensor as the receiver. The LED luminaires are modulated using On-Off Keying with unique frequencies. YOLOv5 is used for classification and estimation of the position of each LED luminaire in the image. The pose of the receiver is estimated using a perspective-n-point algorithm. The system was validated using a real-world sized setup containing eight LED luminaires, and achieved an average positioning error of 3.5 cm. The average time to compute the camera pose is approximately 52 ms, which makes it suitable for real-time positioning. To the best of our knowledge, this is the first application of the YOLOv5 algorithm in the field of VLP for indoor environments.

We present a 2D-stitched, 316MP, 120FPS, high dynamic range CMOS image sensor with 92 CML output ports operating at a cumulative date rate of 515 Gbit/s. The total die size is 9.92 cm × 8.31 cm and the chip is fabricated in a 65 nm, 4 metal BSI process with an overall power consumption of 23 W. A 4.3 µm dual-gain pixel has a high and low conversion gain full well of 6600e- and 41,000e-, respectively, with a total high gain temporal noise of 1.8e- achieving a composite dynamic range of 87 dB. [ABSTRACT FROM AUTHOR]

In electric network frequency (ENF)-based video forensics, the analysis of videos captured by rolling shutter systems, where each row of a frame is exposed at different time instances, is critical. To gain the advantage of increased sampling frequency in these videos, in contrast to those captured by the global shutter where an entire frame is exposed at a time, the ENF-related luminance signal that is essential for ENF estimation is built by concatenating ENF-related luminance estimates across consecutive frames. However, this approach brings about some issues or phenomena owing to an idle period at the end of each frame. First, the ENF harmonics may be replaced by new ENF components and attenuated, thereby affecting the reliability of the ENF estimates from these videos. Another critical phenomenon is ENF reversal, which is yet to receive much research. This study comprehensively investigates this phenomenon to explore how and under what conditions the ENF is reversed. Further investigations led this study to examine how the ENF in the emerging components is mainly reconstructed from multiple ENF-related luminance harmonics, depending on the idle period. This helps identify reliable ENF components from which the ENF signal can be accurately estimated. In addition, it reveals the optimal idle periods for any ENF component. Using this outcome, this study also proposes a technique to enhance the effectiveness of an ENF component based on idle period modification. The experimental results show that the proposed method may boost the efficiency of an unreliable ENF component, outperforming the existing techniques.

Many unmanned aerial vehicles (UAV) host rolling shutter (RS) cameras, i.e., cameras where image rows are exposed at slightly different times. As the camera moves in the meantime, this causes inconsistencies in homologous ray intersections in the bundle adjustment, so correction models have been proposed to deal with the problem. This paper presents a series of test flights and simulations performed with different UAV platforms at varying speeds over terrain of various morphologies with the objective of investigating and possibly optimising how RS correction models perform under different conditions, in particular as far as block control is concerned. To this aim, three RS correction models have been applied in various combinations, decreasing the number of fixed ground control points (GCP) or exploiting GNSS-determined camera stations. From the experimental tests as well as from the simulations, four conclusions can be drawn: (a) RS affects primarily horizontal coordinates and varies notably from platform to platform; (b) if the ground control is dense enough, all correction models lead practically to the same mean error on checkpoints; however, some models may cause large errors in elevation if too few GCP are used; (c) in most cases, a specific correction model is not necessary since the affine deformation caused by RS can be adequately modelled by just applying the extended Fraser camera calibration model; (d) using GNSS-assisted block orientation, the number of necessary GCP is strongly reduced. [ABSTRACT FROM AUTHOR]

Geotagging images of interest are increasingly important to law enforcement, national security, and journalism. Today, many images do not carry location tags that are trustworthy and resilient to tampering; and landmark-based visual clues may not be readily present in every image, especially in those taken indoors. In this paper, we exploit an environmental signature from the power grid, the electric network frequency (ENF) signal, which can be inherently captured in a sensing stream at the time of recording and carries useful time–location information. Compared to the recent art of extracting ENF traces from audio and video recordings, it is very challenging to extract an ENF trace from a single image. We address this challenge by first mathematically examining the impact of the ENF embedding steps such as electricity to light conversion, scene geometry dilution of radiation, and image sensing. We then incorporate the verified parametric models of the physical embedding process into our proposed entropy minimization method. The optimized results of the entropy minimization are used for creating a two-level ENF presence–classification test for region-of-capturing localization. It identifies whether a single image has an ENF trace; if yes, whether it is at 50 or 60 Hz. We quantitatively study the relationship between the ENF strength and its detectability from a single image. This paper is the first comprehensive work to bring out a unique forensic capability of environmental traces that shed light on an image’s capturing location. [ABSTRACT FROM AUTHOR]

In this research, the possibility of estimating the time a video was recorded at through electric network frequency is explored by examining various light sources in differentiating circumstances. This research focuses on videos made with smartphones. The smartphone cameras make use of an integrated complementary metal oxide semiconductor sensor. The filmed videos are analyzed using software, which employs a small electric network frequency (ENF) database to determine the time of recording of a video made in experimental circumstances. This research shows that in ideal circumstances, it is possible to determine the time stamp of a video recording made with a smartphone. However, it becomes clear that different light sources greatly influence the outcome. The best results are achieved with Halogen and Incandescent light sources, both of which also seem promising in less ideal circumstances. LED sources do work in ideal circumstances and, however, do not show much success in lesser circumstances. This research further demonstrates that there is potential in using ENF to determine a time stamp of recorded videos and provides validation on prior research on this topic. It proves usable in ideal circumstances with the presence of a clear light source on a white wall. With additional research, it has potential to become a feasible method to use for forensic settings in circumstances that are less ideal. [ABSTRACT FROM AUTHOR]

We propose a marker-based geometric framework for the high-frequency absolute 3D pose estimation of a binocular camera system by using the data captured during the exposure of a single rolling shutter scanline. In contrast to existing approaches enforcing temporal or motion models among scanlines (e.g. linear motion, constant velocity or small motion assumptions), we strive to determine the pose from instantaneous binocular capture (i.e. without using data from previous scanlines) and achieve drift-free pose estimation. We leverage the projective invariants of a novel rigid planar pattern, to both define a geometric reference as well as to determine 2D-3D correspondences from raw edge detection measurements from individual scanlines. Moreover, to tackle the ensuing multi-view estimation problem, achieve real-time operation, and minimize latency, we develop a pair of custom solvers leveraging our geometric setup. To mitigate sensitivity to noise, we propose a geometrically consistent measurement refinement mechanism. We verify the quality of our solvers by comparing with state of the art general solvers for absolute pose estimation of generalized cameras. Finally, we demonstrate the effectiveness of our proposed approach with an FPGA-based implementation which achieves a localization throughput of 129.6 KHz with a $1.5\ \mu \mathsf{s}$ latency. [ABSTRACT FROM AUTHOR]

Current consumer-grade UAV technology is economical, highly automated, and well-suited for large area prospection. At the same time, rapid recording of sites and monuments that cannot be preserved or protected in their entirety is becoming a key research topic. We address the critical issue of designing UAV-based workflows to maximize area coverage on a strictly limited time budget. We illustrate our rationale and the evolution of our methods and results with two case studies from Central Asia: the Medieval fortifications in the Oasis of Bukhara (Uzbekistan) and the Orkhon Valley (Mongolia), which contains a multitude of sites preserved as subtle topographic features that spread across an immense area. We discuss the appropriate use of videogrammetry to complement single-shot imagery and to provide rapid and gap-free data coverage. We also review planning tools and provide best practice guidelines (implementable at low cost) for using current, off-the-shelf hardware and software to highest efficiency. [ABSTRACT FROM AUTHOR]

A method has been developed for calibrating the time intervals of video recorders using a LED panel known as “SEXTA”, which utilizes Global Positioning System (GPS) signal as time reference. This method directly measures the frame intervals frame by frame down to a time resolution of 2 ms of the recordings by video recorders equipped with CCD or CMOS sensors in global shutter or rolling shutter modes respectively. The method has been validated by using a Digital Single Lens Reflex (DSLR) camera and a previous studied dashboard camera. Frame by frame analysis of the calibrated video recording of SEXTA can reveal characteristics of sensors, such as type of sensors (CMOS vs CCD), exposure time, the direction and speed of rolling shutter for CMOS sensors as well as the common characteristics of dashboard cameras such as frame skipping.

Recently, unmanned aerial vehicles (UAVs) have proven to be reliable and affordable alternatives to conventional manned-airplane aerial survey. UAV with small format rolling shutter camera that causes geometric distortion in image tends to suffer in accuracy, while uncertainty assessment mostly focuses on ground control point's and check point's accuracy, yet its impact on intrinsic parameters has not been fully explored. On the other hand, the advancement of small Global Navigation Satellite Systems (GNSS) and Inertial Measurement Units (IMU) have enabled direct georeferencing that doesn't require ground control points, or at least minimum number. The use of less accurate GNSS and IMU in UAV than conventional aerial survey complicates parameter estimation in such as camera calibration parameters, exterior orientation parameters, and so on, which requires attention on the combined impact with rolling shutter compensation option. In this study, the focus is on investigating the combined effect of rolling shutter and GNSS/IMU Georeferencing. Four sets of data with different direct georeferencing and rolling shutter option scenarios were processed using Agisoft Metashape professional. They were analyzed in terms of control/check points accuracy, boresight misalignment, camera calibration parameters, and exterior orientation parameters. The study showed that rolling shutter compensation plays a role in meeting conservative horizontal 1/2 pixels and vertical 1 pixel Root Mean Square Error (RMSE) criteria and that lens distortion residuals are reduced noticeably with rolling shutter option ON. The further investigation showed a coupling effect between intrinsic parameters and rolling shutter compensation effect with different direct georeferencing scenarios, which indicates a strong dependency among principal point offset, affinity, and tangential distortion. [ABSTRACT FROM AUTHOR]

Photogrammetry with unmanned aerial vehicles (UAVs) has become a source of data with extensive applications. The accuracy is of utmost significance, yet the intention is also to find the best possible solutions for data acquisition in economic terms. The objective of the research was the analysis of various variants of the bundle block adjustment. The analysis concerns data which is diversified with respect to the type of shutter (rolling/global), the measurement of external orientation elements, the overlap and the number of ground control points (GCPs). [ABSTRACT FROM AUTHOR]

In optical camera communication (OCC) systems leverage on the use of commercial off-the-shelf image sensors to perceive the spatial and temporal variation of light intensity to enable data transmission. However, the transmission data rate is mainly limited by the exposure time and the frame rate of the camera. In addition, the camera's sampling will introduce intersymbol interference (ISI), which will degrade the system performance. In this paper, an artificial neural network (ANN)-based equaliser with the adaptive algorithm is employed for the first time in the field of OCC to mitigate ISI and therefore increase the data rate. Unlike other communication systems, training of the ANN network in OCC is done only once in a lifetime for a range of different exposure time and the network can be stored with a look-up table. The proposed system is theoretically investigated and experimentally evaluated. The results record the highest bit rate for OCC using a single LED source and the Manchester line code (MLC) non-return to zero (NRZ) encoded signal. It also demonstrates 2 to 9 times improved bandwidth depending on the exposure times where the system's bit error rate is below the forward error correction limit.

In this article we study the adaptation of the concept of homography to Rolling Shutter (RS) images. This extension has never been clearly adressed despite the many roles played by the homography matrix in multi-view geometry. We first show that a direct point-to-point relationship on a RS pair can be expressed as a set of 3 to 8 atomic 3x3 matrices depending on the kinematic model used for the instantaneous-motion during image acquisition. We call this group of matrices the RS Homography. We then propose linear solvers for the computation of these matrices using point correspondences. Finally, we derive linear and closed form solutions for two famous problems in computer vision in the case of RS images: image stitching and plane-based relative pose computation. Extensive experiments with both synthetic and real data from public benchmarks show that the proposed methods outperform state-of-art techniques. [ABSTRACT FROM AUTHOR]

Optical wireless communication (OWC) has emerged as a complementary or alternative technology to the radio-frequency (RF) communication. OWC based on image sensor, which is also called optical camera communication (OCC) has attracted much attention from industrial and academic societies. Here, we discuss several recent OCC technologies. We propose and demonstrate a light emitting diode (LED) light panel and rolling shutter image sensor based OCC system using frame-averaging background removal (FABR) technique, Z-score normalization, and neural network (NN). Here, a driver circuit for the LED display panel based on a bipolar-junction-transistor (BJT) and a metal-oxide-semiconductor field-effect-transistor (MOSFET) is also discussed. It can provide low enough driving frequency from a few Hz to kHz and high bias current for the LED light panel. Experimental results show that the proposed scheme can mitigate the inter-symbol interference (ISI) observed in the rolling shutter pattern produced by the high noise-ratio (NR) of the display contents. [ABSTRACT FROM AUTHOR]

This article describes our ongoing research on real-time digital video stabilization using MEMS-sensors. The authors propose to use the described method for stabilizing the video that is transmitted to the mobile robot operator who controls the vehicle remotely, as well as increasing the precision of video-based navigation for subminiature autonomous models. The article describes the general mathematical models needed to implement the video stabilization module based on the MEMS sensors readings. These models includes the camera motion model, frame transformation model and rolling-shutter model. The existing approaches to stabilization using sensors data were analyzed and considered from the point of view of the application in a real-time mode. This article considers the main problems that came up during the experiments that were not resolved in the previous research papers. Such problems include: calibration of the camera and sensors, synchronization of the camera and sensors, increasing the accuracy of determining the camera position from sensors data. The authors offer possible solutions to these problems that would help improve quality of the work of existing algorithms, such as a system for parallel synchronized recording of video and sensor data based on the Android operating system. As the main result, the authors represent a framework for implementing video stabilization algorithms based on MEMS sensors readings.

We propose an original approach to absolute pose and structure-from-motion (SfM) which handles rolling shutter (RS) effects. Unlike most existing methods which either augment global shutter projection with velocity parameters or impose continuous time and motion through pose interpolation, we use local differential constraints. These are established by drawing analogies with non-rigid 3D vision techniques, namely shape-from-template and non-rigid SfM (NRSfM). The proposed idea is to interpret the images of a rigid surface acquired by a moving RS camera as those of a virtually deformed surface taken by a GS camera. These virtually deformed surfaces are first recovered by relaxing the RS constraint using SfT or NRSfM. Then we upgrade the virtually deformed surface to the actual rigid structure and compute the camera pose and ego-motion by reintroducing the RS constraint. This uses a new 3D-3D registration procedure that minimizes a cost function based on the Euclidean 3D point distance. This is more stable and physically meaningful than the reprojection error or the algebraic distance used in previous work. Experimental results obtained with synthetic and real data show that the proposed methods outperform existing ones in terms of accuracy and stability, even in the known critical configurations. [ABSTRACT FROM AUTHOR]

LED-to-camera communication allows LEDs deployed for illumination purposes to modulate and transmit data which can be received by camera sensors available in mobile devices like smartphones, wearable smart-glasses, etc. Such communication has a unique property that a user can visually identify a transmitter (i.e., LED) and specifically receive information from the transmitter. It can support a variety of novel applications such as augmented reality through mobile devices, navigation using smart signs, fine-grained location specific advertisement, etc. However, the achievable data rate in current LED-to-camera communication techniques remains very low to support any practical application. In this paper, we present $\mathsf {ColorBars}$ ColorBars , an LED-to-camera communication system that utilizes Color Shift Keying (CSK) to modulate data using different colors transmitted by the LED. It exploits the increasing popularity of Tri-LEDs (RGB) that can emit a wide range of colors. We show that commodity cameras can efficiently and accurately demodulate the color symbols. $\mathsf {ColorBars}$ ColorBars ensures flicker-free and reliable communication even in the presence of inter-frame loss and diversity of rolling shutter cameras. We implement $\mathsf {ColorBars}$ ColorBars on embedded platform and evaluate it with Android and iOS smartphones as receivers. Our evaluation shows that $\mathsf {ColorBars}$ ColorBars can achieve a data rate of 7.7 Kbps on Nexus 5, 3.7 Kbps on iPhone 5S, and 2.9 Kbps on Samsung Note8. It is also shown that lower CSK modulations (e.g., four and eight CSK) provide extremely low symbol error rates ($<10^{-3}$ < 10 - 3 ), making them a desirable choice for reliable LED-to-camera communication. [ABSTRACT FROM AUTHOR]

We present minimal, non-iterative solutions to the absolute pose problem for images from rolling shutter cameras. The absolute pose problem is a key problem in computer vision and rolling shutter is present in a vast majority of today's digital cameras. We discuss several camera motion models and propose two feasible rolling shutter camera models for a polynomial solver. In previous work a linearized camera model was used that required an initial estimate of the camera orientation. We show how to simplify the system of equations and make this solver faster. Furthermore, we present a first solution of the non-linearized camera orientation model using the Cayley parameterization. The new solver does not require any initial camera orientation estimate and therefore serves as a standalone solution to the rolling shutter camera pose problem from six 2D-to-3D correspondences. We show that our algorithms outperform P3P followed by a non-linear refinement using a rolling shutter model. [ABSTRACT FROM AUTHOR]

Mobile wireless communication heavily relies on the radio frequency to convey message and data. However, its limited spectrum can hardly meet the demands for the future high data rate applications. Optical wireless communication, in particular visible light communication, opens up vast optical spectrum for communication, and meanwhile can retrofit the light sources as the communication transmitters in the existing working or living environments. In conjunction with the ubiquitous cameras in hand-held consumer electronics such as smartphones and pads, optical camera communication (OCC) further takes advantages of image sensors as the communication receivers and realizes low-cost communication systems. This article first provides an overview of OCC systems. It then addresses some practical constraints, ranging from sensor low frame rate and instability, rolling shutter readout, to visual qualities of displayed images and videos, and link blockage between the transmitter and receiver. Accordingly, it introduces existing and new solutions to deal with those constraints by data modulation, newly developed camera structures, post-processing of sensed signals and non-line of sight OCC as a new form. In particular, indirect paths by either the indoor surface reflection or the outdoor atmospheric scattering are explored for link connectivity under blockage. Finally, some future research directions are suggested. This article is part of the theme issue 'Optical wireless communication'. [ABSTRACT FROM AUTHOR]

Satellite platform jitter is a non-negligible factor that affects the image quality of optical cameras. Considering the limitations of traditional platform jitter detection methods that are based on attitude sensors and remote sensing images, this paper proposed a jitter detection method using sequence CMOS images captured by rolling shutter for high-resolution remote sensing satellite. Through the three main steps of dense matching, relative jitter error analysis, and absolute jitter error modeling using sequence CMOS images, the periodic jitter error on the imaging focal plane of the spaceborne camera was able to be measured accurately. The experiments using three datasets with different jitter frequencies simulated from real remote sensing data were conducted. The experimental results showed that the jitter detection method using sequence CMOS images proposed in this paper can accurately recover the frequency, amplitude, and initial phase information of satellite jitter at 100 Hz, 10 Hz, and 2 Hz. Additionally, the detection accuracy reached 0.02 pixels, which can provide a reliable data basis for remote sensing image jitter error compensation.

The use of consumer grade unmanned aerial vehicles (UAV) is becoming more and more ubiquitous in photogrammetric applications. A large proportion of consumer grade UAVs are equipped with CMOS image sensor and rolling shutter. When imaging with a rolling shutter camera, the image sensor is exposed line by line, which can introduce additional distortions in image space since the UAV navigates at a relatively high speed during aerial acquisitions. In this paper, we propose (1) an approach to calibrate the readout time of rolling shutter camera, (2) a two-step method to correct the image distortion introduced by this effect. The two-step method makes assumption that during exposure, the change of camera orientation is negligible with respect to the change of camera position, which is often the case when camera is fixed on a stabilized mount. Firstly, the camera velocity is estimated from the results of an initial bundle block adjustment; then, one camera pose per scan-line of the image sensor is recovered and image observations are corrected. To evaluate the performance of the proposed method, four datasets of block and corridor configurations are acquired with the DJI Mavic 2 Pro and its original Hasselbald L1D-20c camera. The proposed method is implemented in MicMac , a free, open-source photogrammetric software; comparisons are carried out with other two mainstream software, AgiSoft MetaShape and Pix4D , which also have the functionality of rolling shutter effect correction. For block configuration datasets, the three software give comparable results. AgiSoft Metashape and Pix4D are sensitive to the flight configuration and encounter difficulties when processing datasets in corridor configurations. The proposed method shows good robustness both in block and corridor configurations, and is the only method that works in corridor configuration. After the application of the rolling shutter effect correction, the 3D accuracy is improved by 30–60% in block configuration and 15–25% in corridor configuration. A further improvement can be expected if a precise dating of image is available or if the camera positions can be directly extracted from GNSS data. [ABSTRACT FROM AUTHOR]

This paper addresses the problem of star identification in the presence of high slew rates, false objects and image deformations introduced by the rolling shutter. These problems can affect the operating life of star trackers and worsen the nominal performances. The proposed methodology relies on a technique named Improved Multi-Poles Algorithm, especially designed for robustness to false objects and slew rates. Angular velocities up to five degrees per second are considered so that stars are seen no more as near-circular spots but appear as streaks. The image deformation due to the rolling shutter of modern active pixel sensor detectors is compensated by means of a mathematical model based on a first order approximation of problem. A star tracker high fidelity simulator generates the input images considering typical noises due to the electronics and space environment. The reported results show that the proposed approach guarantees a reliable star identification and attitude determination with angular velocity from zero to five degrees per second. • Star identification performed with slew rates, false objects and APS deformation. • Robustness to false objects guaranteed by the Improved Multi-Poles Algorithm. • A rolling shutter compensation model is proposed to consider image deformation. • The results are obtained using a high-fidelity star tracker simulator. • Reliable star identification guaranteed with angular rate from 0 to 5 deg/sec. [ABSTRACT FROM AUTHOR]

Light-to-Camera Communications (LCC) have emerged as a new wireless communication technology with great potential to benefit a broad range of applications. However, the existing LCC systems either require cameras directly facing to the lights or can only communicate over a single link, resulting in low throughputs and being fragile to ambient illuminant interference. We present HYCACO, a novel LCC system, which enables multiple light emitting diodes (LEDs) with an unaltered camera to communicate via the non-line-of-sight (NLoS) links. Different from other NLoS LCC systems, the proposed scheme is resilient to the complex indoor luminous environment. HYCACO can decode the messages by exploring the mixed reflected optical signals transmitted from multiple LEDs. By further exploiting the rolling shutter mechanism, we present the optimal optical frequencies and camera exposure duration selection strategy to achieve the best performance. We built a hardware prototype to demonstrate the efficiency of the proposed scheme under different application scenarios. The experimental results show that the system throughput reaches 4.5 kbps on iPhone 6s with three transmitters. With the robustness, improved system throughput and ease of use, HYCACO has great potentials to be used in a wide range of applications such as advertising, tagging objects, and device certifications. [ABSTRACT FROM AUTHOR]

Electric network frequency (ENF) is a time-varying signal of the frequency of mains electricity in a power grid. It continuously fluctuates around a nominal value (50/60 Hz) due to changes in the supply and demand of power over time. Depending on these ENF variations, the luminous intensity of a mains-powered light source also fluctuates. These fluctuations in luminance can be captured by video recordings. Accordingly, the ENF can be estimated from such videos by the analysis of steady content in the video scene. When videos are captured by using a rolling shutter sampling mechanism, as is done mostly with CMOS cameras, there is an idle period between successive frames. Consequently, a number of illumination samples of the scene are effectively lost due to the idle period. These missing samples affect the ENF estimation, in the sense of the frequency shift caused and the power attenuation that results. This paper develops an analytical model for videos captured using a rolling shutter mechanism. This model illustrates how the frequency of the main ENF harmonic varies depending on the idle period length, and how the power of the captured ENF attenuates as idle period increases. Based on this, a novel idle period estimation method for potential use in camera forensics that is able to operate independently of video frame rate is proposed. Finally, a novel time-of-recording verification approach based on the use of multiple ENF components, idle period assumptions, and the interpolation of missing ENF samples is also proposed. [ABSTRACT FROM AUTHOR]

This paper revisits the problem of continuous-time structure from motion, and introduces a number of extensions that improve convergence and efficiency. The formulation with a C 2 -continuous spline for the trajectory naturally incorporates inertial measurements, as derivatives of the sought trajectory. We analyze the behavior of split spline interpolation on SO (3) and on R 3 , and a joint spline on SE (3) , and show that the latter implicitly couples the direction of translation and rotation. Such an assumption can make good sense for a camera mounted on a robot arm, but not for hand-held or body-mounted cameras. Our experiments in the Spline Fusion framework show that a split spline on R 3 and SO (3) is preferable over an SE (3) spline in all tested cases. Finally, we investigate the problem of landmark reprojection on rolling shutter cameras, and show that the tested reprojection methods give similar quality, whereas their computational load varies by a factor of two. [ABSTRACT FROM AUTHOR]

Structure from small motion has become an important topic in 3D computer vision as a method for estimating depth, since capturing the input is so user-friendly. However, major limitations exist with respect to the form of depth uncertainty, due to the narrow baseline and the rolling shutter effect. In this paper, we present a dense 3D reconstruction method from small motion clips using commercial hand-held cameras, which typically cause the undesired rolling shutter artifact. To address these problems, we introduce a novel small motion bundle adjustment that effectively compensates for the rolling shutter effect. Moreover, we propose a pipeline for a fine-scale dense 3D reconstruction that models the rolling shutter effect by utilizing both sparse 3D points and the camera trajectory from narrow-baseline images. In this reconstruction, the sparse 3D points are propagated to obtain an initial depth hypothesis using a geometry guidance term. Then, the depth information on each pixel is obtained by sweeping the plane around each depth search space near the hypothesis. The proposed framework shows accurate dense reconstruction results suitable for various sought-after applications. Both qualitative and quantitative evaluations show that our method consistently generates better depth maps compared to state-of-the-art methods. [ABSTRACT FROM AUTHOR]

This article investigates the viability of using a smartphone as a cost-effective tool for rotational speed extraction. The proposed method exploits the geometrical image deformations induced by a smartphone's rolling shutter camera, which reads pixel lines sequentially with a delay equal to the rolling shutter period. This characteristic period is the basis for the developed methodology and allows measuring constant and varying speeds above the Nyquist limit related to the camera's frame rate. With respect to the rotating shaft, the smartphone is pointed to the side surface of the shaft (radial viewpoint). A method is proposed to measure the speed from the deformation of a zebra pattern as a consequence of the sequential readout of a rolling shutter camera. A mathematical model, which also takes into account a possible misalignment between the shaft and the camera's rolling shutter reading direction, is developed and validated on an in-house test rig. It is shown that the proposed method is robust against changing lighting conditions and a misalignment between the shaft and the smartphone camera. Normalized Root-Mean-Square Errors (NRMSE) of about 3% (or less) are reached for the smartphone aligned with the shaft with peaks in the percentage error of about 5% to 8%. Nevertheless, even for a misalignment of 60°, the NRMSE is only 4% with variations in the percentage error to about 10%. The proposed method shows promising results with room for improvements, making it possible to develop a smartphone application to measure a shaft's rotational speed in the future. [Display omitted] • A smartphone camera is a cost-effective tool for rotational speed extraction. • The speed is extracted from a radial viewpoint. • A smartphone camera allows measuring constant and varying speeds. • The measured speed can be above the Nyquist limit related to the camera's frame rate. • The rolling shutter effect is the key to measure high speeds with low frame rates. [ABSTRACT FROM AUTHOR]

Complementary metal-oxide semiconductor image sensors suffer from undesirable geometric image distortion due to the rolling shutter effect. However, conventional algorithms cannot be used for strict real-time use in mobile devices. An extremely low computational method to compensate rolling shutter effect for the mobile devices is proposed. The proposed method efficiently predicts the rotational camera motion in three-dimensional space from the global motion vectors predicted in twodimensional image space. Hence, the computational complexity is extremely reduced. To further improve the video quality, the video stabilization to reduce the high-frequency jitter of the camera motion without increasing the complexity is introduced. The results of simulation demonstrate that while the computational complexity is extremely low, the proposed algorithm provides acceptable performance. [ABSTRACT FROM AUTHOR]

We propose a 3-step method for structure and motion computation from two or more images taken by a one or multiple moving rolling shutter cameras. This work is motivated by the realization that existing reconstruction methods using rolling shutter images do not give satisfactory results or even fail in many configurations due to singularities and degenerate configurations. The first contribution consists in decoupling the rotate ego motion from the remaining parameters by adding a constraint on image curves basing on the a priori knowledge that they correspond to world 3D straight lines with unknown directions. Straight lines frequently appear in man-made environments such as urban or indoor scenes. After introducing the parameterization of a curve projected from a 3D straight line observed by a moving camera using three rolling shutter projection models, we show how to linearly extract angular velocity of each camera by using detected curves. Then we develop a linear method to recover the translational velocities and the motion between the cameras using point-matches, after compensating effects of angular velocity on each image. The second contribution consists in a novel point based bundle adjustment for rolling shutter cameras (C-RSBA) which does not consider a static row index during structure and motion optimization contrarily to existing methods. This enables to refine the parameters obtained thanks to the straightness constraint by avoiding degenerate configurations, thus outperforming existing RSBA methods. The approach was evaluated on both synthetic and real data. [ABSTRACT FROM AUTHOR]