Search

In this paper, we provide a solution based on fast Fourier Transform (FFT) for frequency detection and interference analysis in a non-line-of-sight (NLOS) multiple-input multiple-output (MIMO) optical camera communications (OCC) link. The proposed NLOS MIMO-OCC uses two organic light-emitting diodes (OLEDs) and a camera as transmitters (Txs) and the receiver, respectively, and reflections from various surfaces such as plain wall, paper and wooden surfaces. In this paper, for the first time, we analyze the performance of NLOS MIMO-OCC for two indoor scenarios utilizing two OLED Txs that are in phase and modulated at the same modulation frequency $f_{s}$ and especially when they are out of phase and under different $f_{s}$ . We carry out FFT for $f_{s}$ detection and interference analysis for OLEDs operating at different phases and $f_{s}$ . To analyze the performance of NLOS MIMO-OCC, first FFT-based $f_{s}$ detection is performed to identify Txs followed by evaluation of the reception success rates $R_{\mathrm {rs}}$ for a range of camera exposure times $t_{\exp }$ and the gain $G_{v}$ for a given $f_{s}$ . The results show that for two OLEDs in phase and different phases at $t_{\exp }$ of 1500, $G_{v}$ of 45 dB and a link span of 2 m, $R_{\mathrm {rs}}$ of 94% is achieved at $f_{s}$ of 600 Hz for the plain wall and paper reflectors. For interference analysis, the proposed FFT analysis gives accurate, i.e., 100 % $f_{s}$ detection in all the above cases.

Traffic monitoring systems featuring robust, multi-sensor fusion capabilities are rapidly growing in demand to observe traffic flow, reduce congestion and to detect and report traffic accidents. However, monitoring outdoor environments using cameras remains challenging due to complex weather conditions, including fog, rain, snow and variable lighting conditions. The presence of these weather conditions can significantly reduce vehicle detection and classification performance using machine learning methods. Unfortunately, openly available datasets for multi-sensor traffic monitoring development and testing remain limited, especially those featuring infrastructure-based cameras and millimeter wave (mmWave) radar. To address these challenges, we evaluate open camera and mmWave radar data using vehicle classification models for cars, trucks, vans and buses on embedded hardware. We also provide an open multi-sensor traffic monitoring dataset with more than 8,000 manually annotated frames as well as mmWave radar point clouds recorded in an urban environment under sunny, partially cloudy, cloudy, rainy and night conditions.

Environment perception using camera, radar, and/or lidar sensors has significantly improved in the last few years because of deep learning-based methods. However, a large group of these methods fall into the category of supervised learning, which requires a considerable amount of annotated data. Due to uncertainties in multi-sensor data, automating the data labeling process is extremely challenging; hence, it is performed manually to a large extent. Even though full automation of such a process is difficult, semi-automation can be a significant step to ease this process. However, the available work in this regard is still very limited; hence, in this paper, a novel semi-automatic annotation methodology is developed for labeling RGB camera images and 3D automotive radar point cloud data using a smart infrastructure-based sensor setup. This paper also describes a new method for 3D radar background subtraction to remove clutter and a new object category, GROUP, for radar-based object detection for closely located vulnerable road users. To validate the work, a dataset named INFRA-3DRC is created using this methodology, where 75 % of the labels are automatically generated. In addition, a radar cluster classifier and an image classifier are developed, trained, and tested on this dataset, achieving accuracy of 98.26% and 94.86%, respectively. The dataset and Python scripts are available at https://fraunhoferivi.github.io/INFRA-3DRC-Dataset/.

The integration of cameras and millimeter-wave radar into sensor fusion algorithms is essential to ensure robustness and cost effectiveness for vehicle pose estimation. Due to the low resolution of traditional radar, several studies have investigated 4D imaging radar, which provides range, Doppler, azimuth, and elevation information with high resolution. This paper presents a method for robustly estimating vehicle pose through 4D radar and camera fusion, utilizing the complementary characteristics of each sensor. Leveraging the single-view geometry of the detected vehicle bounding box, the lateral position is derived based on the camera images, and the yaw rate is calculated through feature matching between consecutive images. The high-resolution 4D radar data are used to estimate the heading angle and forward velocity of the target vehicle by leveraging the position and Doppler velocity information. Finally, an extended Kalman filter (EKF) is employed to fuse the physical quantities obtained by each sensor, resulting in more robust pose estimation results. To validate the performance of the proposed method, experiments were conducted in foggy environments, including straight and curved driving scenarios. The experimental results indicate that the performance of the camera-based method is reduced due to frame loss in visually challenging scenarios such as foggy environments, whereas the proposed method exhibits superior performance and enhanced robustness.

Detection based on LiDAR and camera fusion is increasingly popular for researchers in the autonomous driving domain. Compared to the camera-only and LiDAR-only methods, the fusion-based methods indeed improve the detection accuracy on public-available datasets. However, due to the complexity of the projection or fusion mechanism, few of these methods can run in real time even on an advanced desktop GPU. Thus, in this paper, we propose a new fusion detection model G-Fusion with a light and fast image view-transform module. According to our receptive field analysis of image feature maps, we directly project image features to only one voxel layer located on the ground, then fuse the LiDAR and image features by concatenation and convolution. With this delicately designed module, G-Fusion greatly boosts the state-of-the-art speed performance on the nuScenes dataset, achieving a good balance with the competitive detection scores. Meanwhile, since the precision of sensor extrinsic parameters is important for most fusion-based methods, we also deeply dig into our model’s calibration error tolerance ability and discover the failure noise condition.

Extrinsic calibration of a 2D camera and a 2D LiDAR is necessary to fuse information from two sensors by representing the information under the same frame. Various geometric constraints such as point-plane, point-line, and point-point are used for the extrinsic calibration. Usually, these require a manual step, including control points selection for camera calibration and LiDAR points. We propose a new algorithm for automatic extrinsic calibration with point-line correspondences. A calibration structure with two perpendicular planes having a chessboard on both sides is used for the extrinsic calibration. First, we use predefined colors at specific locations on a chessboard to quickly find the origin of the coordinate system. Second, we robustly detect three control points on LiDAR raw data using a geometric constraint that two end points among three control points should lie on the same line. The initial linear solution is obtained by using a point-line constraint. Finally, it is refined by nonlinear minimization, which gives a 15.3% improvement compared to the linear solution. Experimental results show the feasibility of the proposed algorithm.

Measuring the level of engagement among participants in a meeting is crucial for evaluating collective understanding. While previous studies have utilized multiple sensors, such as wearable devices, to gauge engagement levels in offline environments, the shift to remote meetings due to the COVID-19 pandemic presents new challenges. In this study, we propose a method for measuring student engagement during online meetings using only the built-in web cameras on their devices. We collect high, middle, and low engagement level recording data from 24 students. We decided to collect data using the role-acting approach instead of conventional self-reporting or post-experiment annotation. With the feature extraction based approach, we achieved a classification rate of 46.7%. With a deep learning based approach, we achieved a classification rate of 89.5% using MobileNetV2 for leave-one-participant-out cross-validation, which demonstrated higher accuracy than previous studies. From the model, we implement an application EnGauge and conduct a pilot study. The results demonstrate a new approach to data collection, an optimal engagement level recognition model, and application scenarios.

Many events, such as robberies, missing people, and other suspicious activities, are often captured by cameras. However, these videos are, more often than not, short and not from an optimal angle. Biometric recognition techniques such as facial or iris recognition are inadequate for situations like this. Gait recognition techniques are more suitable than other types of biometrics in such situations. In this work, we propose a new gait recognition technique using viewpoint normalization and a sliding window process. The proposed technique is designed to handle short walking videos captured from any angle. The proposed technique consists of 3 steps. First, a 2-second walk is preprocessed using the sliding window process. This step allows us to generate more gait data in a form of a set of sliding windows from only a 2-second walk. Then sliding windows are transformed into the optimal viewpoint using $\mathit {ViewNet}$ , a proposed neural network designed for finding and transforming sliding windows into the optimal viewpoint angle. Finally, local joint movement information is extracted from sliding windows and used to identify a person using $\mathit {IdenNet}$ , a proposed neural network designed for identifying a person from local joint movements. Four evaluation methods, the top $k$ accuracy test, the precision-recall curves, the cumulative matching characteristic curves, and the gallery-size test, are used to assess the proposed technique. The experimental results show that the proposed technique outperforms existing techniques on all four tests. In particular, the proposed technique can provide a small group, not more than 5, of suspects with a chance above 90% that the real person of interest is in the group. Moreover, the proposed technique still maintains high accuracy even when used with a larger pool of people.

With the rapid growth of self-driving vehicles, automobiles demand diverse data from multiple sensors to perceive the surrounding environment. Calibrating preprocessing between multiple sensors is necessary to utilize the data effectively. In particular, the LiDAR-camera pair, a suitable complement with 2D-3D information for each other, has been widely used in autonomous vehicles. Most traditional calibration methods require specific calibration targets set up under complicated environmental conditions, which require expensive human manual work. In this study, we propose a deep neural network that does not require any specific targets and offline setup to find the six degrees of freedom (6 DoF) transformation between LiDAR and the camera. Unlike previous deep learning CNN-based methods, which use raw 3D point clouds and 2D images frame by frame, CalibBD utilizes Bi-LSTM for sequence data to extract temporal features between consecutive frames. It not only predicts the calibration parameters by minimizing both transformation and depth losses but also calibrates the camera parameters by using temporal loss to refine the calibration parameters. The proposed model achieves a steady performance under various deviations of mis-calibration parameters and achieved higher results in terms of accuracy than the state-of-the-art CNN-based method on the KITTI datasets.

Behavioural symptoms of dementia present a significant risk within Long Term Care (LTC) homes, which face difficulties supporting residents and monitoring their safety with limited staffing resources. Many LTC facilities have installed video surveillance systems in common areas that can help staff to observe residents; however, typically these video streams are not monitored. In this paper, we present the development of a computer vision algorithm to use these video streams to detect episodes of clinically important agitation in people with dementia. Given that episodes of agitation are rare in comparison to normal behaviours, we formulated this as an anomaly detection problem. This involves using the video camera to monitor the scene rather than tracking individuals. We developed a customized spatio-temporal convolution autoencoder that is trained on the normal behaviours and then identified agitation during testing as anomalous behaviour. We present a proof-of-concept using video data collected from a specialized dementia unit and annotated for agitation events. We trained the unsupervised neural network on approximately 24 hours of normal activities and tested on 11 hours of videos containing both normal activities and agitation events, and obtained an area under the curve of the receiver operating characteristic curve of 0.754. This research paves the way for leveraging existing surveillance infrastructure in LTC and other mental health settings to detect agitation or aggression, with the potential for improved health and safety.

Communication protocols and wireless networking technique is a cohesive part of any light management system. IEEE 802.15.4 standard based networking techniques for the climate-adaptive light management system and its energy model for sensor actuator node is the focus of this paper. This light integrated scheme with adaptive controls provides the desired illuminance at appropriate times with uniformity, by reducing the discomfort glare and energy use. The first part of this paper investigates the architecture and energy consumption of camera-based wireless sensor-actuator nodes with the help of energy models, including a control unit for the Light Management System. Then focused on the energy savings and techno-economic analysis of this wireless networked lighting system in a test room with automated control of Light Emitting Diode luminaire and Venetian blinds. The wireless sensor actuator networked lighting system performance is analyzed by evaluating the node energy consumption in idle and active mode with real time measurements. The energy consumption evaluation of the nodes allows users to improve node life time and think about power management schemes. The climate-adaptive control scheme shows improved uniformity and significant energy savings.

This paper proposes a map-matching based precise positioning system for highways. The proposed system fuses multiple sensors such as a camera, conventional GPS, and vehicle motion sensors widely adapted in mass-produced vehicles. While conventional map-matching based systems consume a large amount of computation because of the landmark detection, the proposed system achieves both precise positioning and small computations by introducing a detection-verification-cascade strategy: the proposed system generates vehicle position candidates per lane by utilizing lane endpoints detectable with small computation, and then it selects the candidate on an ego-lane by verifying the existence of road signs within the region of interest generated by a map and each candidate. To reduce the computation further, the proposed system adapts an extended Kalman filter (EKF) as a positioning filter instead of a particle filter. As the noise distribution of the EKF is limited to a unimodal, the proposed system maintains multiple EKFs to track the candidates for each lane until the ego lane is identified. After ego-lane identification, the system leaves only one EKF to track the candidate on an ego-lane. The proposed system achieves 0.17m average positioning error in highway situations and it takes about 53ms to process the whole procedure from landmark detection to position estimation in low-end hardware equipped with a cortex-A9 CPU whose clock is 1.0 GHz.

Autonomous driving applications use two types of sensor systems to detect vehicles - depth sensing LiDAR and radiance sensing cameras. We compare the performance (average precision) of a ResNet for vehicle detection in complex, daytime, driving scenes when the input is a depth map [D = d(x,y)], a radiance image [L = r(x,y)], or both [D,L]. (1) When the spatial sampling resolution of the depth map and radiance image are both equal to typical camera resolutions, a ResNet detects vehicles at higher average precision from depth than radiance. (2) When the spatial sampling of the depth map matches the range of current LiDAR devices, the average precision is higher for radiance than depth. (3) A hybrid system that combines depth and radiance has higher average precision than systems using depth or radiance alone. We confirm these observations in both simulation and real-world data. We explain the advantage of combining depth and radiance by noting that the two types of information have complementary weaknesses. The radiance data are limited by dynamic range, motion blur and illumination variation; the LiDAR data have low spatial resolution. The ResNet effectively combines the two data sources to improve vehicle detection.

The measurement of heart rate variability (HRV) is the preferred method for assessing the function of the autonomic nervous system (ANS). Traditional HRV detection requires an electrocardiogram (ECG) or photoelectric sensor. In this paper, we propose a new method for HRV measurement using a rear smartphone camera as a sensor. Video signals from the fingertips of 24 college students were acquired using the rear camera of an HTC M8d smartphone. ECG signals were simultaneously recorded as reference. The video signals were converted into single-frame image sequences over time. Each image frame was transformed into point data through superpositioning of pixel color attribute values and averaging according to space. The point data were sorted by time to obtain a photoplethysmogram (PPG). Finally, the Hilbert transform was used to extract the pulse-to-pulse interval and the R-to-R interval for the PPG and ECG, respectively. Sixteen HRV parameters (mean HR, max HR, min HR, SDNN, RMSSD, NN50, pNN50, VLF, LF, HF, TP, LFnu, HFnu, LF/HF, SD1, and SD2) were analyzed. All 16 HRV parameters were highly correlated (all rs > 0.95, ps

Fast and accurate global localization of autonomous ground vehicles is often required in indoor environments and GPS-shaded areas. Typically, with regard to global localization problem, the entire environment should be observed for a long time to converge. To overcome this limitation, a new initialization method called deep initialization is proposed and it is applied to Monte Carlo localization (MCL). The proposed method is based on the combination of a three-dimensional (3D) light detection and ranging (LiDAR) and a camera. Using a camera, pose regression based on a deep convolutional neural network (CNN) is conducted to initialize particles of MCL. Particles are sampled from the tangent space to a manifold structure of the group of rigid motion. Using a 3D LiDAR as a sensor, a particle filter is applied to estimate the sensor pose. Furthermore, we propose a re-localization method for performing initialization whenever a localization failure or the situation of robot kidnapping is detected. Either the localization failure or the kidnapping is detected by combining the outputs from a camera and 3D LiDAR. Finally, the proposed method is applied to a mobile robot platform to prove the method's effectiveness in terms of both the localization accuracy and time consumed for estimating the pose correctly.

Achieving accurate and reliable positioning in dynamic urban scenarios using low-cost vehicular onboard sensors, such as the global navigation satellite systems (GNSS), camera, and inertial measurement unit (IMU), is still a challenging problem. Multi-Agent collaborative integration (MCI) opens a new window for achieving this goal, by sharing the sensor measurements between multiple agents to further improve the accuracy of respective positioning. One of the major difficulties in MCI is to effectively connect all the sensor measurements arising from multiple independent agents. The popular approach is to find the overlapping areas between agents using active sensors, such as cameras. However, the performance of overlapping area detection is significantly degraded in outdoor urban areas due to the challenges arising from numerous unexpected moving objects and unstable illumination conditions. To fill this gap, this paper proposes to leverage both the camera-based overlapping area detection and the inter-ranging measurements to boost the cross-connection between multi-agents and brings the MCI to outdoor urban scenarios using low-cost onboard sensors. Moreover, a novel MCI framework is proposed to integrate the sensor measurements from the low-cost GNSS receiver, camera, IMU, and inter-ranging using state-of-the-art factor graph optimization (FGO) to fully explore their complementary properties. The proposed MCI framework is validated using two challenging datasets collected in urban canyons of Hong Kong. We conclude that the proposed MCI framework can effectively improve the positioning accuracy of the respective agents in the evaluated datasets. We believe that the proposed MCI framework has the potential to be prevalently adopted by the connected intelligent transportation systems (ITS) applications to provide robust positioning using low-cost onboard sensors in urban scenarios.

For the first time, the characteristics of an evolving commercial camera technology that can operate at millimeter-wave frequencies has been independently investigated. In this work, we benchmark the TeraSense camera against a custom-built single-pixel camera at W-band, for image quality and aperture reflectance. It is found that the Tera-1024 TeraSense camera exhibits limited image resolution and fidelity, with significant levels of systematic spatial noise. In a poor signal-to-noise ratio scenario, the addition of random noise exacerbates these problems. Possible causes of both beam and image distortion have been identified in quasi-optical applications, which gives important insight into the best use of (sub-)THz cameras and interpretation of their images. Inherent standing waves caused by the significant power reflectance of the camera aperture is investigated in detail. A simple W-band one-port quasi-optical scalar network analyzer is developed, to determine the levels of reflectance for both cameras, with its bespoke calibration routine derived from first principles - providing a low-cost solution for many non-destructive testing applications. It is found that the TeraSense camera (with additional RAM) and single-pixel camera (having default RAM) have measured reflectance values of 27% and 3%, respectively, over a corresponding aperture area ratio of approximately 714:1. While our single-pixel camera provides excellent image resolution and fidelity, it inherently suffers from very slow raster-scanning speeds and operational bandwidth limitations. For this reason, the TeraSense camera technology is excellent for performing qualitative measurements in real time, with the caveats outlined in this paper.

Extrinsic calibration of a camera and a LiDAR is necessary to fuse information from each sensor. The real trajectory of the LiDAR is not visible on an image, therefore the accuracy of the extrinsic calibration is usually checked by evaluating residuals of constraints. In this paper, we present an improved extrinsic calibration algorithm of a camera and a 2D LiDAR using an additional dummy camera removing IR cut filter, which make it possible to observe the real trajectory of LiDAR. Some previous algorithms used the real trajectory of LiDAR for the extrinsic calibration. However, they used IR filter directly on the calibrating camera by adjusting exposure time, which can affect the result of the extrinsic calibration. We use an initial solution using the Hu algorithm which makes extrinsic calibration possible by using just one shot of data. The Hu algorithm gives a sensitive result according to pose variation between a system consisted of a camera and a LiDAR and a calibration structure, which is verified using the real trajectory of LiDAR. We cope with this problem by refining the initial solution through nonlinear minimization in a 3D space using the real trajectory of LiDAR. Experimental results show that the proposed algorithm gives an improved solution.

The rigid-body transformation between a LiDAR and monocular camera is required for sensor fusion tasks, such as SLAM. While determining such a transformation is not considered glamorous in any sense of the word, it is nonetheless crucial for many modern autonomous systems. Indeed, an error of a few degrees in rotation or a few percent in translation can lead to 20 cm reprojection errors at a distance of 5 m when overlaying a LiDAR image on a camera image. The biggest impediments to determining the transformation accurately are the relative sparsity of LiDAR point clouds and systematic errors in their distance measurements. This paper proposes (1) the use of targets of known dimension and geometry to ameliorate target pose estimation in face of the quantization and systematic errors inherent in a LiDAR image of a target, (2) a fitting method for the LiDAR to monocular camera transformation that avoids the tedious task of target edge extraction from the point cloud, and (3) a “cross-validation study” based on projection of the 3D LiDAR target vertices to the corresponding corners in the camera image. The end result is a 50% reduction in projection error and a 70% reduction in its variance with respect to baseline.

This paper presents a visible light-communication-based vehicle-to-vehicle tracking system using a new positioning algorithm and modified version of the Kalman filter. In this system, LED head and tail lamps on the vehicles are used to transmit the positioning signals to other vehicles. Two CMOS dashboard cameras on each vehicle are used to receive these signals. From the geometric relationship between two cameras and the images of LEDs captured by these two cameras, the instantaneous position of the target vehicle can be determined, given that at least one LED of the target vehicle is in the view frame of the two cameras. The discrete positioning result always contains unavoidable errors, which consist of systematic errors caused by the CMOS rolling shutter artifact and the weak spatial separability of the sensor, and other random errors. The contribution of this paper is twofold. First, a new positioning algorithm with two compensation mechanisms is proposed to eliminate systematic errors. Second, a modified Kalman filter is proposed to filter out random errors to achieve a smooth and accurate tracking result for the vehicle position. The performance of the system is verified through simulations.

The last two decades have witnessed significant advances in the field of indoor positioning systems, which have led to technologies with high location precision. However, there still lacks robust meter-level indoor positioning approaches which only rely on in-building communication infrastructures and smartphones. We present STARLIT, a system that enables a single LED beacon to localize smartphones to within sub-meter. As the smartphone camera contains millions of pixels, we create a virtual sensor array with the camera to measure the received signal strength (RSS) of the LED beacon. Different from the existing camera-based approaches, which need to capture images of the LED within a short light-to-camera distance, we utilize the reflection light from the floor. By exploiting the rolling shutter mechanism in the smartphone cameras, we propose a solution to separate the signal layer from the image background and noise. Given the measured RSSs, we establish an equation set with the Lambertian model and the camera projection model to solve the location of the smartphone. We have implemented STARLIT and evaluated its performance in an office room. Our experiments demonstrate that the STARLIT can achieve a median error of 23cm and an 80-percentile error of 55cm.

Scientific advances build on reproducible researches which need publicly available benchmark data sets. The computer vision and speech recognition communities have led the way in establishing benchmark data sets. There are much less data sets available in mobile computing, especially for rich locomotion and transportation analytics. This paper presents a highly versatile and precisely annotated large-scale data set of smartphone sensor data for multimodal locomotion and transportation analytics of mobile users. The data set comprises seven months of measurements, collected from all sensors of four smartphones carried at typical body locations, including the images of a body-worn camera, while three participants used eight different modes of transportation in the south-east of the U.K., including in London. In total, 28 context labels were annotated, including transportation mode, participant's posture, inside/outside location, road conditions, traffic conditions, presence in tunnels, social interactions, and having meals. The total amount of collected data exceed 950 GB of sensor data, which corresponds to 2812 h of labeled data and 17 562 km of traveled distance. We present how we set up the data collection, including the equipment used and the experimental protocol. We discuss the data set, including the data curation process, the analysis of the annotations, and of the sensor data. We discuss the challenges encountered and present the lessons learned and some of the best practices we developed to ensure high quality data collection and annotation. We discuss the potential applications which can be developed using this large-scale data set. In particular, we present how a machine-learning system can use this data set to automatically recognize modes of transportations. Many other research questions related to transportation analytics, activity recognition, radio signal propagation and mobility modeling can be addressed through this data set. The full data set is being made available to the community, and a thorough preview is already published.