Search

Over the last decade, robotic perception algorithms have significantly benefited from the rapid advances in deep learning (DL). Indeed, a significant amount of the autonomy stack of different commercial and research platforms relies on DL for situational awareness, especially vision sensors. This work explored the potential of general-purpose DL perception algorithms, specifically detection and segmentation neural networks, for processing image-like outputs of advanced lidar sensors. Rather than processing the three-dimensional point cloud data, this is, to the best of our knowledge, the first work to focus on low-resolution images with a 360° field of view obtained with lidar sensors by encoding either depth, reflectivity, or near-infrared light in the image pixels. We showed that with adequate preprocessing, general-purpose DL models can process these images, opening the door to their usage in environmental conditions where vision sensors present inherent limitations. We provided both a qualitative and quantitative analysis of the performance of a variety of neural network architectures. We believe that using DL models built for visual cameras offers significant advantages due to their much wider availability and maturity compared to point cloud-based perception. [ABSTRACT FROM AUTHOR]

Recently, two types of common sensors, LiDAR and Camera, show significant performance on all tasks in 3D vision. LiDAR provides accurate 3D geometry structure, while camera captures more scene context and semantic information. The fusion of two different sensor becomes a fundamental and common idea to achieve better performance. To give a thorough cognition of the complementary and boosting about two kind of sensors. This paper briefly reviews the fusion and enhancement systems between both two sensors in the field of depth completion, 3D object detection, 2D\3D semantic segmentation and 3D object tracking. Meanwhile, the state of art fusion algorithms is quantitatively demonstrated, in this paper, based on the in KITTI widely-used public dataset. Furthermore, the technical challenge and the future potential of LiDAR and camera fusion are also discussed. [ABSTRACT FROM AUTHOR]

Networked detector systems can be deployed in urban environments to aid in the detection and localization of radiological and/or nuclear material. However, effectively responding to and interpreting a radiological alarm using spectroscopic data alone may be hampered by a lack of situational awareness, particularly in complex environments. This study investigates the use of Light Detection and Ranging (LiDAR) and streaming video to enable real-time object detection and tracking, and the fusion of this tracking information with radiological data for the purposes of enhanced situational awareness and increased detection sensitivity. This work presents an object detection, tracking, and novel source–object attribution analysis that is capable of operating in real time. By implementing this analysis pipeline on a custom-developed system that comprises a static 2 in. × 4 in. × 16 in. NaI(Tl) detector colocated with a 64-beam LiDAR and four monocular cameras, we demonstrate the ability to accurately correlate trajectories from tracked objects to spectroscopic gamma-ray data in real time and use physics-based models to reliably discriminate between source-carrying and nonsource-carrying objects. In this work, we describe our approach in detail and present a quantitative performance assessment that characterizes the source–object attribution capabilities of both video and LiDAR. Additionally, we demonstrate the ability to simultaneously track pedestrians and vehicles in a mock urban environment and use this tracking information to improve both detection sensitivity and situational awareness using our contextual-radiological data fusion methodology. [ABSTRACT FROM AUTHOR]

We present a flexible sensor fusion approach to retrieve scale information in monocular visual odometry (VO) through integrating range measurements from a wide variety of depth sensors spanning from small-resolution time-of-flight (ToF) cameras to 2-D and potentially 3-D LiDARs. While many algorithms exist in literature for range-enhanced monocular VO, the majority of them are tailored for a specific sensor choice, limiting the integration on generic mobile systems. Our monocular VO algorithm builds on a standard front end, where the camera tracking is performed relative to a map of triangulated landmarks. The inherent scale ambiguity and drift in monocular perception are resolved by optimizing both the camera poses as well as the landmark map with the depth information provided by the range sensor. Performances have been tested on the custom data sets created with an experimental platform comprising a stereo camera, a low-resolution ToF camera, and a 2-D LiDAR. We present a detailed overview of the extrinsic calibration procedures including an ad hoc solution applicable to very low-resolution depth sensors. The proposed system is tested on short- and long-range motions, showing that: 1) performances in each of the tested configurations are on par or better than the state-of-the-art stereo systems and 2) it is sufficient to use a reduced number of range measurements to obtain a scaled trajectory accurately. [ABSTRACT FROM AUTHOR]