Search

Cognitive scientists believe that adaptable intelligent agents like humans perform spatial reasoning tasks by learned causal mental simulation. The problem of learning these simulations is called predictive world modeling. We present the first framework for a learning open-vocabulary predictive world model (OV-PWM) from sensor observations. The model is implemented through a hierarchical variational autoencoder (HVAE) capable of predicting diverse and accurate fully observed environments from accumulated partial observations. We show that the OV-PWM can model high-dimensional embedding maps of latent compositional embeddings representing sets of overlapping semantics inferable by sufficient similarity inference. The OV-PWM simplifies the prior two-stage closed-set PWM approach to the single-stage end-to-end learning method. CARLA simulator experiments show that the OV-PWM can learn compact latent representations and generate diverse and accurate worlds with fine details like road markings, achieving 69 mIoU over six query semantics on an urban evaluation sequence. We propose the OV-PWM as a versatile continual learning paradigm for providing spatio-semantic memory and learned internal simulation capabilities to future general-purpose mobile robots. [ABSTRACT FROM AUTHOR]

Transformer-based models have gained popularity in the field of natural language processing (NLP) and are extensively utilized in computer vision tasks and multi-modal models such as GPT4. This paper presents a novel method to enhance the explainability of transformer-based image classification models. Our method aims to improve trust in classification results and empower users to gain a deeper understanding of the model for downstream tasks by providing visualizations of class-specific maps. We introduce two modules: the "Relationship Weighted Out" and the "Cut" modules. The "Relationship Weighted Out" module focuses on extracting class-specific information from intermediate layers, enabling us to highlight relevant features. Additionally, the "Cut" module performs fine-grained feature decomposition, taking into account factors such as position, texture, and color. By integrating these modules, we generate dense class-specific visual explainability maps. We validate our method with extensive qualitative and quantitative experiments on the ImageNet dataset. Furthermore, we conduct a large number of experiments on the LRN dataset, which is specifically designed for automatic driving danger alerts, to evaluate the explainability of our method in scenarios with complex backgrounds. The results demonstrate a significant improvement over previous methods. Moreover, we conduct ablation experiments to validate the effectiveness of each module. Through these experiments, we are able to confirm the respective contributions of each module, thus solidifying the overall effectiveness of our proposed approach. [ABSTRACT FROM AUTHOR]

In the field of intelligent vehicle technology, there is a high dependence on images captured under challenging conditions to develop robust perception algorithms. However, acquiring these images can be both time-consuming and dangerous. To address this issue, unpaired image-to-image translation models offer a solution by synthesizing samples of the desired domain, thus eliminating the reliance on ground truth supervision. However, the current methods predominantly focus on single projections rather than multiple solutions, not to mention controlling the direction of generation, which creates a scope for enhancement. In this study, we propose a generative adversarial network (GAN)–based model, which incorporates both a style encoder and a content encoder, specifically designed to extract relevant information from an image. Further, we employ a decoder to reconstruct an image using these encoded features, while ensuring that the generated output remains within a permissible range by applying a self-regression module to constrain the style latent space. By modifying the hyperparameters, we can generate controllable outputs with specific style codes. We evaluate the performance of our model by generating snow scenes on the Cityscapes and the EuroCity Persons datasets. The results reveal the effectiveness of our proposed methodology, thereby reinforcing the benefits of our approach in the ongoing evolution of intelligent vehicle technology. [ABSTRACT FROM AUTHOR]

Cooperative recognition has been proposed as a method for human-automation cooperation in autonomous driving. In cooperative recognition, when assigning tasks to operators (i.e., request intervention), it is necessary to consider not only driving safety and efficiency but also human capabilities. In this study, we optimize the intervention request planning by formulating the problem with a Partially Observable Markov Decision Process. We evaluated the proposed method with a simple simulation to discuss the validity of the method and obtained the implications for future development. [ABSTRACT FROM AUTHOR]

Weather variation in the distribution of image data can cause a decline in the performance of existing visual algorithms during evaluation. Adding additional samples of target domain to training data or using pre-trained image restoration methods such as de-hazing, de-raining, and de-snowing, to improve the quality of input images are two promising solutions. In this work, we propose Multiple Weather Translation GAN (MWTG), a CycleGAN-based, dual-purpose framework that simultaneously learns weather generation and its removal from image data. MWTG consists of four GANs constrained using cycle consistency that carry out domain translation tasks between hazy, rainy, snowy, and clear weather, using an asymmetric approach. To increase network capacity, we employ a spatial feature transform (SFT) layer to fuse the features extracted from the weather layer, which contains high-level domain information from the previous generators. Further, we collect an unpaired, real-world driving dataset recorded under various weather conditions called Realistic Driving Scenes under Bad Weather (RDSBW). We qualitatively and quantitatively evaluate MWTG using the RDSBW and the variation of Cityscapes that synthesize weather effects, eg., FoggyCityscape. Our experimental results suggest that MWTG can generate realistic weather in clear images and also accurately remove noise from weather images. Furthermore, the SOTA pedestrian detector ASCP is shown to achieve an impressive gain in detection precision after image restoration using the proposed MWTG method. [ABSTRACT FROM AUTHOR]

Automated Driving Systems (ADS) open up a new domain for the automotive industry and offer new possibilities for future transportation with higher efficiency and comfortable experiences. However, perception and sensing for autonomous driving under adverse weather conditions have been the problem that keeps autonomous vehicles (AVs) from going to higher autonomy for a long time. This paper assesses the influences and challenges that weather brings to ADS sensors in a systematic way, and surveys the solutions against inclement weather conditions. State-of-the-art algorithms and deep learning methods on perception enhancement with regard to each kind of weather, weather status classification, and remote sensing are thoroughly reported. Sensor fusion solutions, weather conditions coverage in currently available datasets, simulators, and experimental facilities are categorized. Additionally, potential ADS sensor candidates and developing research directions such as V2X (Vehicle to Everything) technologies are discussed. By looking into all kinds of major weather problems, and reviewing both sensor and computer science solutions in recent years, this survey points out the main moving trends of adverse weather problems in perception and sensing, i.e., advanced sensor fusion and more sophisticated machine learning techniques; and also the limitations brought by emerging 1550 nm LiDARs. In general, this work contributes a holistic overview of the obstacles and directions of perception and sensing research development in terms of adverse weather conditions. [ABSTRACT FROM AUTHOR]

Continuous point cloud data is being used more and more widely in practical applications such as mapping, localization and object detection in autonomous driving systems, but due to the huge volume of data involved, sharing and storing this data is currently expensive and difficult. One possible solution is the development of more efficient methods of compressing the data. Other researchers have proposed converting 3D point cloud data into 2D images, or using tree structures to store the data. In a previous study targeting streaming point cloud data, we proposed an MPEG-like compression method which utilizes simultaneous localization and mapping (SLAM) results to simulate LiDAR’s operating process. In this paper, instead of imitating MPEG, we propose new strategy for more efficient reference frame distribution and more natural frame prediction, and use a different algorithm to encode the residual, greatly improving the algorithm’s performance and its stability in different scenarios. We also discuss how various parameters affect compression performance. Using our proposed method, streaming point cloud data collected by LiDAR sensors can be compressed to 1/50th of its original size, with only 2 cm of Root Mean Square Error for each detected point. We evaluate our proposed method by comparing its performance with several other existing point cloud compression methods in three different driving scenarios, demonstrating that our proposed method outperforms them. [ABSTRACT FROM AUTHOR]

This paper introduces a method to extract driving behaviors from a human expert driver which are applied to an autonomous agent to reproduce proactive driving behaviors. Deep learning techniques were used to extract latent features from the collected data. Extracted features were clustered into behaviors and used to create velocity profiles allowing an autonomous driving agent could drive in a human-like manner. By using proactive driving behaviors, the agent could limit potential sources of discomfort such as jerk and uncomfortable velocities. Additionally, we proposed a method to compare trajectories where not only the geometric similarity is considered, but also velocity, acceleration and jerk. Experimental results in a simulator implemented in ROS show that the autonomous agent built with the driving behaviors was capable of driving similarly to expert human drivers. [ABSTRACT FROM AUTHOR]

Subjective risk assessment is an important technology for enhancing driving safety, because an individual adjusts his/her driving behavior according to his/her own subjective perception of risk. This study presents a novel framework for modeling personalized subjective driving risk during expressway lane changes. The objectives of this study are twofold: (i) to use ego vehicle driving signals and surrounding vehicle locations in a data-driven and explainable approach to identify the possible influential factors of subjective risk while driving and (ii) to predict the specific individual's subjective risk level just before a lane change. We propose the personalized subjective driving risk model, a combined framework that uses a random forest-based method optimized by genetic algorithms to analyze the influential risk factors, and uses a bidirectional long short term memory to predict subjective risk. The results demonstrate that our framework can extract individual differences of subjective risk factors, and that the identification of individualized risk factors leads to better modeling of personalized subjective driving risk. [ABSTRACT FROM AUTHOR]

This paper describes an implementation of a mobile robot system for autonomous navigation in outdoor concurred walkways. The task was to navigate through nonmodified pedestrian paths with people and bicycles passing by. The robot has multiple redundant sensors, which include wheel encoders, an inertial measurement unit, a differential global positioning system, and four laser scanner sensors. All the computation was done on a single laptop computer. A previously constructed map containing waypoints and landmarks for position correction is given to the robot. The robot system's perception, road extraction, and motion planning are detailed. The system was used and tested in a 1-km autonomous robot navigation challenge held in the City of Tsukuba, Japan, named “Tsukuba Challenge 2007.” The proposed approach proved to be robust for outdoor navigation in cluttered and crowded walkways, first on campus paths and then running the challenge course multiple times between trials and the challenge final. The paper reports experimental results and overall performance of the system. Finally the lessons learned are discussed. The main contribution of this work is the report of a system integration approach for autonomous outdoor navigation and its evaluation. © 2009 Wiley Periodicals, Inc. [ABSTRACT FROM AUTHOR]