Search

In this paper, we present an accurate approach to estimate vehicles' pose and shape from off-board multiview images. The images are taken by monocular cameras and have small overlaps. We utilize state-of-the-art convolutional neural networks (CNNs) to extract vehicles' semantic keypoints and introduce a Cross Projection Optimization (CPO) method to estimate the 3D pose. During the iterative CPO process, an adaptive shape adjustment method named Hierarchical Wireframe Constraint (HWC) is implemented to estimate the shape. Our approach is evaluated under both simulated and real-world scenes for performance verification. It's shown that our algorithm outperforms other existing monocular and stereo methods for vehicles' pose and shape estimation. This approach provides a new and robust solution for off-board visual vehicle localization and tracking, which can be applied to massive surveillance camera networks for intelligent transportation., Comment: 8 pages, 8 figures, published in ROBIO 2018

In the field of autonomous driving, abundant researches have been carried out in development of advanced Lidar or vision algorithms for mapping and localization, and some efforts in the field of sensor fusion have been made to combine the advantages of multiple sensor types. However, most works focused on the 2D map construction, which had its own limitation with complex environments. In this paper, a new algorithm based on the fusion of Lidar and Inertial Measurement Unit (IMU) data is developed to construct a 2.5D map. We integrate the height information transformed from IMU sensor with laser scan results to construct a 2.5D occupied grid map. The proposed algorithm is convenient to implement on real-world mobile vehicles, and it is fast and robust. The effectiveness of our algorithm is validated by both simulation and practical experiment.

A vision-based perception system for understanding the outdoor environment has been proposed. This system utilizes and applies the useful information from the binocular cameras of a legged robot for landform — terrain — ground environmental perception. The ground texture recognition can provide information about the material of the ground then a legged robot can choose an appropriate walking pose accordingly. With the terrain perception and obstacle detection, combined with semantic segmentation of the environment, a legged robot can move towards the target with intelligent obstacle avoidance strategy. Based on this vision-based perception system, we propose a vision-based perception framework for outdoor navigation tasks applicable to legged robots. In this framework, environmental modeling and situation assessment will be firstly carried out combined with multi-mode sensor fusion, and then footsteps and local path planning can be generated.

Monocular cameras are not generally considered as a 3D measurement tool in the field of intelligent transportations. This belief limits the application of monocular cameras in many applications such as guiding robots or vehicles for autonomous driving. In this paper, we demonstrate that with the reference of a license plate, a monocular camera can be used as a tool to measure 3 dimensional parameters (width, height and length) of a passenger vehicle down to 10 cm accuracy. The proposed measurement method does not reply on any training process, but is designed as a fast and direct approach utilizing a series of fundamental computer vision techniques such as ROI extraction, edge/line detection, space coordinates transformations, etc. A series of validation tests were conducted and the results demonstrated the robustness of this measurement method and implies the possibility to broaden the application scope of monocular cameras in the field of autonomous driving and intelligent transportation systems.

Localization and navigation are fundamental issues to autonomous mobile robotics. In the case of the environmental map has been built, the traditional two-dimensional (2D) lidar localization and navigation system can't be matched to the initial position of the robot in dynamic environment and will be unreliable when kidnapping occurs. Moreover, it relies on high-cost lidar for high accuracy and long range. In view of this, the paper presents a low cost navigation system based on a low cost lidar and a cheap webcam. In this approach, 2D-codes are attached to the ceiling, to provide reference points to aid the indoor robot localization. The mobile robot is equipped with webcam pointing to the ceiling to identify 2D-codes. On the other hand, a low-cost 2D laser scanner is applied to build a map in unknown environment and detect obstacles. Adaptive Monte Carlo Localization (AMCL) is implements for lidar positioning, A* and Dynamic Window Approach (DWA) are applied in path planning based on a 2D grid map. The error analysis and experiments has validated the proposed method.

Mapping, localization, and navigation are functions necessitated by a diverse range of household service robot solutions. This chapter presents a wide variety of mapping, localization, and navigation techniques that suit a wide range of robot platforms and applications. The problem of mapping and localization has been approached in many ways and we review successful results attained by traditional radio tracking, ultra wideband sensing, RFID communication, North-Star infrared sensing, two-dimensional special pattern detection, laser-range surveying, various camera system sensing, time-of-flight camera ranging, and finally low-cost contact and near-field sensing. The problem of navigation, path planning, and obstacle avoidance requires not only sensing described above but adequate robot responses in a dynamic situation. There are many approaches to successful obstacle avoidance and our review of the literature covers the use of structured light as well as linear and planar sensors, along with visual processing techniques such as optic flow, ground plane segmentation, and feature-based obstacle avoidance, in addition to the optimization of path planning, robot interaction, and behavior.

Robotic grasping and manipulation comprise three subtasks, i.e., target detection, planning, and control. The robot first needs to detect the target. Here “detect” means to find (or recognize) the target and locate it. The target could be the object to be manipulated (e.g., electrical plugs) or the destination of the object (electrical sockets). Then the robot approaches the target in an optimized trajectory. In the process of approaching and grasping, the robot should be controlled to perform the task correctly and smoothly. In this chapter, we review the approaches to accomplish these three subtasks, i.e., target detection, planning, and control.

This paper introduces a real-time video surveillance system which can track people and detect human abnormal behaviors. In the blob detection part, an optical flow algorithm for crowd environment is studied experimentally and a comparison study with respect to traditional subtraction approach is carried out. The different approaches in segmentation and tracking enable the system to track persons when they change movement unpredictably in occlusion. We developed two methods for the human abnormal behavior analysis. The first one employs Principal Component Analysis for feature selection and Support Vector Machine for classification of human behaviors. The proposed feature selection method is based on the border information of four consecutive blobs. The second approach computes optical flow to obtain the velocity of each pixel for determining whether a human behavior is normal or not. Both algorithms are successfully developed in crowded environments to detect the following human abnormal behaviors: (1) Running people in a crowded environment; (2) falling down movement while most are walking or standing; (3) a person carrying an abnormal bar in a square; (4) a person waving hand in the crowd. Experimental results demonstrate these two methods are robust in detecting human abnormal behaviors.

This paper presents a novel shape representation approach, Finite Element Skeleton (FES), based on studies of planar shape analysis from the perspective of Finite Element Method (FEM). First, we derive the shape contour of an object in image with edge detection approach. Then, the approach builds connections of all pixels on the shape contour with the inner center point of the shape, and models these virtual connections as bendable beam elements. Finally, we obtain a multi-degree-of-freedom (MDOF) model of the original shape. By this model, we can calculate its natural frequency, which is one of mechanical properties that directly related to the geometric shape, and employed it as the shape representation. FES shape descriptor possesses the natural properties on translation and rotation invariant. Especially, FES descriptor has the feature of articulation insensitivity which is the most important property of a shape descriptor to ensure a good performance of the intra-class similarity measurement. Experimental results validated that the proposed FES descriptor is capable of identifying shape for object recognition.

The human behavior, such as gait, is controlled by the brain and other peripheral neural system, and thus is significantly affected by the internal emotion. It is important to carry out the affective sensation of abnormal internal emotion in a non-invasive and non-privacy-infringed way. This paper presents an experiment method and platform to sense the abnormal affective status through gaits. We used a wearable intelligent shoe system for affective sensing, designed an experiment for the human subject to have the normal gait and the gait with emotional disorder, and finally classified the gaits through machine learning.

In this paper ceiling affixed ARToolKitPlus 2D code artificial landmarks are evaluated for purposes of robot localization and navigation. Ceiling affixed codes rarely come in contact with people, equipment or robots, and for this reason they are more likely to stay detectable over a longer period of time. Multi threshold averaging, light gradient compensation and neighbourhood search techniques further enhanced AR-ToolKitPlus performance. Multi threshold averaging collects positioning results at each gray scale threshold level. After all of the threshold levels are analyzed, the related results are averaged into a final result. Light gradient compensation eliminates effects of uneven lighting in the neighbourhood of a 2D code. Neighbourhood search for a 2D code requires fewer computational resources than a global search. Further repeatability improvements are achieved by means of averaging localizations. Localization performance is evaluated at varying distances of the 2D code from the camera. Experimental results show substantial improvements in repeatability and reliability over the baseline ARToolKitPlus performance. Improved performance will allow for realtime 2D code based localization for indoor robot navigation.

Hand posture is a natural and effective human robot interaction way. In this paper, an user-independent hand posture recognition system using depth and color images captured from an RGB-D camera is presented. To recognize hand posture against complicated background conditions, we propose a novel method for automatic and accurate hand posture segmentation which detects the hand with Chamfer matching, tracks the hand with Kalman filter and segments the hand with region growing algorithm only in the depth space. A new hand posture descriptor invariant to scale, shift and in-plane rotation is constructed with the combination of local contour Fourier descriptor and global Bag-of-Features (BoF) descriptor based on Scale Invariance Feature Transform (SIFT). The sparse representation-based classification (SRC) is applied to perform the hand posture recognition task in the system. Experiments with a self-built large scale hand posture database collected online show the robustness and effectiveness of the proposed system.

This paper1 presents a novel shape representation approach, Finite Element Contour (FEC), based on studies of shape analysis from the perspective of Finite Element Method (FEM). We assume that an edge of a contour can be modeled as a bendable beam element. Linking finite number of beam elements end to end along the contour, we obtain a closed-loop Finite Element Contour model as an approximate physical model of the original shape. By this model, we can calculate its natural frequency, which is one of mechanical properties that directly related to the geometric shape, and employed it as the shape representation. FEC shape feature possesses translation and rotation invariant properties naturally. We also realized scale and unique affine normalization in few simple steps based on intrinsic physical properties of shape from FEM viewpoint. Experimental results validated that the proposed FEC feature is capable of identifying shape in object recognition task. It also can describe shape deformation. In the well-known MPEG 7 shape retrieval task, the enhanced FEC approach obtains Bullseye score 87.11%.

This paper presents a novel shape representation approach based on Finite Element Region (FER). We assume an image blob can be treated as a mesh model that consists of a finite number of deformable quadrilateral pixel elements. These elements connecting side by side form an approximate physical model of the surface of real object. By employing Finite Element Method (FEM), the model's natural frequency can be derived, which is one of mechanical properties that only related to geometric shape, as the shape representation. Experimental results validated the proposed FER descriptor possessing good performance on shape representation. It also has translation, rotation invariant properties naturally, and scale and affine normalization can be realized in few steps. Experiments also demonstrated that the FER descriptor is sensitive to shape deformation. It can precisely describe subtle changing processes between different gestures. This property allows us to establish a conceptual graph model of hand gestures changing to improve the performance of ambiguous gesture recognition.

As the most critical function of mobile robots, navigation substantially relies on self-localization. Among all the approaches, vision based localization is widely adopted. Unlike the complex algorithms developed before, we present a simple real-time visual localization approach which regards the circular lamps on the ceiling as the landmarks. Two sets of point correspondences are applied to solving Perspective-n-Point (PnP) problem. One set contains the centers of the ellipses in image and the centers of the circles in 3D. The other set contains the tangent points on the circles and ellipses. At least two lamps are required in the field of view to avoid ambiguities. The experimental results show that no blind zone exists since the required number of lamps is small. The position and orientation errors are small enough for indoor navigation. This approach is validated by experiments for real-time application.

The internet and computer are more popular in the world, and the keyboard is the important input for the computer. However, in our society, many disable people cannot type by using keyboard. In order to assist them, we developed a shoe platform which senses the foot motion and use it as the input for the computer. In this system, we used force sensing resistors (FSRs) and a 3-axis accelerometer to detect foot movements. Based on these sensors, we analyzed human foot motions in different positions, and selected those with more comfortable motions when people sitting in front of the computer with a healthy attitude. To reduce the computational cost and power consumption, and enhance the real-time performance, we use the principle component analysis (PCA) for sensor reduction. Through PCA, we selected some important FSRs in this system. Also we compared two classification methods, chose two layer back propagation(BP) networks, to detect the foot movements for typing.

With the acquisition of an image, the first step is to distinguish objects of interest from the background. In surveillance applications, those objects of interest are usually humans. Their various shapes and different motions, including walking, jumping, bending down, and so forth, represent significant challenges in the extraction of foreground pixels from the image.

In many cases, we cannot obtain the background image easily, and if the occlusion is serious, then it is difficult to segment masses of blobs into single blobs.We therefore use optical flow to detect abnormal behavior in crowded environments [57], [58].

After foreground extraction, if nonsignificant merging occurs among people, then segmentation is necessary, and the merged blobs can be split into separate ones equal in number to the number of people in the merged blob. Thereafter, all of the blobs can be indexed and tracked for motion and behavior analysis.

In the previous chapter, we presented behavior analysis, which is based on dynamic information, including background extraction based on a series of frames, blob detection based on two consecutive frames, and so forth. However, for intelligent surveillance, we can also explore static information, i.e., facial images. Facial detection is straightforward as it does not depend on whether or not background information is extractable. Hence, it is useful in highly cluttered environments, with the only requirement being sufficient image resolution of the image.

In this paper, an integrated video surveillance system for robust tracking is introduced. In the blob detection part, an optical flow algorithm for crowded environment is studied experimentally and a comparison study with respect to traditional subtraction approach is carried out. In the segmentation part, different algorithms are fused to develop a hybrid algorithm for stable segmentation, and validation rules for successful segmentation are also presented preventing from false segmentation. In the tracking part, a blob's parameter, which we call color spectrum, is developed to identify different persons and track them robustly. A hybrid algorithm for tracking is also developed to combine color tracking with traditional distance tracking. The hybrid algorithms in segmentation and tracking enable the system to track persons when they change movement unpredictably in occlusion. Experimental results validate the proposed algorithm.

This paper introduces a real-time video surveillance system which detects human abnormal behaviors. We present two approaches to such a problem. The first one employs principal component analysis for feature selection and support vector machine for classification of human behaviors. The proposed feature selection method is based on the border information of four consecutive blobs. The second approach computes optical flow to obtain the velocity of each pixel for determining whether a human behavior is normal or not. Both algorithms are successfully implemented in crowded environments for detecting the human abnormal behaviors, such as (1) running people in a crowded environment, (2) bending down movement while most are walking or standing, (3) a person carrying a long bar and (4) a person waving hand in the crowd. Experimental results demonstrate the two methods proposed are robust and efficient in detecting human abnormal behaviors.