Search

The rapid development of autonomous driving in recent years presents many challenges for scene understanding. As an essential step towards scene understanding, semantic segmentation has received increased attention in the past few years. Although deep learning based approaches have achieved great success in improving the segmentation accuracy, most of them suffer from an inefficiency problem and can hardly be applied to real-time applications. In this paper, we analyze the computational cost of Convolutional Neural Network (CNN) and find that the inefficiency of CNNs is mainly caused by their wide structure rather than deep structure. In addition, the success of pruning based model compression methods proves that there are many redundant channels in CNNs. Thus, we design a narrow while deep backbone network to improve the efficiency of semantic segmentation. By casting our network to fully convolutional network (FCN32) segmentation architecture, the basic structure of most segmentation methods, we achieve 61.5% mIoU on Cityscapes validation dataset with only 4.2G floating-point operations (FLOPs) on $1024\times 2048$ inputs, which already outperforms one of the earliest real-time deep learning based segmentation methods: ENet (58.3% mIoU, 3.8G FLOPs on $640\times 360$ inputs). By further refining the output resolution of our network to the 1/8 of the input resolution with a simple encoder-decoder structure, we achieve 65.3% mIoU on Cityscapes test set with 14.0G FLOPs and 39.9 frames per second (FPS) on Titan X card. We have made our model publicly available at https://github.com/zgyang-hnu/NDNet .

Various reports have shown that the rate of road traffic accidents has increased due to reduced driver vigilance. Therefore, an accurate estimation of the driver’s alertness status plays an important part. To estimate vigilance, we adopt a novel strategy that is a deep autoencoder with subnetwork nodes (DAE SN ). The proposed network model is designed not only for sparse representation but also for dimension reduction. Some hidden layers are not calculated by randomly acquired, but by replacement technologies. Unlike the traditional electrooculogram (EOG) signals, the forehead EOG (EOG F ) signals are collected through forehead electrodes that do not have to surround the eyes, which has a convenient and effective practical application. The root-mean-square error (RMSE) and correlation coefficient (COR) while separately using three EOG F features improved to 0.11/0.79, 0.10/0.83, and 0.11/0.80, respectively. Implemented in an experimental environment, percentage of eye closure over time is calculated in real time through SMI eye-tracking-glasses, up to 120 frames/s. In addition, the time to extract features from the raw signal and display the prediction is only 34 ms, that is the level of the driver’s fatigue can be detected quickly. The experimental study shows that the proposed model for vigilance analysis has better robustness and learning capability.

Purpose This paper aims to design a novel feature descriptor to improve the performance of feature matching in challenge scenes, such as low texture and wide-baseline scenes. Common descriptors are not suitable for low texture scenes and other challenging scenes mainly owing to encoding only one kind of features. The proposed feature descriptor considers multiple features and their locations, which is more expressive. Design/methodology/approach A graph neural network–based descriptors enhancement algorithm for feature matching is proposed. In this paper, point and line features are the primary concerns. In the graph, commonly used descriptors for points and lines constitute the nodes and the edges are determined by the geometric relationship between points and lines. After the graph convolution designed for incomplete join graph, enhanced descriptors are obtained. Findings Experiments are carried out in indoor, outdoor and low texture scenes. The experiments investigate the real-time performance, rotation invariance, scale invariance, viewpoint invariance and noise sensitivity of the descriptors in three types of scenes. The results show that the enhanced descriptors are robust to scene changes and can be used in wide-baseline matching. Originality/value A graph structure is designed to represent multiple features in an image. In the process of building graph structure, the geometric relation between multiple features is used to establish the edges. Furthermore, a novel hybrid descriptor for points and lines is obtained using graph convolutional neural network. This enhanced descriptor has the advantages of both point features and line features in feature matching.

Although deep learning has achieved great success in many computer vision tasks, its performance relies on the availability of large datasets with densely annotated samples. Such datasets are difficult and expensive to obtain. In this article, we focus on the problem of learning representation from unlabeled data for semantic segmentation. Inspired by two patch-based methods, we develop a novel self-supervised learning framework by formulating the jigsaw puzzle problem as a patch-wise classification problem and solving it with a fully convolutional network. By learning to solve a jigsaw puzzle comprising 25 patches and transferring the learned features to semantic segmentation task, we achieve a 5.8% point improvement on the Cityscapes dataset over the baseline model initialized from random values. It is noted that we use only about 1/6 training images of Cityscapes in our experiment, which is designed to imitate the real cases where fully annotated images are usually limited to a small number. We also show that our self-supervised learning method can be applied to different datasets and models. In particular, we achieved competitive performance with the state-of-the-art methods on the PASCAL VOC2012 dataset using significantly fewer time costs on pretraining.

Semantic segmentation is a key step in scene understanding for autonomous driving. Although deep learning has significantly improved the segmentation accuracy, current high-quality models such as PSPNet and DeepLabV3 are inefficient given their complex architectures and reliance on multi-scale inputs. Thus, it is difficult to apply them to real-time or practical applications. On the other hand, existing real-time methods cannot yet produce satisfactory results on small objects such as traffic lights, which are imperative to safe autonomous driving. In this paper, we improve the performance of real-time semantic segmentation from two perspectives, methodology and data. Specifically, we propose a real-time segmentation model coined Narrow Deep Network (NDNet) and build a synthetic dataset by inserting additional small objects into the training images. The proposed method achieves 65.7% mean intersection over union (mIoU) on the Cityscapes test set with only 8.4G floating-point operations (FLOPs) on $1024\times 2048$ inputs. Furthermore, by re-training the existing PSPNet and DeepLabV3 models on our synthetic dataset, we obtained an average 2% mIoU improvement on small objects.

High-performance 6D pose estimation and dense 3D mapping with an RGB-D camera has recently attracted substantial research attention since this type of camera can simultaneously capture RGB and depth information. However, there is an unresolved problem in estimating pose and generating highly accurate 3D maps from challenging indoor scenes. This paper presents a real-time simultaneous localization and mapping (SLAM) system based on the RGB-D camera for indoor mobile robots. Our contributions are fourfold. First, we propose a complete high-accuracy SLAM system based on a combination of information from points and lines, which differs from most solutions that rely on only point features. Second, we propose a novel pose solver to handle point and line correspondences, in which a line-based inliers refinement (LBIR) algorithm is proposed to remove outliers. Third, we construct a unified optimization model to concurrently minimize point and line reprojection errors, and extend it to the bundle adjustment (BA) method. Fourth, extensive experiments demonstrate the robustness, accuracy, and real-time performance of the proposed system on public TUM datasets and real world scenes. The empirical results show that the proposed system achieves a comparable or better performance than state-of-the-art methods. Notably, our system can operate in nearly texture-less scenes, while other methods are prone to failure.

Rigid robot pose estimation with an RGB-D camera has attracted substantial research attention recently for the reason that the RGB-D camera can capture RGB and depth information simultaneously. Despite the huge progress that has been made, there are still some unresolved issues like the pose estimation in texture-less or structure-less scenes. Aiming at this problem, this paper presents a robust real-time pose estimation method with an RGB-D camera for texture-less and structure-less scenes. Our contributions are threefold. First, we present an improved ORB algorithm for extracting reliable inliers, in which adaptive threshold setting method of FAST corners decision is proposed for extracting sufficient keypoints. In addition, an effective inliers refinement method, based on motion smoothness consistency constraint, is introduced for obtaining fine inliers. Second, based on the characteristics of RGB-D camera, this paper proposes a novel hybrid reprojection errors optimization model (HREOM) to estimate pose by concurrently minimizing 3D-3D and 3D-2D reprojection errors. Third, we carry out comprehensive experiments on TUM public datasets to demonstrate the robustness, accuracy, and real-time of the proposed system. The quantitative evaluations show that our system can extract sufficient inliers in those extreme scenes. Furthermore, our method performs as good as or better than other state-of-the-art solutions. Notably, our system can operate in the texture-less and structure-less environment, while other methods are prone to failure.

In unknown environments, multiple-robot cooperation for target searching is a hot and difficult issue. Swarm intelligence algorithms, such as Particle Swarm Optimization (PSO) and Fruit Fly Optimization Algorithm (FOA), are widely used. To overcome local optima and enhance swarm diversity, this paper presents a novel multi-swarm hybrid FOA-PSO (MFPSO) algorithm for robot target searching. The main contributions of the proposed method are as follows. (1) The improved FOA (IFOA) provides a better value for the improved PSO (IPSO) to find the next optimal robot position value. (2) Multi-swarm strategy is introduced to enhance the diversity and achieve an effective exploration to avoid premature convergence and falling into local optima. (3) An escape mechanism named MSCM (Multi-Scale Cooperative Mutation) is used to address the limitation of local optima and enhance the escape ability for obstacle avoidance. All of the aspects mentioned above lead robots to the target without falling into local optima and allow the search mission to be performed more quickly. Several experiments in four parts are performed to verify the better performance of MFPSO. The experimental results show that the performance of MFPSO is much more significant than that of other current approaches.

In recent years, semantic segmentation methods based on fully convolutional networks (FCNs) have achieved great success. However, these methods tend to produce inconsistency and isolated class labels, mainly because the end-to-end mapping of FCN essentially treats each pixel independently. As a post-processing approach, the conditional random field (CRF) has been widely used to alleviate this problem. However, the inference of CRF is usually very time-consuming in computation. To solve this problem, we present a new method, called locally shared features (LSFs), to model local dependence between pixels. The LSF encourages adjacent pixels to have similar features by making them share certain properties with each other. This is achieved by concatenating features around a pixel, including the pixel itself. Our experimental results indicate—the LSF approach delivers comparable or better performance than the CRF method with respect to the accuracy and local smoothness in segmentation output, while obtaining a significant gain in computational efficiency.

WiFi-Fingerprint is extensively utilized for indoor localization with the advent of the high-density wireless networks deployment and research on ubiquitous intelligence. Nevertheless, establishing an elaborate radio map for localization is a highly time-consuming task. Aiming to alleviate this problem and enhance WiFi-based indoor localization accuracy, this paper has done the following contributions. First, we present Gaussian process regression models to predict the spatial distribution of signal strength in the uncalibrated domain with limited known labeled fingerprints in reference points. As a result, deployment effort for radio mapping can be greatly reduced. Second, in order to acquire fingerprints data with higher accuracy, the compound kernels for received signal strength (RSS) prediction models are presented. Finally, the definite position is determined with the weighted Similarity K-Nearest Neighbor localization algorithm when new observation RSS is collected. The experiments show that compared with the original reference fingerprints localization system, the proposed localization system explicitly reduces the localization error. The results further demonstrate that our method can augment the fingerprints and improve the accuracy of fingerprint-based indoor localization without extra manual calibration or adding dedicated infrastructure.

Background: It is well-known that many chronic diseases are associated with unhealthy diet. Although improving diet is critical, adopting a healthy diet is difficult despite its benefits being well understood. Technology is needed that allows assessment of dietary intake accurately and easily in real-world settings so that effective intervention to manage overweight, obesity and related chronic diseases can be developed. In recent years, new wearable imaging and computational technologies have emerged. These technologies are capable of objective and passive dietary assessment with much simplified procedure than traditional questionnaires. However, a critical task is required to estimate the portion size (in this case, the food volume) from a digital image. Currently, this task is very challenging because the volumetric information in the two-dimensional images is incomplete, and the estimation involves a great deal of imagination, beyond the capacity of the traditional image processing algorithms.Method : A novel Artificial Intelligent (AI) system is proposed to mimic the thinking of dietitians who use a set of common objects as gauges (e.g., a teaspoon, a golf ball, a cup, and so on) to estimate the portion size. Specifically, our human-mimetic system "mentally" gauges the volume of food using a set of internal reference volumes that have been learned previously. At the output, our system produces a vector of probabilities of the food with respect to the internal reference volumes. The estimation is then completed by an "intelligent guess", implemented by an inner product between the probability vector and the reference volume vector.Dataset: The datasets utilized for model validation include: 1) two virtual food datasets produced by computer simulation, and 2) two real-world food datasets collected by us.Results: The average relative volumetric errors of our AI method were less than 9% on both virtual datasets, and 11.7% and 20.1% , respectively, on the two real-world food datasets.Discussion: We discuss: 1) the use of AI to estimate the "relative volume" of food in a plate, 2) the case of multiple foods in a plate, and 3) the potential of AI in advancing nutrition science.Conclusion: Our AI system is able to use the same food volume estimation strategy as the human uses.

Semantic segmentation, also called scene labeling, refers to the process of assigning a semantic label (e.g. car, people, and road) to each pixel of an image. It is an essential data processing step for robots and other unmanned systems to understand the surrounding scene. Despite decades of efforts, semantic segmentation is still a very challenging task due to large variations in natural scenes. In this paper, we provide a systematic review of recent advances in this field. In particular, three categories of methods are reviewed and compared, including those based on hand-engineered features, learned features and weakly supervised learning. In addition, we describe a number of popular datasets aiming for facilitating the development of new segmentation algorithms. In order to demonstrate the advantages and disadvantages of different semantic segmentation models, we conduct a series of comparisons between them. Deep discussions about the comparisons are also provided. Finally, this review is concluded by discussing future directions and challenges in this important field of research.

Multirobot task allocation (MRTA) has been extensively utilized on a number of occasions along with the development in robotics system. In this paper, we focus on the multi-type tasks allocation with the proposed heterogeneous self-organizing map (SOM) algorithm. To endow the robots with excellent discernibility, the concept of pheromones between different type of tasks and robots are introduced. Also, the winner neuron selection criterion is based on the proposed robot utility according to different prescriptive task mode. The weight updating mechanism guides the robots to the specified task points. To validate the universality and practicality of the algorithm, the experiments were conducted in three different cases and a dynamic environment, the results show that the proposed heterogeneous SOM can adjust the mechanism of the multi-type MRTA to different situations, performing outstanding scalability.

Multirobot target searching in unknown environments is a currently trending topic of discussion. In this paper, an improved bat algorithm (BA) for multirobot target searching in unknown environments, named adaptive robotic bat algorithm (ARBA), is proposed; it acts as the controlling mechanism for robots. The obstacle avoidance problem is considered in the proposed ARBA. The adaptive inertial weight strategy helps ARBA improve its diversity and provides an effective mechanism for escaping from local optima. In addition, the Doppler effect is introduced to improve ARBA; the effect can be adaptively compensated when the robot moves and helps robots avoid premature convergence. Moreover, the location of the target in an unknown environment is unknown, and a multi-swarm strategy is introduced into the ARBA to improve the diversity and expand the search space of robots so that robots can find the location of the target as well as the target itself faster than the existing algorithms. Experiments were conducted in three aspects to verify the effectiveness and efficiency of ARBA. We compared ARBA with the other algorithms in this field; the experimental results demonstrate that ARBA exhibits better performance in multirobot target searching and can be applied to multirobot intelligent systems.

Camera calibration plays a critical role in 3D computer vision tasks. The most commonly used calibration method utilizes a planar checkerboard and can be done nearly fully automatically. However, it requires the user to move either the camera or the checkerboard during the capture step. This manual operation is time consuming and makes the calibration results unstable. In order to solve the above problems caused by manual operation, this paper presents a full-automatic camera calibration method using a virtual pattern instead of a physical one. The virtual pattern is actively transformed and displayed on a screen so that the control points of the pattern can be uniformly observed in the camera view. The proposed method estimates the camera parameters from point correspondences between 2D image points and the virtual pattern. The camera and the screen are fixed during the whole process; therefore, the proposed method does not require any manual operations. Performance of the proposed method is evaluated through experiments on both synthetic and real data. Experimental results show that the proposed method can achieve stable results and its accuracy is comparable to the standard method by Zhang.

The appearance model is very effective in tracking multiple persons. The main difficulty in tracking persons is to represent appearance reliably and effectively, especially in the presence of occlusions. In this paper, an effective Attributed Relational Graph (ARG) based tracking algorithm is presented to track multiple persons even under occlusions. The appearance of each person is expressed by an ARG model which not only combines color feature with spatial information but also illustrates the relations among body parts. The similarity of ARG models is computed to build a matching matrix in consecutive frames. Four tracking situations are determined according to the matching matrix. In addition, to track persons under occlusions, probabilistic relaxation labeling in the ARG models of body parts is deduced to label occluded persons optimally. Experimental validation of the proposed tracking method is verified and presented on indoor and outdoor sequences.

Inspired by the human 3D visual perception system, we present an obstacle detection and classification method based on the use of Time-of-Flight (ToF) cameras for robotic navigation in unstructured environments. The ToF camera provides 3D sensing by capturing an image along with per-pixel 3D space information. Based on this valuable feature and human knowledge of navigation, the proposed method first removes irrelevant regions which do not affect robot's movement from the scene. In the second step, regions of interest are detected and clustered as possible obstacles using both 3D information and intensity image obtained by the ToF camera. Consequently, a multiple relevance vector machine (RVM) classifier is designed to classify obstacles into four possible classes based on the terrain traversability and geometrical features of the obstacles. Finally, experimental results in various unstructured environments are presented to verify the robustness and performance of the proposed approach. We have found that, compared with the existing obstacle recognition methods, the new approach is more accurate and efficient.

Time of Flight (ToF) cameras have become a competitive alternative to traditional distance sensing techniques like laser or stereo vision as it can deliver grayscale images and 3D information simultaneously at high frame-rates. However the low resolution of ToF camera limits the applications for accurate segmentation or classification. This paper presents a fast and robust solution to combine the 3D information of ToF camera and high resolution color image. Firstly, we setup a 2D/3D stereo camera with fixed spatial relation and similar visual field. Based on characteristic of ToF camera and the principles of stereo vision system, the 3D information of ToF camera is registered with high resolution color image by matching the high resolution color image with grayscale image of ToF camera. This method has very low computation cost while the matching accuracy is only determined by physical characteristics or parameters of the 2D/3D stereo camera system without extra computation error. Experiments results have demonstrated the feasibility, efficiency and accuracy of proposed algorithms.

A visual odometry optimization algorithm based on Kinect camera is proposed. The optimization algorithm uses Oriented FAST and Rotated BRIEF (ORB) algorithm to extract and matching feature point pairs, culls the mismatching feature point pairs through the bidirectional consistency check, the constraints based on the color space information and the Random Sample Consensus (RANSAC) algorithm, and then indexes the corresponding three-dimensional (3D) information of the eventually matching feature point pairs. It uses the least squares method to calculate the rotation and translation matrix so as to complete motion estimation between the two images, and then implements the local optimization of the motion estimation using the Sparse Bundle Adjustment (SBA) algorithm, eventually it could draw a global visual odometry simulated trajectory according to the optimized motion estimation. The experiment result shows that this method has high accuracy and robustness.

Accessible frontier is an important factor for mobile robot autonomous exploration. This paper presents a fast and robust frontier line segment extracting method based on fuzzy c-means clustering algorithm for robot exploration. Firstly, the proposed method divides robot's local occupancy map into sub-regions with same size. In the next step, this paper analyzes the characteristic of robot exploration frontier with occupancy grid map, and the optimal number of FCM cluster center in each sub-region is defined. Consequently, line segments corresponding to exploration frontiers based on fuzzy c-mean algorithm are calculated in sub-region level to alleviate the extensive computation. Following those steps, line segments merging, line endpoints extending and line excluding are conducted to get more accurate frontier segment parameters in global level. In the end, the effectiveness of proposed method is verified by experiments results in lab environment.

This paper presents a novel method for real-time obstacle detection and recognition in natural terrain for a field mobile robot using a image information, geometric information and support vector machine(SVM). Firstly, the scene is divided into two distinct regions: interest regions and uninterested regions. Then detected obstacle points are clustered into objects on the basis of their geometric information, i.e., depth and horizontal information, connectivity. The key obstacle characteristics are identified as width, height, their ratio, the ratio of area and depth. In the paper, the SVM method is used to classifying the objects into four classes. In order to determine the slope value, a SVM slope estimation approximation model was also proposed. Experimental results are presented to demonstrate the capability of the proposed approach for recognition of different obstacle in natural terrain.

In order to correctly sense incline terrain, its geometrical calculated model is analyzed. Based on the change trend of distance between the mobile robot and slope, their relative position can be determined. Then a novel method which takes the use of the powerful nonlinearity approach capability of RBF network is introduced to estimate the slope of the terrain with respect to the robot's current angular tilt. The experimental results show that the proposed approach can be used for mobile robot incline terrain perception and has the merits of easy, trustiness and robustness.

This paper presents an improved neural network model interpretating sonar readings to build occupancy grids of mobile robot. The proposed model interprets sensor readings in the context of their space neighbors and relevant successive history readings simultaneously. Consequently the presented method can greatly weaken the effects by multiple reflections or specular reflection. The output of the neural network is the probability vector of three possible status(empty, occupancy, uncertainty) for the cell. As for sensor readings integration, three probabilities of cell's status are updated by the Bayesian update formula respectively, and the final status of cell is defined by Max-Min principle.Experiments performed in lab environment has shown occupancy map built by proposed approach is more consistent, accurate and robust than traditional method while it still could be conducted in real time.

Camera calibration is an important prerequisite for retrieving 3D information of scene. Considered with peculiar configuration of stereo camera, some reasonable assumptions are introduced to obtain simplified stereo camera model which assumes that principle points locating in the image center, camera axes perpendicular with camera plane, focal length of stereo camera remaining constant during movement. Based on those constrains, a self-calibration method for moving stereo camera is represented in this paper, which determines camera parameters through calculating fundamental matrix corresponding to successive motion sequence. Experimental results on dynamic stereo camera shows that this self-calibration approach has good real-time performance and high resolution when vergence angel and tilt angel is confined to certain range during movement.

Sharing mechanisms with biological vision systems, real-time active stereovision system built from component parts was developed for mobile robot autonomous navigation. Firstly, four degrees of freedom of stereo platform with optimal baseline 34 cm, was designed for accomplishing visual tasks independently like pan, tilt and vergence of stereo camera. Secondly, step-motor drive scheme combined with closed loop servo controller was selected for multi-freedom controller in view of motor resolution requirement and system cost. Then, camera parameters controller was highlighted to enhance system adaptability by adjusting aperture, focus and zoom. Finally, stereo camera synchronization was guaranteed with the adoption two independent image-grabbers combined external synchronous signal. The strategies adopted for the design of the system has been proven to be very effective with applications on the pioneer 2-DXE mobile robot.

Aiming at the selection and refreshment of background, moving objects detecting, sequence matching, research on multi-moving targets detecting and tracking based on single fixed camera was performed. Firstly, a recursive filter in temporal domain to form the reference image for adaptive background updating was highlighted. Then foreground was separated from background based on difference between reference image and current image. Secondly, moving regions in foreground were extracted through analysis on connected binary foreground image. Detected binary regions were then grouped into moving targets based on fuzzy clustering analysis. Thirdly, an extended Kalman filter and C-constant velocity Kalman tracking algorithm were presented for the matching and tracking multi-moving targets represented by region. Finally, experiments on outdoors video streams have demonstrated the significant performance of proposed strategies.