Search

Simultaneous and proportional control (SPC) using surface electromyography (sEMG) signals has become a prevailing solution for the intuitive control of prosthesis and human–robot interaction. However, only time and frequency domain features are generally involved in conventional SPC algorithms, ignoring the globally spatial information across channels. In this article, we presented an instantaneous sEMG-image-based model for continuous estimation on multiple degrees of freedom wrist torques. Specifically, a convolutional neural network was used to extract features from sEMG-images, and then to estimate wrist torques. The results demonstrated that the sEMG-image-based model outperformed three conventional regression methods and a deep-learning-based method with superior performance on estimation accuracy and smoothness. Furthermore, we found a strong linear correlation between the first two principal components of extracted features and the recorded wrist torques, validating the effectiveness of the proposed approach. The outcomes open up a new perspective for SPC and potentially enhance the performance of myoelectric control in practical application.

TargefTablets can be detected easily from the background of infrared images due to their significantly discriminative thermal radiations, while visible images contain textural details with high spatial resolution which are beneficial to the enhancement of target recognition. Therefore, fused images with abundant detail information and effective target areas are desirable. In this paper, we propose an end-to-end model for infrared and visible image fusion based on detail preserving adversarial learning. It is able to overcome the limitations of the manual and complicated design of activity-level measurement and fusion rules in traditional fusion methods. Considering the specific information of infrared and visible images, we design two loss functions including the detail loss and target edge-enhancement loss to improve the quality of detail information and sharpen the edge of infrared targets under the framework of generative adversarial network. Our approach enables the fused image to simultaneously retain the thermal radiation with sharpening infrared target boundaries in the infrared image and the abundant textural details in the visible image. Experiments conducted on publicly available datasets demonstrate the superiority of our strategy over the state-of-the-art methods in both objective metrics and visual impressions. In particular, our results look like enhanced infrared images with clearly highlighted and edge-sharpened targets as well as abundant detail information.

We study the problem of 3D object detection from RGB-D images so as to achieve localization (i.e., producing a bounding box around the object) and classification (i.e., determining the object category) simultaneously. Its challenges arise from high intra-class variability, illumination change, background clutter and occlusion. To solve this problem, we propose a novel solution that integrates the 2D information (RGB images), the 3D information (RGB-D images) and the object/scene context information together, and call it the Context-Assisted 3D (C3D) method. In the proposed C3D method, we first use a convolutional neural network (CNN) to jointly detect a 3D object in a scene and its scene category. Then, we improve the detection result furthermore with a Conditional Random Field (CRF) model that incorporates the object potential, the scene potential, the scene/object context, the object/object context, and the room geometry. Extensive experiments are conducted to demonstrate that the proposed C3D method achieves the state-of-the-art performance for 3D object detection against the SUN RGB-D benchmark dataset.

In order to improve the accuracy of traditional motion image pose contour extraction and shorten the extraction time, a motion image pose contour extraction method based on B-spline wavelet is proposed. Moving images are acquired through the visual system, the information fusion process is used to perform statistical analysis on the images containing motion information, the location of the motion area is determined, convolutional neural network technology is used to preprocess the initial motion image pose contour, and B-spline wavelet theory is used. The preprocessed motion image pose contour is detected, combined with the heuristic search method to obtain the pose contour points, and the motion image pose contour extraction is completed. The simulation results show that the proposed method has higher accuracy and shorter extraction time in extracting motion image pose contours.

As providing objects that pigs prefer can reduce the occurrence of tail-biting and aggression and consequently improve animal welfare, automatic recognition of pigs’ engagement with different objects can have practical value. Therefore, aim of this study was to develop a computer vision based approach that utilised a recurrent neural network-based deep learning algorithm to recognise pig enrichment engagement (EE) behaviours and preliminarily determine the preference to objects. Two pig pens were studied. 1 day of video was recorded in pen 1, which generated 2400 1 s EE and 2400 1 s non-EE episodes. 80% of these data was randomly selected as training set and the remaining 20% as validation set. Moreover, 4 days of video were recorded and used as the test set in pen 2. Firstly, the HSV (Hue, Saturation, Value) colour space-based tracking algorithm was developed to locate object region of interest. Secondly, the convolutional neural network (CNN) architecture InceptionV3 was used to extract spatial features from each frame. These features were input into the long short-term memory (LSTM) framework to extract spatial-temporal features from each episode. Through the fully connected layer, the prediction function Softmax was finally used to classify these episodes as EE or non-EE behaviour. In the validation set, the proposed algorithm could recognise EE with blue ball, golden ball and wooden beam with an accuracy of 95.2%, 95.4% and 97.3%, respectively. By shortening the radius of the region of interest into a half of the average length of pig body, the corresponding accuracy could be improved into 96.9%, 97.1% and 97.9%, respectively. In the test set, the proposed algorithm could recognise EE with each of these 3 objects with an accuracy of 96.5%, 96.8% and 97.6%, respectively. The proportion of EE with each of these 3 objects was 75.8%, 6.0% and 18.2%, respectively. These results indicate that the proposed method can be used to recognise EE behaviours of pigs, and halving the radius of the region of interest can improve the recognition accuracy of EE behaviours. Moreover, the preference of pigs to objects based on EE duration were preliminarily determined as blue ball > wooden beam > golden ball. The obtained duration of EE behaviours can help farmers to evaluate the enrichment used and thereby to increase the health and welfare of the pigs in their care. Furthermore, the proposed algorithm has reference value for the classification of the behaviours with similar motion patterns.

Deep Convolutional Neural Network models achieve state-of-the-art performance on object detection, but it is driven by large labeled data sets. For general practical object detection problems, the existing public data set is no longer applicable, and there is no ready-made data set to solve the problem, usually only the description of object detection task is available. The method of collecting data sets from real world is really time consuming and unreachable in some special scenarios. In this paper, we define this problem as a new object detection problem named Task Description Object Detection, and present a synthetic image-based method to solve this problem. We design a system to generate annotated synthetic images quickly and inexpensively according to the task description. What's more, we propose a novel layer called SOMConv for object detection network to adapt the object detection model trained with the synthetic data sets to detection in real world. Our experiments evidence the effectiveness of our approach on real-world object detection problems, which only has the description of the object detection problem and no real images.

Sequence-based view invariant transform can effectively cope with view variations.Enhanced skeleton visualization method encodes spatio-temporal skeletons as visual and motion enhanced color images in a compact yet distinctive manner.Multi-stream convolutional neural networks fusion model is able to explore complementary properties among different types of enhanced color images.Our method consistently achieves the highest accuracies on four datasets, including the largest and most challenging NTU RGB+D dataset for skeleton-based action recognition. Human action recognition based on skeletons has wide applications in humancomputer interaction and intelligent surveillance. However, view variations and noisy data bring challenges to this task. Whats more, it remains a problem to effectively represent spatio-temporal skeleton sequences. To solve these problems in one goal, this work presents an enhanced skeleton visualization method for view invariant human action recognition. Our method consists of three stages. First, a sequence-based view invariant transform is developed to eliminate the effect of view variations on spatio-temporal locations of skeleton joints. Second, the transformed skeletons are visualized as a series of color images, which implicitly encode the spatio-temporal information of skeleton joints. Furthermore, visual and motion enhancement methods are applied on color images to enhance their local patterns. Third, a convolutional neural networks-based model is adopted to extract robust and discriminative features from color images. The final action class scores are generated by decision level fusion of deep features. Extensive experiments on four challenging datasets consistently demonstrate the superiority of our method.

A novel CNN architecture and non-iterative method to rapidly reconstruct signals from CS measurements for single pixel camera was proposed. This network reconstructed CS signals by dealing with different signal resolution scales step by step. The method divided the CS reconstruction into 2 stages. The first stage is the signal reconstruction at a low-resolution scale. The second stage is the upsampling procedure. At the same time, this study improved the loss function of the CNN model by adding a feature-based error. This loss function helped improve detail reconstruction of the signal. On a standard image set, the network showed significant improvement in performance, especially in time complexity. The network was more robust than traditional methods in noisy signal and low measurement-rate (MR) conditions. Even in extremely low MRs, the network could reconstruct an image very well. Furthermore, the network was tested with real data obtained from a single-pixel camera. The experiments showed that the network had better performance than other methods, and that it also exhibited robustness against sensor noise.

With the increasing number of vehicles, the number of traffic accidents also increases, especially at traffic lights. To enhance the driving safety at traffic lights, in this paper, we propose an intelligent safe driving assistant to provide drivers with driving advice based on traffic light phases, which information has been neglected by existing research. The driving assistant consists of an image recognition system with a single on-board camera, which can ameliorate the difficulties of observing traffic light phases. The recognition system obtains traffic light countdown information using a Convolutional Neural Network, and estimates the countdown time using the results of traffic light information. In addition, we develop a model to calculate the distance between the traffic light and vehicle by using the information of camera and traffic light. Based on the traffic light phase and the distance obtained, the driving assistant can provide a velocity control strategy to improve driver's safety. Finally, extensive experiments are conducted to verify the effectiveness of proposed methods.

While significant work has been conducted to perform source camera model identification for images, little work has been done specifically for video camera model identification. This is problematic because different forensic traces may be left in digital images and videos captured by the same camera. As our experiments in this paper will show, a system trained to perform camera model identification for images yields unacceptably low performance when given video frames from the same cameras. To overcome this problem, new systems for identifying a videos source must be developed. In this paper, we propose a deep learning based system for determining the source camera model that captured a digital video. To do this, we use a convolutional neural network to produce camera model identification scores for small patches taken from video frames. These patches are chosen by a patch selection system that obtains patches from several appropriate frames temporally distributed throughout the video. Forensic information obtained by the CNN is provided to a fusion system, which combines it to produce a single, more accurate identification result. Through a series of experiments, we evaluate several system design choices and show that our system can achieve 95.9% video camera model identification accuracy.

Simultaneous localization and mapping (SLAM) is a key component for mobile robot navigation that enables many service robotic applications. The capacity of acquiring accurate 3D-map of an environment is critical for robots to perform various tasks with a high degree of autonomy. Due to the indoor environment complexity and sensor uncertainties, SLAM remains a challenging task in the domain of 3D reconstruction. In this paper, we propose a simple yet effective solution for RGB-D based SLAM by integrating an Inertial Measurement Unit (IMU) into a recurrent and convolutional neural network that leads to enhanced pose estimation and point cloud registration. The IMU data provide an advantage of fast rate inertial data measurement and drift error reduction. Specifically, by imposing additional constraints from the IMU device, an optimal long-short term memory LSTM) is trained to mitigate scale ambiguity thus improve the concatenated ego-motion estimation. Compared to existing SLAM techniques and recent effort of RNN based solutions for 3D reconstruction, we show that our approach is competitive with high accuracy and robustness.

Regularly inspecting the components inside nuclear power plants is necessary to ensure their safe performance. Current inspection practices, however, are time consuming, tedious, and subjective that need technicians to watch videos and manually annotate cracks. While an autonomous inspection approach is desirable, state-of-the-art crack detection algorithms cannot perform well since the cracks on nuclear power plant reactors are typically very tiny with low contrast. The existence of scratches, welds, and grind marks on the surface makes the autonomous detection even more challenging. This study introduces a new framework that consists of a convolutional neural network (CNN) based on deep learning, a spatiotemporal registration process, and a Nave Bayes data fusion scheme based on statistics. The proposed framework is evaluated using several inspection videos and achieves 98.3% hit rate against 0.1 false positives per frame where the hit rate is much higher than other state-of-the-art algorithms.

The automated building detection in aerial images is a fundamental problem encountered in aerial and satellite images analysis. Recently, thanks to the advances in feature descriptions, Region-based CNN model (R-CNN) for object detection is receiving an increasing attention. Despite the excellent performance in object detection, it is problematic to directly leverage the features of R-CNN model for building detection in single aerial image. As we know, the single aerial image is in vertical view and the buildings possess significant directional feature. However, in R-CNN model, direction of the building is ignored and the detection results are represented by horizontal rectangles. For this reason, the detection results with horizontal rectangle cannot describe the building precisely. To address this problem, in this paper, we proposed a novel model with a key feature related to orientation, namely, Oriented R-CNN (OR-CNN). Our contributions are mainly in the following two aspects: 1) Introducing a new oriented layer network for detecting the rotation angle of building on the basis of the successful VGG-net R-CNN model; 2) the oriented rectangle is proposed to leverage the powerful R-CNN for remote-sensing building detection. In experiments, we establish a complete and bran-new data set for training our oriented R-CNN model and comprehensively evaluate the proposed method on a publicly available building detection data set. We demonstrate State-of-the-art results compared with the previous baseline methods.

Action recognition using depth sequences plays important role in many fields, e.g., intelligent surveillance, content-based video retrieval. Real applications require robust and accurate action recognition method. In this paper, we propose a skeleton visualization method, which efficiently encodes the spatial-temporal information of skeleton joints into a set of color images. These images are served as inputs for convolutional neural networks to extract more discriminative deep features. To enhance the ability of deep features to capture global relationships, we extend the color images into multi-temporal version. Additionally, to solve the effect of view point changes, a spatial transform method is adopted as a preprocessing step. Extensive experiments on NTU RGB+D dataset and ICME2017 challenge show that our method can accurately distinguish similar actions and shows robustness to view variations.

It remains a challenge to efficiently represent spatial-temporal data for 3D action recognition. To solve this problem, this paper presents a new skeleton-based action representation using data visualization and convolutional neural networks, which contains four main stages. First, skeletons from an action sequence are mapped as a set of five dimensional points, containing three dimensions of location, one dimension of time label and one dimension of joint label. Second, these points are encoded as a series of color images, by visualizing points as RGB pixels. Third, convolutional neural networks are adopted to extract deep features from color images. Finally, action class score is calculated by fusing selected deep features. Extensive experiments on three benchmark datasets show that our method achieves state-of-the-art results.

In this paper, we address the problem of cross-view image geo-localization. Specifically, we aim to estimate the GPS location of a query street view image by finding the matching images in a reference database of geo-tagged bird's eye view images, or vice versa. To this end, we present a new framework for cross-view image geo-localization by taking advantage of the tremendous success of deep convolutional neural networks (CNNs) in image classification and object detection. First, we employ the Faster R-CNN to detect buildings in the query and reference images. Next, for each building in the query image, we retrieve the $k$ nearest neighbors from the reference buildings using a Siamese network trained on both positive matching image pairs and negative pairs. To find the correct NN for each query building, we develop an efficient multiple nearest neighbors matching method based on dominant sets. We evaluate the proposed framework on a new dataset that consists of pairs of street view and bird's eye view images. Experimental results show that the proposed method achieves better geo-localization accuracy than other approaches and is able to generalize to images at unseen locations.

The task of estimating the spatial layout of cluttered indoor scenes from a single RGB image is addressed in this work. Existing solutions to this problem largely rely on hand-crafted features and vanishing lines, and they often fail in highly cluttered indoor scenes. The proposed coarse-to-fine indoor layout estimation (CFILE) method consists of two stages: (1) coarse layout estimation; and (2) fine layout localization. In the first stage, we adopt a fully convolutional neural network (FCN) to obtain a coarse-scale room layout estimate that is close to the ground truth globally. The proposed FCN combines the layout contour property and the surface property so as to provide a robust estimation in the presence of cluttered objects. In the second stage, we formulate an optimization framework that enforces several constraints such as layout contour straightness, surface smoothness and geometric constraints for layout detail refinement. Our proposed system offers the state-of-the-art performance on two commonly used benchmark datasets.