Search

Inspired by human behavior when traveling over unknown terrain, this study proposes the use of probing strategies and integrates them into a traversability analysis framework to address safe navigation on unknown rough terrain. Our framework integrates collapsibility information into our existing traversability analysis, as vision and geometric information alone could be misled by unpredictable non-rigid terrains such as soft soil, bush area, or water puddles. With the new traversability analysis framework, our robot has a more comprehensive assessment of unpredictable terrain, which is critical for its safety in outdoor environments. The pipeline first identifies the terrain’s geometric and semantic properties using an RGB-D camera and desired probing locations on questionable terrains. These regions are probed using a force sensor to determine the risk of terrain collapsing when the robot steps over it. This risk is formulated as a collapsibility metric, which estimates an unpredictable region’s ground collapsibility. Thereafter, the collapsibility metric, together with geometric and semantic spatial data, is combined and analyzed to produce global and local traversability grid maps. These traversability grid maps tell the robot whether it is safe to step over different regions of the map. The grid maps are then utilized to generate optimal paths for the robot to safely navigate to its goal. Our approach has been successfully verified on a quadrupedal robot in both simulation and real-world experiments.

Multi-human tracking in the crowded environment is a challenging problem due to occlusions, pose change, viewpoint variation and cluttered background. In this work, we propose a robust feature learning for tracking-by-detection methods based on second-order attention network that can capture higher-order relationships between salient features at the early stages of Convolutional Neural Network (CNN). Guided Second-Order Attention Network (GSAN) that, unlike the existing attention learning methods which are weakly-supervised, uses a supervisory signal based on the quality of the self-learned attention maps. More specifically, GSAN looks into the attended maps of a person having the highest confidence and supervise itself to look into the correct regions in the images of the person. Attention maps learned this way are spatially aligned and thus robust to camera-view changes and body pose variations. We verify the effectiveness of our approach by comparing with the state-of-the-art methods on challenging person re-identification and multi object tracking (MOT) datasets.

Localization can be achieved by different sensors and techniques such as a global positioning system (GPS), wifi, ultrasonic sensors, and cameras. In this paper, we focus on the laser-based localization method for unmanned aerial vehicle (UAV) applications in a GPS denied environment such as a deep tunnel system. Other than a low-cost 2D LiDAR for the planar axes, a single axis Lidar for the vertical axis as well as an inertial measurement unit (IMU) device is used to increase the reliability and accuracy of the localization performance. We present a comparative analysis of the three selected laser-based simultaneous localization and mapping(SLAM) approaches:(i) Hector SLAM; (ii) Gmapping; and(iii) Cartographer. These algorithms have been implemented and tested through real-world experiments. The results are compared with the ground truth data and the experiments are available at https://youtu.be/kQc3mJjw_mw., Accepted to IROS 2019 Workshop on Challenges in Vision-based Drones Navigation

A new approach for autonomous exploration in an unknown scenario based on the concept of frontiers is proposed in this paper. Exploration frontiers introduced by [4], are the regions on the boundary between open space and unexplored space. A mobile robot is able to construct its map by adding new space and moving to unvisited frontiers until the entire environment has been explored. However, the original frontier strategy, suffering from local minima, only considers distance and size of unknown spaces, resulting in low exploration efficiency in complex environments. By making use of the robot heading information using wheel odometry and coarse graph representation of the environment, the modified exploration method is able to balance the mapping coverage and time expenditure to a greater extent. The proposed method is experimentally verified on a mobile platform, exploring a real-world office environment cluttered with a variety of obstacles.

Automatically delivering objects to in-car passengers has many potential applications. Such a system generally consists of two sub-systems: a perception system and an action system. The perception system basically looks for the targets' positions and the action system delivers objects to the targets. In this paper, we propose a novel perception system, which contains two major functions: estimation of reaching points and discovering potential risks. The reaching points are the locations where robot arms needs to reach. Moreover, it should be able to reach with comfort by passengers and keep a safe distance from the car body. In order to achieve this, all the vehicle components (side surfaces, side mirrors etc.), which may cause collision, need to be detected. Potential risks are usually caused by moving objects or changing door state (close to open) during the operation. It is necessary to monitor these two situations to avoid any potential risks during operation. Our offline test shows that the accuracy of reaching points estimation can reach up to 94% and the response time for moving objects detection or door state changes is less than 1 millisecond.

In recent years, an attempt is being made to control robots more intuitive and intelligible by exploiting and integrating anthropomorphic features to boost social human-robot interaction. The design and construction of anthropomorphic robots for this kind of interaction is not the only challenging issue -- smooth and expectation-matching motion control is still an unsolved topic. In this work we present a highly configurable, portable, and open control framework that facilitates anthropomorphic motion generation for humanoid robot heads by enhancing state-of-the-art neck-eye coordination with human-like eyelid saccades and animation. On top of that, the presented framework supports dynamic neck offset angles that allow animation overlays and changes in alignment to the robots communication partner whileretaining visual focus on a given target. In order to demonstrate the universal applicability of the proposed ideas we used this framework to control the Flobi and the iCub robot head, both in simulation and on the physical robot. In order to foster further comparative studies of different robot heads, we will release all software, based on this contribution, under an open-source license.

This paper presents an approach to guide a robot arm visually, by using an active stereo camera mounted on an active head platform. The camera and head system is capable of changing the look-direction and fixating on the target object. Unlike the traditional AI approach, the proposed approach does not require full perspective camera calibration or scene reconstruction. Visual parameters which contain relationships with the robot or world space are used instead of the reconstructed three-dimensional (3D) world information. The necessary visual cues required to control the robot are developed. These visual cues are computed solely from the image coordinates. The relationship between these visual cues and the robot space is then established via a mapping matrix. Using this relation together with visual feedback, the end-effector can be guided towards the target object. Even though the 3D world coordinates are not recovered, the positioning accuracy attainable is high. To allow for vergence movement of the stereo camera or pan–tilt movements of the head platform, methods to compensate the mapping matrix are proposed. This approach eliminates the need for re-calibration after any deliberate change in the configuration of the head platform. Therefore, building an autonomous robot with coordinated head, eye and hand becomes technically feasible. Simulation studies on the capability of the system to reach arbitrary target objects are presented. High flexibility and large accessible workspace are the main advantages of the proposed autonomous robot system.

The conventional mouth gender recognition is based on a static image and ignore the dynamic information. In this paper, we propose a lip movement gender recognition method to improve the accuracy by exploring the dynamic information while a user is speaking. In order to overcome the difficulty caused by the nonlinear distribution of the lip images, Gausian Mixture Models (GMM) is adopted to represent the lip images. A similarity measure is defined to measure the difference between the successive frames. Gender recognition, as a soft biometric trait, can provide useful information for improving the performance of the speaker recognition systems. The accuracy of voice-based speaker gender recognition is high if the condition of the environment is good. But it will be drastically decreased if the test is conduted in a noisy environment. In this paper, we showed that lip movement, considered as a sequence of mouth images, can provide additional information than mouth alone for recognizing gender Experimental result obtained showed the effectiveness of the proposed method which is comparable to using just the voice information.

Over the last decade, automated analysis of human affective behavior has become an active research area in computer science, psychology, neuroscience, and related fields. This study investigates the application of Gabor filter based features in combination of Genetic Algorithm (GA) and Support Vector Machine (SVM) for dynamic analysis of six basic facial expressions from video sequences. Traditionally, a set of Gabor filters is used for feature extraction from static images of face. However, we employed Sum of Difference (SOD) approach to analysis the dynamics of facial expression from a video sequence. We also used GA to overcome the problem of high dimensional feature vectors and computation cost. A local Gabor filter bank with selected frequencies and orientations is produced by GA. The experimental results show that the proposed method is effective for temporal analysis of affective states. The detection rate of six basic emotions has been reached to 92.97% for Cohn-Kanade (CK+) database. ASTAR (Agency for Sci., Tech. and Research, S’pore) Accepted version

The use of vision system to control a robot arm increases the flexibility of the robot arm in executing its tasks. At present, such eye-hand coordination is mostly implemented using rigid or passive camera system. Usually, camera calibration is required to recover the homogeneous transformation matrix and to reconstruct the three dimensional world in the robot space based on the image data acquired. However, the passive camera system limits the operating range and the accuracy of the eye-hand system. Increasing the operating range of such a system inevitably reduces the achievable precision and vice versa. Furthermore, the need for camera calibration causes the system to be sensitive to perturbations of the vision system. The work presented is an attempt to address these shortcomings. Doctor of Philosophy (EEE)

Computed Tomography is widely used for non-invasive diagnosis of the human brain. Together with stereotactic frames, CT images allow for high precision in brain lesion resection operations. In view of the numerous images produced by the CT machine and with increasing cases that require follow-up operations, computer assisted neurosurgery becomes a useful aid for neurosurgeons. Master of Engineering

There has been increasing interest on the use of video surveillance to analyze human biometrics, especially in places where the chance of occurring crime is high, such as in large cities. As an example, 2185 children are being reported missing daily in the United States. It has been rarely looked into belonging a person to the same family as a soft biometric modal. We term the ability to verify query images as family members “family (kinship) verification.” Several studies have been conducted on family photo albuming, but the problem of family verification has not yet been directly addressed and analyzed. The lack of comprehensive published family datasets is evidence that the matter has received very little attention to date. This thesis focuses on providing solutions for challenges of family verification as a recent application of computer vision. We propose frameworks to recognize members of the same family based on the facts and psychological studies on family members’ resemblance to each other. We also investigate the demanding case of missing/unknown member verification as a critical real-life application that will benefit from this research. In the first major section of this thesis, we study true assumptions behind the resemblance of family members to one another, and the challenges of family verification. We develop traditional face recognition approaches to facilitate family verification by extraction of facial resemblance among family members. We investigate the very recent family verification problem, define scenarios and collect a dataset of family albums. Our aim is to recognize if the query sample belongs to the same family. In the first stage, we investigate facial resemblance extraction among family members to perform family verification. Similarities of facial patches are utilized as multiple experts to recognize family members. Finally, our proposed inter-patch constraint-free algorithm of facial patches analysis determines the redundant patches among family members. On average, five facial patches (29% of the facial patches) are chosen to achieve the same accuracy obtained by using all patches for family verification. Consequently, the computational time is saved by 71% compared to utilizing all patches. The second major section of our thesis is focused on family verification in real-time systems to obtain more discriminative features with the least dimension. Firstly, we minimize the quantization error of the Local Binary Pattern (LBP) operator through the incorporation of uniformly-sampled thresholds for LBP (UTLBP). Up to 3.4% improvement was achieved for family verification compared to various features in the literature, while the feature dimension is considerably smaller than the well-known Haar-like features (60%). Moreover, the proposed operator is more robust than the conventional operators, as tested on the large CAS-PEAL dataset against background, illumination, aging, and accessories changes. It achieves up to 8% performance improvement depending on the CAS-PEAL probe set, compared to the best operator in the literature. Then, we propose a novel redundant feature removal algorithm based on the fact that there is less similarity among the faces of family members than that of an individual’s face samples. The proposed algorithms are then employed on the UTLBP feature operator to select the most informative set of thresholds to outperform the Haar-like features, with 20 times less feature dimension. The proposed approach converges 10 times faster than the state-of-the-art face recognition algorithms do. The third contribution of the thesis is to utilize from each member’s image segment resemblance to the entire family. Our proposed method is to incorporate the degree of resemblance of each individual member’s patch to perform family verification. Our analysis and the important consensuses of psychologists’ studies reveal that the facial resemblance between family members differs from member to member and is facial patch specific. We propose to estimate the amount of resemblance for each patch based on the prior information of the member. As the score level fusion is preferred in the literature due to its wealth of information and low data dimension, we embed the available a priori information in the score fusion rule. The proposed method outperforms the state-of-the-art score fusion rules in all scenarios by 14%. Moreover, the resemblance estimation slightly outperforms human observation in the designated survey. The proposed method could also improve object family verification on the selected Caltech-256 database by 8%. In summary, two automated frameworks for recognizing family members are proposed. The first framework utilizes UTLBP operator and the novel redundant feature removal provides fast and low computational algorithms suitable for real-time processing. The other framework of facial patch resemblance extraction among family members benefits from offline training with parallel computing capability. Finally, the enhancement of family verification can potentially solve critical applications of daily life to find missing children and unknown parents. DOCTOR OF PHILOSOPHY (EEE)

In recent decades, automatic fingerprint identification system has attracted significant interest among the scientific research community. Though large amount of effort has been made in both the academic and industrial communities, there are still unsolved problems. Noisy image processing and large scale database processing are two typical problems. Fingerprint images consist of oriented texture. Therefore the orientation information plays a very important role in fingerprint processing. Moreover, the orientation of the fingerprint follows a certain structure that is not random. Such oriented structure provides a possibility to recover the orientation information in fingerprints corrupted by noise, even for a large noise patch which cannot be solved by the traditional gradient based methods. Accurate reconstruction of the orientations is useful for fingerprint filtering, segmentation, classification and recognition. Several methods have been proposed to reconstruct the fingerprint orientation. In this thesis, phase portrait model approach is used. The first order phase portrait is demonstrated to be robust for both occluded and noisy fingerprints. It also has the capability to describe the orientation patterns of the core point and delta point in a fingerprint image. Therefore a method based on the first order phase portrait is proposed to reconstruct the local orientation near the core and delta points. Furthermore, considering the regularity of the fingerprint orientation and linearity of the first order phase portrait, a model based on the piecewise phase portrait is used to predict the orientations far away from the singular point region. The advantage of this prediction model is that it provides a way to overcome the large patch of noise or missing orientation. To obtain the fingerprint orientation with higher precision, more complicated model based on the constrained nonlinear phase portrait approach is proposed. The model employs the advantages of both the first order and high order phase portrait model. Hence accurate reconstructed orientations at both the local and global area can be obtained simultaneously. This thesis also addresses fingerprint classification using the coefficients of the constrained nonlinear phase portrait model together with the singularities information. Since the coefficients of the constrained nonlinear phase portrait model play a key role in reconstructing the fingerprint orientation, it is also expected to be able to effectively classify fingerprints. A fingerprint classification algorithm using the orientation model and the singular points' information which complement the orientation information is developed and is shown to have good classification performance. DOCTOR OF PHILOSOPHY (EEE)

Computer vision deals with providing visual capabilities to a computer so that it can understand its surrounding environment. This has given rise to numerous applications such as human-computer interactions, object detection, scene understanding, surveillance etc. Understanding human behaviour is a difficult task due to its fickle nature. A person’s action is one of the visible clues towards one’s behaviour or intention. Due to this, research in action recognition has gained momentum in recent years. Actions can be mainly characterized by the motion of body parts and evolution of the pose of the person. In an unconstrained setting, achieving a clear and distinct action representation is very difficult due to pose variations caused as a result of variation in camera view angle changes, occlusion of body parts, cluttered background etc. Also, traditional action recognition datasets portray actions that have subjects mainly in upright postures such as walking, running, boxing etc. There is lack of study related non-upright posture action recognition. Thus in this research, we develop a novel pose-invariant action recognition which combines both motion of body parts and pose of the person in a mutually reinforcing manner. The proposed method is generalized to handle possible human posture variations (including non-upright postures). The assumption made is that the neck point of the person and major viewing direction of the person during motion are available in the frame. Our pose-invariant action recognition framework is inspired from closed loop framework of control system theories with two components, namely, pose-invariant motion feature extraction component (Forward path) and canonical pose component (Feedback path) with the output of each used as input to the other to reinforce the initial prediction in a manner that improves the overall final result. The two components of our proposed action recognition framework are • Pose-Invariant Motion Component or Propagation Motion Forward (PMF) Path In this component, we aim to capture motion of different body parts that characterizes an action. To achieve this, a propagation mechanism is used to divide the region of interest is divided into 3 grids corresponding to the head, torso and legs respectively based on the estimated body orientation of the person in the frame. To capture the motion in each grid, we use a set of kinematic features derived from optical flow and encode them in a pose-invariant manner by converting them to a human-body centric space. Using these initial set of extracted invariant kinematic motion features, we provide initial hypothesis of possible actions. • Canonical Pose Component or Canonical Pose Feedback (CPF) Path The initial action recognized based on invariant kinematic motion features are used as input by this component. Each action is characterized by specific poses, we call these as canonical stick poses. Given an action and available training videos, we propose an algorithm to extract normalized canonical stick poses that can be compared with the test videos. For comparing these stick poses with the visual data, we propose a novel pose hypothesis generation scheme that compares each of the extracted canonical sticks of the action recognized by the first component with the video frame to identify the most likely canonical stick pose in it. The identified canonical stick pose in the frame is used to improve the pose-invariant motion feature extraction by realigning the grids used in the first component and also compute another kinematic motion feature of each body part with respect to the center of that body part. To capture temporal dynamics of action, identified stick poses are used to compute a new set of temporal stick features. The new set of pose-invariant motion features extracted from both forward and feedback paths using the realigned grids serves as the final action representation. Using these features, action hypothesis is refined until the action converges to yield the recognized action. In our canonical pose feedback path, to compare each canonical stick pose with the test frame, we propose a canonical pose hypothesis scheme that uses a body part detector as it’s basis. Most body part detectors or person detectors available work based on the assumption that the person is in an upright posture. Therefore, we propose our own body part detector that can work even when the person is in non-upright postures. In our body part detector, we do not impose any connectivity or shape constraints between the parts allowing us to detect body parts in non-upright postures. We divide the human body into 4 body parts head, torso, arm and leg. For detecting these body parts, training images were collected from the web using a python crawler. Using the training images, body part classifiers were trained. Based on the Bayes’ theorem for conditional probability, we propose an algorithm to compute the likelihood scores for each of the detected parts in the frame. The algorithm uses the developed classifier’s output to determine a likelihood score for the detected body parts. This likelihood score is used by the pose hypothesis generation scheme of the canonical pose component. To test our method’s effectiveness under non-upright posture, we introduce a new action dataset comprising of 8 actions in different postures captured from 3 views. The dataset comprises of visual data for 35 subjects. We have conducted several experiments on publicly available benchmark action datasets such as Weizmann, KTH, UCF Sports, Hollywood, HMDB51 and this new dataset to test the effectiveness of our recognition framework. The results show that the proposed reinforcement framework for action recognition is pose-invariant, partially view-invariant and is able to work even if there is partial occlusion of the person performing the action. Doctor of Philosophy (EEE)

Face detection, localization and recognition from multi-poses are one of the most challenging topics in the area of computer vision and pattern recognition. During the past decade several methods have been proposed for face detection and recognition but they mainly concentrate on frontal faces. Although in recent years, some methods have been proposed based on sliding windows to exhaustively search through the image in many scales and views, they are slow and their performance on multi-view or multi-pose faces are not as good as frontal face detections. Training these methods is also a bottleneck. Most of the proposed methods require thousands of positive and negative samples from different face poses for training. These images for training have to be manually cropped, aligned and labeled, thus is manually intensive and expensive to perform. Note that since the effect of variation in the face pose captured on a fix camera and change in camera view but with a fixed face pose is the same, we used both words interchangeably, In this research project, a probabilistic approach is proposed for face detection, localization and identification from multi-views. The proposed method does not supervision during the training stage. The main focus of the project is on multi-view analysis as well as unsupervised or automated learning. The proposed approach provides a unified framework to learn a multi-view model for face detection, localization and identification. For face detection and localization from multi poses, given the images of different people in multi-poses, it obtains a multi-view face model obtained through a constellation of corresponding face features in the training set. The obtained model is pruned such that only the most distinctive features are retained. The model is then used to detect and localize multiple face images in multi-poses. Two versions of model construction are proposed for face detection and localization. One requires manually labeling only two control points in one image of the training set regardless of the number of images in the training set. The other version is completely automatic. Even the control points are estimated automatically and the system only requires some training images that include face of different people from multi-views. The trade-off is that the accuracy is slightly lower compared to the manually labeled approach. Another effort is face identification and localization from multi poses. The proposed method is completely automatic. The data collection is also performed by the system using web images obtained from web search. It requires only the textual name of the queried person, which is used as input to the search engine. Using the images returned by the search engine, the system classifies these images and constructs a multi-view face model for the query. With the model, the system is then able to identify the query candidate in his/her digital photo album or gallery. The user does not have to manually tag the images or provide images to train the face recognition module. Since system is completely automatic, it has widely application including search for people in news video or tagging actors and actresses in movies or targeted advertisement for interactive televisions (IPTV). DOCTOR OF PHILOSOPHY (EEE)

Facial Emotion detection under natural conditions is an interesting topic with a wide range of potential applications like human-computer interaction. Although there is significant research progress in this field, there are still challenges related to real-world unconstrained situations. One essential challenge is to find pose invariant spatio-temporal volumetric features to analyze the video sequence efficiently. Another important issue is how to deal with noisy and imperfect data recorded in uncontrolled environments such as illumination variations, partial occlusion, and head movements. The focus of this research is to develop a robust system for facial expression recognition as a dynamic event in natural situations. Two strategies have been proposed in this research to address the uncontrolled environments related challenges: Robust representation framework: we propose a novel spatio-temporal descriptor based on Optical Flow (OF) components which is very distinctive and also pose-invariant.  Robust recognition framework: we explored the effectiveness of sparse representation obtained by supervised learning a set of basis (dictionary). Extreme Sparse Learning (ESL) is proposed to jointly learn a dictionary and a nonlinear classification model to robustly detect the facial expression in real-world natural situations. The proposed approach combines the discriminative power of the Extreme Learning Machine (ELM) with the reconstruction property of the sparse representation to deal with noisy signal and imperfect data recorded in natural settings. Since the facial feature extraction performance is highly dependent on facial pose, we propose a novel spatio-temporal descriptor which is robust to facial pose variations. However, the feature encoding may fail in the presence of extreme head pose variations, where some parts of the face are not visible in the recorded images. To address this problem and also dealing with illumination variations and occlusion, we suggested following the idea of sparse representation where the noisy data can be reconstructed from the clean data provided by the dictionary of the sparse representation. While the sparse representation approach has the ability to enhance noisy data using a dictionary learned from clean data, it is not sufficient because the end goal is to correctly recognize the facial expression. In a sparse-representation-based classification task, the desired dictionary should have both representational ability and discriminative power. Since separating the classification training from dictionary learning may cause the learned dictionary to be sub-optimal for the classification task, we propose to jointly learn a dictionary and classification model. In other words, in contrast with most existing schemes that attempt to update the dictionary and classifier parameters alternately by iteratively solving each sub-problem, we propose to solve them simultaneously. This joint dictionary learning and classifier training can be expected to result in a dictionary that is both reconstructive and discriminative for a robust recognition system. To the best of our knowledge, this is the only work that attempts to simultaneously learn the sparse representation of the signal and train a nonlinear classifier to be discriminative for sparse codes. The proposed method jointly learns a single dictionary and also an optimal nonlinear classifier. We have performed extensive experiments on both acted and spontaneous emotion databases to evaluate the effectiveness of the proposed feature extraction and classification schemes under different scenarios. Our results clearly demonstrate the robustness of the proposed emotion recognition framework, especially in challenging scenarios that involve illumination changes, occlusion, and head pose variations. DOCTOR OF PHILOSOPHY (EEE)

Gaussian process (GP) is a stochastic process that has been studied for a long time and gained wide interests in the machine learning community in recent years. In this thesis, several interesting pattern analysis problems are solved using Gaussian process. Gaussian process models can be interpreted in two views, the weight space view and the function space view. Their different interpretations can be helpful in applying GPs to solve real problems. Its capability as functional prior inspired the original work on learning nonparametric similarity measure in this thesis. The similarity between pairwise inputs is considered a smooth function described by a GP prior. Given known similarity constraints, the model can be tuned by maximum likelihood and similarity on new inputs can be inferred. The advantage is that the learned similarity measure does not assume any parametric form. It is flexible to model various data with arbitrary distributions, and handles noise and outliers in the data. Gaussian process latent variable model (GPLVM) is a data modeling tool that is derived based on Gaussian processes from the weight space view. It was originally proposed for visualization of high dimensional data. DOCTOR OF PHILOSOPHY (EEE)

Estimating human poses from 2D images or video sequences can provide the moving trajectories of the body joints for the high level processing, human activity recognition, which is applicable in surveillance, human-computer interaction and clinical and sport analysis. This project proposes a new statistical formulation called the data-driven Monte Carlo hidden Markov model to estimate human poses from random initializations. MASTER OF ENGINEERING (EEE)