Search

There has been increasing interest on the use of video surveillance to analyze human biometrics, especially in places where the chance of occurring crime is high, such as in large cities. As an example, 2185 children are being reported missing daily in the United States. It has been rarely looked into belonging a person to the same family as a soft biometric modal. We term the ability to verify query images as family members “family (kinship) verification.” Several studies have been conducted on family photo albuming, but the problem of family verification has not yet been directly addressed and analyzed. The lack of comprehensive published family datasets is evidence that the matter has received very little attention to date. This thesis focuses on providing solutions for challenges of family verification as a recent application of computer vision. We propose frameworks to recognize members of the same family based on the facts and psychological studies on family members’ resemblance to each other. We also investigate the demanding case of missing/unknown member verification as a critical real-life application that will benefit from this research. In the first major section of this thesis, we study true assumptions behind the resemblance of family members to one another, and the challenges of family verification. We develop traditional face recognition approaches to facilitate family verification by extraction of facial resemblance among family members. We investigate the very recent family verification problem, define scenarios and collect a dataset of family albums. Our aim is to recognize if the query sample belongs to the same family. In the first stage, we investigate facial resemblance extraction among family members to perform family verification. Similarities of facial patches are utilized as multiple experts to recognize family members. Finally, our proposed inter-patch constraint-free algorithm of facial patches analysis determines the redundant patches among family members. On average, five facial patches (29% of the facial patches) are chosen to achieve the same accuracy obtained by using all patches for family verification. Consequently, the computational time is saved by 71% compared to utilizing all patches. The second major section of our thesis is focused on family verification in real-time systems to obtain more discriminative features with the least dimension. Firstly, we minimize the quantization error of the Local Binary Pattern (LBP) operator through the incorporation of uniformly-sampled thresholds for LBP (UTLBP). Up to 3.4% improvement was achieved for family verification compared to various features in the literature, while the feature dimension is considerably smaller than the well-known Haar-like features (60%). Moreover, the proposed operator is more robust than the conventional operators, as tested on the large CAS-PEAL dataset against background, illumination, aging, and accessories changes. It achieves up to 8% performance improvement depending on the CAS-PEAL probe set, compared to the best operator in the literature. Then, we propose a novel redundant feature removal algorithm based on the fact that there is less similarity among the faces of family members than that of an individual’s face samples. The proposed algorithms are then employed on the UTLBP feature operator to select the most informative set of thresholds to outperform the Haar-like features, with 20 times less feature dimension. The proposed approach converges 10 times faster than the state-of-the-art face recognition algorithms do. The third contribution of the thesis is to utilize from each member’s image segment resemblance to the entire family. Our proposed method is to incorporate the degree of resemblance of each individual member’s patch to perform family verification. Our analysis and the important consensuses of psychologists’ studies reveal that the facial resemblance between family members differs from member to member and is facial patch specific. We propose to estimate the amount of resemblance for each patch based on the prior information of the member. As the score level fusion is preferred in the literature due to its wealth of information and low data dimension, we embed the available a priori information in the score fusion rule. The proposed method outperforms the state-of-the-art score fusion rules in all scenarios by 14%. Moreover, the resemblance estimation slightly outperforms human observation in the designated survey. The proposed method could also improve object family verification on the selected Caltech-256 database by 8%. In summary, two automated frameworks for recognizing family members are proposed. The first framework utilizes UTLBP operator and the novel redundant feature removal provides fast and low computational algorithms suitable for real-time processing. The other framework of facial patch resemblance extraction among family members benefits from offline training with parallel computing capability. Finally, the enhancement of family verification can potentially solve critical applications of daily life to find missing children and unknown parents. DOCTOR OF PHILOSOPHY (EEE)

In recent decades, automatic fingerprint identification system has attracted significant interest among the scientific research community. Though large amount of effort has been made in both the academic and industrial communities, there are still unsolved problems. Noisy image processing and large scale database processing are two typical problems. Fingerprint images consist of oriented texture. Therefore the orientation information plays a very important role in fingerprint processing. Moreover, the orientation of the fingerprint follows a certain structure that is not random. Such oriented structure provides a possibility to recover the orientation information in fingerprints corrupted by noise, even for a large noise patch which cannot be solved by the traditional gradient based methods. Accurate reconstruction of the orientations is useful for fingerprint filtering, segmentation, classification and recognition. Several methods have been proposed to reconstruct the fingerprint orientation. In this thesis, phase portrait model approach is used. The first order phase portrait is demonstrated to be robust for both occluded and noisy fingerprints. It also has the capability to describe the orientation patterns of the core point and delta point in a fingerprint image. Therefore a method based on the first order phase portrait is proposed to reconstruct the local orientation near the core and delta points. Furthermore, considering the regularity of the fingerprint orientation and linearity of the first order phase portrait, a model based on the piecewise phase portrait is used to predict the orientations far away from the singular point region. The advantage of this prediction model is that it provides a way to overcome the large patch of noise or missing orientation. To obtain the fingerprint orientation with higher precision, more complicated model based on the constrained nonlinear phase portrait approach is proposed. The model employs the advantages of both the first order and high order phase portrait model. Hence accurate reconstructed orientations at both the local and global area can be obtained simultaneously. This thesis also addresses fingerprint classification using the coefficients of the constrained nonlinear phase portrait model together with the singularities information. Since the coefficients of the constrained nonlinear phase portrait model play a key role in reconstructing the fingerprint orientation, it is also expected to be able to effectively classify fingerprints. A fingerprint classification algorithm using the orientation model and the singular points' information which complement the orientation information is developed and is shown to have good classification performance. DOCTOR OF PHILOSOPHY (EEE)

Computer vision deals with providing visual capabilities to a computer so that it can understand its surrounding environment. This has given rise to numerous applications such as human-computer interactions, object detection, scene understanding, surveillance etc. Understanding human behaviour is a difficult task due to its fickle nature. A person’s action is one of the visible clues towards one’s behaviour or intention. Due to this, research in action recognition has gained momentum in recent years. Actions can be mainly characterized by the motion of body parts and evolution of the pose of the person. In an unconstrained setting, achieving a clear and distinct action representation is very difficult due to pose variations caused as a result of variation in camera view angle changes, occlusion of body parts, cluttered background etc. Also, traditional action recognition datasets portray actions that have subjects mainly in upright postures such as walking, running, boxing etc. There is lack of study related non-upright posture action recognition. Thus in this research, we develop a novel pose-invariant action recognition which combines both motion of body parts and pose of the person in a mutually reinforcing manner. The proposed method is generalized to handle possible human posture variations (including non-upright postures). The assumption made is that the neck point of the person and major viewing direction of the person during motion are available in the frame. Our pose-invariant action recognition framework is inspired from closed loop framework of control system theories with two components, namely, pose-invariant motion feature extraction component (Forward path) and canonical pose component (Feedback path) with the output of each used as input to the other to reinforce the initial prediction in a manner that improves the overall final result. The two components of our proposed action recognition framework are • Pose-Invariant Motion Component or Propagation Motion Forward (PMF) Path In this component, we aim to capture motion of different body parts that characterizes an action. To achieve this, a propagation mechanism is used to divide the region of interest is divided into 3 grids corresponding to the head, torso and legs respectively based on the estimated body orientation of the person in the frame. To capture the motion in each grid, we use a set of kinematic features derived from optical flow and encode them in a pose-invariant manner by converting them to a human-body centric space. Using these initial set of extracted invariant kinematic motion features, we provide initial hypothesis of possible actions. • Canonical Pose Component or Canonical Pose Feedback (CPF) Path The initial action recognized based on invariant kinematic motion features are used as input by this component. Each action is characterized by specific poses, we call these as canonical stick poses. Given an action and available training videos, we propose an algorithm to extract normalized canonical stick poses that can be compared with the test videos. For comparing these stick poses with the visual data, we propose a novel pose hypothesis generation scheme that compares each of the extracted canonical sticks of the action recognized by the first component with the video frame to identify the most likely canonical stick pose in it. The identified canonical stick pose in the frame is used to improve the pose-invariant motion feature extraction by realigning the grids used in the first component and also compute another kinematic motion feature of each body part with respect to the center of that body part. To capture temporal dynamics of action, identified stick poses are used to compute a new set of temporal stick features. The new set of pose-invariant motion features extracted from both forward and feedback paths using the realigned grids serves as the final action representation. Using these features, action hypothesis is refined until the action converges to yield the recognized action. In our canonical pose feedback path, to compare each canonical stick pose with the test frame, we propose a canonical pose hypothesis scheme that uses a body part detector as it’s basis. Most body part detectors or person detectors available work based on the assumption that the person is in an upright posture. Therefore, we propose our own body part detector that can work even when the person is in non-upright postures. In our body part detector, we do not impose any connectivity or shape constraints between the parts allowing us to detect body parts in non-upright postures. We divide the human body into 4 body parts head, torso, arm and leg. For detecting these body parts, training images were collected from the web using a python crawler. Using the training images, body part classifiers were trained. Based on the Bayes’ theorem for conditional probability, we propose an algorithm to compute the likelihood scores for each of the detected parts in the frame. The algorithm uses the developed classifier’s output to determine a likelihood score for the detected body parts. This likelihood score is used by the pose hypothesis generation scheme of the canonical pose component. To test our method’s effectiveness under non-upright posture, we introduce a new action dataset comprising of 8 actions in different postures captured from 3 views. The dataset comprises of visual data for 35 subjects. We have conducted several experiments on publicly available benchmark action datasets such as Weizmann, KTH, UCF Sports, Hollywood, HMDB51 and this new dataset to test the effectiveness of our recognition framework. The results show that the proposed reinforcement framework for action recognition is pose-invariant, partially view-invariant and is able to work even if there is partial occlusion of the person performing the action. Doctor of Philosophy (EEE)

Face detection, localization and recognition from multi-poses are one of the most challenging topics in the area of computer vision and pattern recognition. During the past decade several methods have been proposed for face detection and recognition but they mainly concentrate on frontal faces. Although in recent years, some methods have been proposed based on sliding windows to exhaustively search through the image in many scales and views, they are slow and their performance on multi-view or multi-pose faces are not as good as frontal face detections. Training these methods is also a bottleneck. Most of the proposed methods require thousands of positive and negative samples from different face poses for training. These images for training have to be manually cropped, aligned and labeled, thus is manually intensive and expensive to perform. Note that since the effect of variation in the face pose captured on a fix camera and change in camera view but with a fixed face pose is the same, we used both words interchangeably, In this research project, a probabilistic approach is proposed for face detection, localization and identification from multi-views. The proposed method does not supervision during the training stage. The main focus of the project is on multi-view analysis as well as unsupervised or automated learning. The proposed approach provides a unified framework to learn a multi-view model for face detection, localization and identification. For face detection and localization from multi poses, given the images of different people in multi-poses, it obtains a multi-view face model obtained through a constellation of corresponding face features in the training set. The obtained model is pruned such that only the most distinctive features are retained. The model is then used to detect and localize multiple face images in multi-poses. Two versions of model construction are proposed for face detection and localization. One requires manually labeling only two control points in one image of the training set regardless of the number of images in the training set. The other version is completely automatic. Even the control points are estimated automatically and the system only requires some training images that include face of different people from multi-views. The trade-off is that the accuracy is slightly lower compared to the manually labeled approach. Another effort is face identification and localization from multi poses. The proposed method is completely automatic. The data collection is also performed by the system using web images obtained from web search. It requires only the textual name of the queried person, which is used as input to the search engine. Using the images returned by the search engine, the system classifies these images and constructs a multi-view face model for the query. With the model, the system is then able to identify the query candidate in his/her digital photo album or gallery. The user does not have to manually tag the images or provide images to train the face recognition module. Since system is completely automatic, it has widely application including search for people in news video or tagging actors and actresses in movies or targeted advertisement for interactive televisions (IPTV). DOCTOR OF PHILOSOPHY (EEE)

Facial Emotion detection under natural conditions is an interesting topic with a wide range of potential applications like human-computer interaction. Although there is significant research progress in this field, there are still challenges related to real-world unconstrained situations. One essential challenge is to find pose invariant spatio-temporal volumetric features to analyze the video sequence efficiently. Another important issue is how to deal with noisy and imperfect data recorded in uncontrolled environments such as illumination variations, partial occlusion, and head movements. The focus of this research is to develop a robust system for facial expression recognition as a dynamic event in natural situations. Two strategies have been proposed in this research to address the uncontrolled environments related challenges: Robust representation framework: we propose a novel spatio-temporal descriptor based on Optical Flow (OF) components which is very distinctive and also pose-invariant.  Robust recognition framework: we explored the effectiveness of sparse representation obtained by supervised learning a set of basis (dictionary). Extreme Sparse Learning (ESL) is proposed to jointly learn a dictionary and a nonlinear classification model to robustly detect the facial expression in real-world natural situations. The proposed approach combines the discriminative power of the Extreme Learning Machine (ELM) with the reconstruction property of the sparse representation to deal with noisy signal and imperfect data recorded in natural settings. Since the facial feature extraction performance is highly dependent on facial pose, we propose a novel spatio-temporal descriptor which is robust to facial pose variations. However, the feature encoding may fail in the presence of extreme head pose variations, where some parts of the face are not visible in the recorded images. To address this problem and also dealing with illumination variations and occlusion, we suggested following the idea of sparse representation where the noisy data can be reconstructed from the clean data provided by the dictionary of the sparse representation. While the sparse representation approach has the ability to enhance noisy data using a dictionary learned from clean data, it is not sufficient because the end goal is to correctly recognize the facial expression. In a sparse-representation-based classification task, the desired dictionary should have both representational ability and discriminative power. Since separating the classification training from dictionary learning may cause the learned dictionary to be sub-optimal for the classification task, we propose to jointly learn a dictionary and classification model. In other words, in contrast with most existing schemes that attempt to update the dictionary and classifier parameters alternately by iteratively solving each sub-problem, we propose to solve them simultaneously. This joint dictionary learning and classifier training can be expected to result in a dictionary that is both reconstructive and discriminative for a robust recognition system. To the best of our knowledge, this is the only work that attempts to simultaneously learn the sparse representation of the signal and train a nonlinear classifier to be discriminative for sparse codes. The proposed method jointly learns a single dictionary and also an optimal nonlinear classifier. We have performed extensive experiments on both acted and spontaneous emotion databases to evaluate the effectiveness of the proposed feature extraction and classification schemes under different scenarios. Our results clearly demonstrate the robustness of the proposed emotion recognition framework, especially in challenging scenarios that involve illumination changes, occlusion, and head pose variations. DOCTOR OF PHILOSOPHY (EEE)

Gaussian process (GP) is a stochastic process that has been studied for a long time and gained wide interests in the machine learning community in recent years. In this thesis, several interesting pattern analysis problems are solved using Gaussian process. Gaussian process models can be interpreted in two views, the weight space view and the function space view. Their different interpretations can be helpful in applying GPs to solve real problems. Its capability as functional prior inspired the original work on learning nonparametric similarity measure in this thesis. The similarity between pairwise inputs is considered a smooth function described by a GP prior. Given known similarity constraints, the model can be tuned by maximum likelihood and similarity on new inputs can be inferred. The advantage is that the learned similarity measure does not assume any parametric form. It is flexible to model various data with arbitrary distributions, and handles noise and outliers in the data. Gaussian process latent variable model (GPLVM) is a data modeling tool that is derived based on Gaussian processes from the weight space view. It was originally proposed for visualization of high dimensional data. DOCTOR OF PHILOSOPHY (EEE)

Estimating human poses from 2D images or video sequences can provide the moving trajectories of the body joints for the high level processing, human activity recognition, which is applicable in surveillance, human-computer interaction and clinical and sport analysis. This project proposes a new statistical formulation called the data-driven Monte Carlo hidden Markov model to estimate human poses from random initializations. MASTER OF ENGINEERING (EEE)