Automated analysis of human family relationship using facial features

There has been increasing interest on the use of video surveillance to analyze human biometrics, especially in places where the chance of occurring crime is high, such as in large cities. As an example, 2185 children are being reported missing daily in the United States. It has been rarely looked into belonging a person to the same family as a soft biometric modal. We term the ability to verify query images as family members “family (kinship) verification.” Several studies have been conducted on family photo albuming, but the problem of family verification has not yet been directly addressed and analyzed. The lack of comprehensive published family datasets is evidence that the matter has received very little attention to date. This thesis focuses on providing solutions for challenges of family verification as a recent application of computer vision. We propose frameworks to recognize members of the same family based on the facts and psychological studies on family members’ resemblance to each other. We also investigate the demanding case of missing/unknown member verification as a critical real-life application that will benefit from this research. In the first major section of this thesis, we study true assumptions behind the resemblance of family members to one another, and the challenges of family verification. We develop traditional face recognition approaches to facilitate family verification by extraction of facial resemblance among family members. We investigate the very recent family verification problem, define scenarios and collect a dataset of family albums. Our aim is to recognize if the query sample belongs to the same family. In the first stage, we investigate facial resemblance extraction among family members to perform family verification. Similarities of facial patches are utilized as multiple experts to recognize family members. Finally, our proposed inter-patch constraint-free algorithm of facial patches analysis determines the redundant patches among family members. On average, five facial patches (29% of the facial patches) are chosen to achieve the same accuracy obtained by using all patches for family verification. Consequently, the computational time is saved by 71% compared to utilizing all patches. The second major section of our thesis is focused on family verification in real-time systems to obtain more discriminative features with the least dimension. Firstly, we minimize the quantization error of the Local Binary Pattern (LBP) operator through the incorporation of uniformly-sampled thresholds for LBP (UTLBP). Up to 3.4% improvement was achieved for family verification compared to various features in the literature, while the feature dimension is considerably smaller than the well-known Haar-like features (60%). Moreover, the proposed operator is more robust than the conventional operators, as tested on the large CAS-PEAL dataset against background, illumination, aging, and accessories changes. It achieves up to 8% performance improvement depending on the CAS-PEAL probe set, compared to the best operator in the literature. Then, we propose a novel redundant feature removal algorithm based on the fact that there is less similarity among the faces of family members than that of an individual’s face samples. The proposed algorithms are then employed on the UTLBP feature operator to select the most informative set of thresholds to outperform the Haar-like features, with 20 times less feature dimension. The proposed approach converges 10 times faster than the state-of-the-art face recognition algorithms do. The third contribution of the thesis is to utilize from each member’s image segment resemblance to the entire family. Our proposed method is to incorporate the degree of resemblance of each individual member’s patch to perform family verification. Our analysis and the important consensuses of psychologists’ studies reveal that the facial resemblance between family members differs from member to member and is facial patch specific. We propose to estimate the amount of resemblance for each patch based on the prior information of the member. As the score level fusion is preferred in the literature due to its wealth of information and low data dimension, we embed the available a priori information in the score fusion rule. The proposed method outperforms the state-of-the-art score fusion rules in all scenarios by 14%. Moreover, the resemblance estimation slightly outperforms human observation in the designated survey. The proposed method could also improve object family verification on the selected Caltech-256 database by 8%. In summary, two automated frameworks for recognizing family members are proposed. The first framework utilizes UTLBP operator and the novel redundant feature removal provides fast and low computational algorithms suitable for real-time processing. The other framework of facial patch resemblance extraction among family members benefits from offline training with parallel computing capability. Finally, the enhancement of family verification can potentially solve critical applications of daily life to find missing children and unknown parents. DOCTOR OF PHILOSOPHY (EEE)