Search

Achieving the upper limits of face identification accuracy in forensic applications can minimize errors that have profound social and personal consequences. Although forensic examiners identify faces in these applications, systematic tests of their accuracy are rare. How can we achieve the most accurate face identification: using people and/or machines working alone or in collaboration? In a comprehensive comparison of face identification by humans and computers, we found that forensic facial examiners, facial reviewers, and superrecognizers were more accurate than fingerprint examiners and students on a challenging face identification test. Individual performance on the test varied widely. On the same test, four deep convolutional neural networks (DCNNs), developed between 2015 and 2017, identified faces within the range of human accuracy. Accuracy of the algorithms increased steadily over time, with the most recent DCNN scoring above the median of the forensic facial examiners. Using crowd-sourcing methods, we fused the judgments of multiple forensic facial examiners by averaging their rating-based identity judgments. Accuracy was substantially better for fused judgments than for individuals working alone. Fusion also served to stabilize performance, boosting the scores of lower-performing individuals and decreasing variability. Single forensic facial examiners fused with the best algorithm were more accurate than the combination of two examiners. Therefore, collaboration among humans and between humans and machines offers tangible benefits to face identification accuracy in important applications. These results offer an evidence-based roadmap for achieving the most accurate face identification possible.

The term soft biometrics typically refers to attributes of people such as their gender, the shape of their head or the color of their hair. There is growing interest in soft biometrics as a means of improving automated face recognition since they hold the promise of significantly reducing recognition errors, in part by ruling out illogical choices. This paper concentrates specifically on soft biometrics as opposed to extended attributes, and presents the results from three experiments quantifying performance gains on a difficult face recognition task when standard face recognition algorithms are augmented using soft biometrics. These experiments include (1) a best-case analysis using perfect knowledge of gender and race, (2) support vector machine-based soft biometric classifiers and (3) face shape expressed through an active shape model. All three experiments indicate small improvements may be made when soft biometrics augment an existing algorithm. However, in all cases, the gains were modest. One reason is that false matches are more likely between faces of people sharing the same soft biometric traits. This is to be expected, since face recognition algorithms utilize appearance information, which is the same information used by algorithms designed to assign soft biometric labels to face images.

While face recognition algorithms perform under many different unconstrained conditions, predicting this performance is not possible when a new location is introduced. Analyzing the impostor distribution of the videos of the Point-and-Shoot Challenge (PaSC) as well as its relationship to the genuine match distribution, we present a method for predicting the performance of an algorithm using only unlabeled data for a new location.

Deep convolutional neural networks (DCNN) based algorithm methods have swept face-recognition. DCNNbased algorithms have shown significant improvements in accuracy on the Labeled Faces in the Wild (LFW) and the YouTube1 Video face-recognition benchmarks. These two benchmarks consist of images and videos of celebrities downloaded from the World Wide Web. Since 2004, the National Institute of Standards and Technology (NIST) has established a series of face-recognition benchmarks that span a range of scenarios and difficulties. The scenarios range from comparing frontal faces taken in studio lighting to comparing faces acquired with cell phone cameras taken outdoors. The VGG-face algorithm [7] was ran on eight NIST face-recognition benchmarks. The Vision Geometry Group (VGG)-face algorithm excelled on the most difficult benchmarks; existing algorithms excelled the benchmarks with higher quality images. This finding is consistent with the design of the algorithms. The VGG-face algorithm was designed to recognize faces in variable illumination; the existing algorithms were designed to operate on face-images taken in controlled illuminations. To accurately characterize the performance of face recognition algorithms, we recommend that performance is reported on multiple benchmarks.

The past few years have seen impressive gains inlong standing and difficult problems in face recognition. Thesegains have come about through the use of deep learning algorithmsthat consist of multi-layered neural networks. In part,the success of these algorithms is due to the easy availability ofextremely large datasets of faces that are annotated and labelledby humans. The reliance on crowd-sourced data for machinelearning and algorithm evaluation raises methodological issuesthat are not widely appreciated in computer vision. Several ofthese issues have come to light in recent work using crowdsourcing to benchmark human face identification on largedatabases that are used to test face recognition algorithms. Wedefine and discuss these issues using face recognition as a casestudy. We focus on: a.) the characteristics of the human participants;b.) the difference between aggregate and fused measuresof human accuracy; and c.) the lack of standard methods forcontrolling critical characteristics of the “imposter” distributionin large and variably diverse data sets. We will show thatestimates of human accuracy can vary widely depending on howthese factors combine in any given evaluation.We conclude withrecommendations on best practices in mitigating this variabilityand arriving at stable estimates of ground truth acquired bycrowd-sourcing.

Since 2005, human and computer performance has been systematically compared as part of face recognition competitions, with results being reported for both still and video imagery. The key results from these competitions are reviewed. To analyze performance across studies, the cross-modal performance analysis (CMPA) framework is introduced. The CMPA framework is applied to experiments that were part of face a recognition competition. The analysis shows that for matching frontal faces in still images, algorithms are consistently superior to humans. For video and difficult still face pairs, humans are superior. Finally, based on the CMPA framework and a face performance index, we outline a challenge problem for developing algorithms that are superior to humans for the general face recognition problem.

Face recognition involves at least three major concepts from statistics: dimension reduction, feature extraction, and prediction. A selective review of algorithms, from seminal to state-of-the-art, explores how these concepts persist as organizing principles in the field. Algorithms based directly upon classical statistical techniques include linear methods like principal component analysis and linear discriminant analysis. Nonlinear manifold methods, such as Laplacianfaces and Stiefel quotients, offer considerable performance improvements. Other noteworthy ideas include three-dimensional morphable models, methods using local regions and/or alternative feature spaces (e.g., elastic bunch graph matching and local binary patterns) and sparse representation approaches. Opportunities for innovative statistical and collaborative research in face recognition are expanding in tandem with the growing complexity and diversity of applications. WIREs Comput Stat 2013, 5:288–308. doi: 10.1002/wics.1262 Conflict of interest: The authors have declared no conflicts of interest for this article. For further resources related to this article, please visit the WIREs website

The purpose of this paper is to introduce an effective and structured methodology for carrying out a biometric system sensitivity analysis. The goal of sensitivity analysis is to provide the researcher/developer with insight and understanding of the key factors-algorithmic, subject-based, procedural, image quality, environmental, among others-that affect the matching performance of the biometric system under study. This proposed methodology consists of two steps: (1) the design and execution of orthogonal fractional factorial experiment designs which allow the scientist to efficiently investigate the effect of a large number of factors-and interactions-simultaneously, and (2) the use of a select set of statistical data analysis graphical procedures which are fine-tuned to unambiguously highlight important factors, important interactions, and locally-optimal settings. We illustrate this methodology by application to a study of VASIR (Video-based Automated System for Iris Recognition)-NIST iris-based biometric system. In particular, we investigated k=8 algorithmic factors from the VASIR system by constructing a (2^6^-^1x3^1x4^1) orthogonal fractional factorial design, generating the corresponding performance data, and applying an appropriate set of analysis graphics to determine the relative importance of the eight factors, the relative importance of the 28 two-term interactions, and the local best settings of the eight algorithms. The results showed that VASIR's performance was primarily driven by six factors out of the eight, along with four two-term interactions. A virtue of our two-step methodology is that it is systematic and general, and hence may be applied with equal rigor and effectiveness to other biometric systems, such as fingerprints, face, voice, and DNA.

In recent years, sparse representation and dictionary-learning-based methods have emerged as powerful tools for efficiently processing data in nontraditional ways. A particular area of promise for these theories is face recognition. In this paper, we review the role of sparse representation and dictionary learning for efficient face identification and verification. Recent face recognition algorithms from still images, videos, and ambiguously labeled imagery are reviewed. In particular, discriminative dictionary learning algorithms as well as methods based on weakly supervised learning and domain adaptation are summarized. Some of the compelling challenges and issues that confront research in face recognition using sparse representations and dictionary learning are outlined.

This paper identifies important factors for face recognition algorithm performance in video. The goal of this study is to understand key factors that affect algorithm performance and to characterize the algorithm performance. We evaluate four factor metrics for a single video as well as two comparative metrics for pairs of videos. This study carried out an investigation of the effect of nine factors on three algorithms using the Point-and-Shoot Challenge (PaSC) video dataset. These factors can be categorized into three groups: 1) image/video (pose yaw, pose roll, face size, and face detection confidence); 2) environment (environmental condition with person’s activity and sensor model); and 3) subject (subject ID, gender, and race). For videobased face recognition, the analysis shows that the distribution-based methods were generally more effective in quantifying factor values. For predicting face recognition performance in a video, we observed that face detection confidence and face size serve as potentially useful quality measure metrics. We also find that male faces are easier to identify than female faces, and Asians are easier than Caucasians. Further, on the PaSC video dataset, the performance of face recognition algorithms are primarily driven by environment and sensor factors.

Inexpensive “point-and-shoot” camera technology has combined with social network technology to give the general population a motivation to use face recognition technology. Users expect a lot; they want to snap pictures, shoot videos, upload, and have their friends, family and acquaintances more-or-less automatically recognized. Despite the apparent simplicity of the problem, face recognition in this context is hard. Roughly speaking, failure rates in the 4 to 8 out of 10 range are common. In contrast, error rates drop to roughly 1 in 1,000 for well controlled imagery. To spur advancement in face and person recognition this paper introduces the Point-and-Shoot Face Recognition Challenge (PaSC). The challenge includes 9,376 still images of 293 people balanced with respect to distance to the camera, alternative sensors, frontal versus not-frontal views, and varying location. There are also 2,802 videos for 265 people: a subset of the 293. Verification results are presented for public baseline algorithms and a commercial algorithm for three cases: comparing still images to still images, videos to videos, and still images to videos.

We investigate the existence of quality measures for face recognition. First, we introduce the concept of an oracle for image quality in the context of face recognition. Next we introduce greedy pruned ordering (GPO) as an approximation to an image quality oracle. GPO analysis provides an estimated upper bound for quality measures, given a face recognition algorithm and data set. We then assess the performance of 12 commonly proposed face image quality measures against this standard. In addition, we investigate the potential for learning new quality measures via supervised learning. Finally, we show that GPO analysis is applicable to other biometrics.

We examine the robustness of algorithm performance over multiple datasets collected with different sensors. This study provides insight as to whether an algorithm performance derived from traditional controlled environment studies will robustly extrapolate to more challenging stand-off/real-world environments. We argue that a systematic methodology is critical in assuring the validity of algorithmic conclusions over the broader arena of applications. We present a structured evaluation protocol and demonstrate its utility by comparing the performance of an open-source algorithm over seven diverse datasets, spanning six different sensors (three stationary, one handheld, and two stand-off). We also provide baseline results for the ranking of the seven datasets as measured by four performance metrics. Finally, we compare our protocol-based ranking with a parallel ranking based on an independent survey of biometrics experts, with high correlation between the two rankings being demonstrated.

We address the problem of automated face recognition on a social network using a loopy belief propagation framework. The proposed approach propagates the identities of faces in photos across social graphs. We characterize performance in terms of structural properties of a social network. This is accomplished by conducting extensive simulations on synthetic networks. We propose a distance metric defined using face recognition results for detecting hidden connections. The result demonstrates that the constraints imposed by the social network have the potential to improve the performance of face recognition methods. The result also shows it is possible to discover hidden connections in a social network based on face recognition.

Face recognition has made significant advances over the last twenty years. State-of-the-art algorithms push the performance envelope to near perfect recognition rates on many face databases. Recently, the Good, the Bad, and the Ugly (GBU) face challenge problem has been introduced to focus on hard aspects of face recognition from still frontal images. In this paper, we introduce the CohortLDA baseline algorithm, which is an Linear Discriminant Analysis (LDA) algorithm with color spaces and cohort normalization. CohortLDA greatly outperforms some well known face recognition algorithms on the GBU challenge problem. The GBU protocol includes rules for creating training sets. We investigate the effect on performance of violating the rules for creating training sets. This analysis shows that violating the GBU protocol can substantially over estimate performance on the GBU challenge problem.

Discriminative appearance features are effective for recognizing actions in a fixed view, but generalize poorly to changes in viewpoint. We present a method for viewinvariant action recognition based on sparse representations using a transferable dictionary pair. A transferable dictionary pair consists of two dictionaries that correspond to the source and target views respectively. The two dictionaries are learned simultaneously from pairs of videos taken at different views and aim to encourage each video in the pair to have the same sparse representation. Thus, the transferable dictionary pair links features between the two views that are useful for action recognition. Both unsupervised and supervised algorithms are presented for learning transferable dictionary pairs. Using the sparse representation as features, a classifier built in the source view can be directly transferred to the target view. We extend our approach to transferring an action model learned from multiple source views to one target view. We demonstrate the effectiveness of our approach on the multi-view IXMAS data set. Our results compare favorably to the the state of the art.

Several studies have shown the existence of biometric zoos. The premise is that in biometric systems people fall into distinct categories, labeled with animal names, indicating recognition difficulty. Different combinations of excessive false accepts or rejects correspond to labels such as: Goat, Lamb, Wolf, etc. Previous work on biometric zoos has investigated the existence of zoos for the results of an algorithm on a data set. This work investigates biometric zoos generalization across algorithms and data sets. For example, if a subject is a Goat for algorithm A on data set X, is that subject also a Goat for algorithm B on data set Y? This paper introduces a theoretical framework for generalizing biometric zoos. Based on our framework, we develop an experimental methodology for determining if biometric zoos generalize across algorithms and data sets, and we conduct a series of experiments to investigate the existence of zoos on two algorithms in FRVT 2006.

Modern vision systems are often a heterogeneous collection of image processing, machine learning, and pattern recognition techniques. One problem with these systems is finding their optimal parameter settings, since these systems often have many interacting parameters. This paper proposes the use of a Genetic Algorithm (GA) to automatically search parameter space. The technique is tested on a publicly available face recognition algorithm and dataset. In the work presented, the GA takes the role of a person configuring the algorithm by repeatedly observing performance on a tuning-subset of the final evaluation test data. In this context, the GA is shown to do a better job of configuring the algorithm than was achieved by the authors who originally constructed and released the LRPCA baseline. In addition, the data generated during the search is used to construct statistical models of the fitness landscape which provides insight into the significance from, and relations among, algorithm parameters.

Recent work indicates that state-of-the-art face recognition algorithms can surpass humans matching identity in pairs of face images taken under different illumination conditions. It has been demonstrated further that fusing algorithm- and human-derived face similarity estimates cuts error rates substantially over the performance of the best algorithms. Here we employed a pattern-based classification procedure to fuse individual human subjects and algorithms with the goal of determining whether strategy differences among humans are strong enough to suggest particular man-machine combinations. The results showed that error rates for the pairwise man-machine fusions were reduced an average of 47 percent when compared to the performance of the algorithms individually. The performance of the best pairwise combinations of individual humans and algorithms was only slightly less accurate than the combination of individual humans with all seven algorithms. The balance of man and machine contributions to the pairwise fusions varied widely, indicating that a one-size-fits-all weighting of human and machine face recognition estimates is not appropriate.

With the mass deployment of cameras, concern has risen about protecting a person's privacy as he goes about his daily life. Many of the cameras are installed to perform surveillance tasks that do not require the identity of a person. In the context of surveillance applications, we examine the trade-off between privacy and security. The trade-off is accomplished by looking at quantitative measures of privacy and surveillance performance. To provide privacy protection we examine the effect on surveillance performance of a parametric family of privacy function. A privacy function degrades images to make identification more difficult. By varying the parameter, different levels of privacy protection are provided. We introduce the privacy operating characteristic (POC) to quantitatively show the resulting trade-off between privacy and security. From a POC, policy makers can select the appropriate operating point for surveillance systems with regard to privacy.

We present the methodology and recognition performance characteristics used in the Face Recognition Vendor Test 2002. We refine the notion of a biometric imposter, and show that the traditional measures of identification and verification performance, are limiting cases of the open-universe watch list task. The watch list problem generalizes the tradeoff of detection and identification of persons of interest against a false alarm rate. In addition, we use performance scores on disjoint populations to establish a means of computing and displaying distribution-free estimates of the variation of verification vs. false alarm performance. Finally we formalize gallery normalization, which is an extension of previous evaluation methodologies; we define a pair of gallery dependent mappings that can be applied as a post recognition step to vectors of distance or similarity scores. All the methods are biometric non-specific, and applicable to large populations.

The NIST HumanID Evaluation Framework, or HEF, is an effort to design, implement, and deploy standards for the robust and complete documentation of the biometric system evaluation process. The HEF leverages contemporary technologies, specifically XML, for the formal description of biometric tests. The HEF was used to facilitate the administration of the Face Recognition Vendor Test (FRVT) 2002. Unlike FRVT 2000 or the FERET 1996 evaluations, FRVT 2002 used a large number (over 100,000) of both still and video facial imagery, warranting the development of a more sophisticated and regular means of describing data presented to the participants. The HEF is one component in NIST's ongoing effort to address the need in the biometrics community for a common evaluation framework.

Algorithms based on principal component analysis (PCA) form the basis of numerous studies in the psychological and algorithmic face-recognition literature. PCA is a statistical technique and its incorporation into a face-recognition algorithm requires numerous design decisions. We explicitly state the design decisions by introducing a generic modular PCA-algorithm. This allows us to investigate these decisions, including those not documented in the literature. We experimented with different implementations of each module, and evaluated the different implementations using the September 1996 FERET evaluation protocol (the de facto standard for evaluating face-recognition algorithms). We experimented with (i) changing the illumination normalization procedure; (ii) studying effects on algorithm performance of compressing images with JPEG and wavelet compression algorithms; (iii) varying the number of eigenvectors in the representation; and (iv) changing the similarity measure in the classification process. We performed two experiments. In the first experiment, we obtained performance results on the standard September 1996 FERET large-gallery image sets. In the second experiment, we examined the variability in algorithm performance on different sets of facial images. The study was performed on 100 randomly generated image sets (galleries) of the same size. Our two most significant results are (i) changing the similarity measure produced the greatest change in performance, and (ii) that difference in performance of ±10% is needed to distinguish between algorithms.

Face recognition is a K class problem. where K is the number of known individuals; and support vector machines (SVMs) are a binary classification method. By reformulating the face recognition problem and reinterpreting the output of the SVM classifier. we developed a SVM -based face recognition algorithm. The face recognition problem is formulated as a problem in difference space. which models dissimilarities between two facial images. In difference space we formulate face recognition as a two class problem. The classes are: dissimilarities between faces of the same person. and dissimilarities between faces of different people. By modifying the interpretation of the decision surface generated by SVM. we generated a similarity metric between faces that is learned from examples of differences between faces. The SVM-based algorithm is compared with a principal component analysis (PCA) based algorithm on a difficult set of images from the FERET database. Performance was measured for both verification and identification scenarios. The identification performance for SVM is 77-78% versus 54% for PCA. For verification. the equal error rate is 7% for SVM and 13% for PCA.

Two of the most critical requirements in support of producing reliable face-recognition systems are a large database of facial images and a testing procedure to evaluate systems. The Face Recognition Technology (FERET) program has addressed both issues through the FERET database of facial images and the establishment of the FERET tests. In this paper, we report on the FERET database and the September 1996 FERET test. This test is the third in a series of supervised face-recognition test administered under the FERET program.