Search

Facial expression recognition (FER) is of great interest to the current studies of human-computer interaction. In this paper, we propose a novel geometry-guided facial expression recognition framework, based on graph convolutional networks and transformers, to perform effective emotion recognition from videos. Specifically, we detect and utilize facial landmarks to construct a spatial-temporal graph, based on both the landmark coordinates and local appearance, for representing a facial expression sequence. The graph convolutional blocks and transformer modules are employed to produce high-semantic emotion-related representations from the structured facial graphs, which facilitate the framework to establish both the local and non-local dependency between the vertices. Moreover, spatial and temporal attention mechanisms are introduced into graph-based learning to promote FER reasoning, via the emphasis on the most informative facial components and frames. Extensive experiments demonstrate that the proposed framework achieves promising performance for geometry-based FER and shows great generalization and robustness in real-world applications. [ABSTRACT FROM AUTHOR]

Learning discriminative representations with good robustness from facial observations serves as a fundamental step towards intelligent facial expression recognition (FER). In this article, we propose a novel geometry-aware FER framework to boost the FER performance based on both the geometric and appearance knowledge. Specifically, we propose an encoding strategy for facial landmarks, and adopt a graph convolutional network (GCN) to fully explore the structural information of the facial components behind different expressions. A convolutional neural network (CNN) is further applied to the whole facial observation to learn the global characteristics of different expressions. The features from these two networks are fused into a comprehensive high-semantic representation, which promotes the FER reasoning from both visual and structural perspectives. Moreover, to facilitate the networks to concentrate on the most informative facial regions and components, we introduce multi-level attention mechanisms into the proposed framework, which enhance the reliability of the learned representations for effective FER. Experiments on two challenging FER benchmarks demonstrate that the attentive graph-based learning on the facial geometry boosts the FER accuracy. Furthermore, the insensitivity of the geometric information to the appearance variations also improves the generalization of the proposed framework. [ABSTRACT FROM AUTHOR]

This paper presents a new method, named implicit and explicit feature purification (IEFP), for age-invariant face recognition. Facial features extracted from a face image contain the information about the identity, age, and other attributes. For age-invariant face recognition, it is important to remove the irrelevant information, and retain the identity information only, in the facial features. Through the two proposed feature purification mechanisms, our framework can produce facial-feature embeddings that preserve identity information as much as possible and are insensitive to age variations. Specifically, on the one hand, a special network module is devised to implicitly purify the original facial features obtained from a face encoder. On the other hand, to obtain purer facial feature representations for age-invariant face recognition, irrelevant information within the implicitly purified features, such as the age, is further removed. This is realized by using a regularizer, based on information theory, to explicitly minimize the correlation between identity-related features and age-related features. Comprehensive ablation studies show that these two feature purification schemes can work independently, as well as collaboratively, to achieve better performance. Extensive evaluations on several benchmark data sets show that the IEFP method is on par with those competitors learned on far more favorable training samples, and it achieves the best performance in a fair comparison. Furthermore, we provide mathematical interpretation to explain the effectiveness of our approach, and find that it tends to generate low-rank, yet high-dimensional, representations for age-invariant face recognition. [ABSTRACT FROM AUTHOR]

While the relationship between facial expressions and emotion has been a productive area of inquiry, research is only recently exploring whether a link exists between facial expressions and cognitive processes. Using findings from psychology and neuroscience to guide predictions of affectation during a cognitive task, this article aimed to study facial dynamics as a mean to understand comprehension. We present a new multimodal facial expression database, named Facial Expressions of Comprehension (FEC), consisting of the videos recorded during a computer-mediated task in which each trial consisted of reading, answering, and feedback to general knowledge true and false statements. To identify the level of engagement with the corresponding stimuli, we present a new methodology using animation units (AnUs) from the Kinect v2 device to explore the changes in facial configuration caused by an event: Event-Related Intensities (ERIs). To identify dynamic facial configurations, we used ERIs in statistical analyses with generalized additive models. To identify differential facial dynamics linked to knowing vs. guessing and true vs. false responses, we employed an SVM classifier with facial appearance information extracted using LPQ-TOP. Results of ERIs in sentence comprehension show that facial dynamics are promising to help understand affective and cognitive states of the mind. [ABSTRACT FROM AUTHOR]

Face recognition across age progression remains one of the area's most challenging tasks, as the aging process affects both the shape and texture of a face. One possible solution is to apply a probabilistic model to represent a face simultaneously with its identity variable, which is stable through time, and its aging variable, which changes with time. However, as the aging process varies for different people, a person may look younger or older than another person, even though their ages are the same. Consequently, using the ‘real’ age labels given by existing face datasets for age-invariant face recognition will inevitably introduce ambiguity to learning algorithms. In this paper, an identity-inference model, based on age-subspace learning from appearance-age labels, is proposed. We first model human identity and aging variables simultaneously using Probabilistic Linear Discriminant Analysis (PLDA). Then, the aging subspace is learnt independently with the appearance-age labels, and the identity subspace is then determined iteratively with the Expectation-Maximization (EM) algorithm. We found that the learned aging subspace is insensitive to the training face images used, and is independent of the identity model. Consequently, the recognition of aging faces becomes simpler as identity inference no longer needs to consider age labels. Furthermore, in our algorithm, different identity features learnt from the identity model are further combined using Canonical Correlation Analysis (CCA), where their correlations are maximized for face recognition. A thorough experimental analysis of face recognition is performed on three public domain face-aging datasets: FGNET, MORPH, and CACD. Experiment results show that the proposed framework can achieve a comparable, or even better, performance against other state-of-the-art methods, especially when the age range is large. [ABSTRACT FROM AUTHOR]

In video surveillance, the captured face images are usually of low resolution (LR). Thus, a framework based on singular value decomposition (SVD) for performing both face hallucination and recognition simultaneously is proposed in this paper. Conventionally, LR face recognition is carried out by super-resolving the LR input face first, and then performing face recognition to identify the input face. By considering face hallucination and recognition simultaneously, the accuracy of both the hallucination and the recognition can be improved. In this paper, singular values are first proved to be effective for representing face images, and the singular values of a face image at different resolutions have approximately a linear relation. In our algorithm, each face image is represented using SVD. For each LR input face, the corresponding LR and high-resolution (HR) face-image pairs can then be selected from the face gallery. Based on these selected LR–HR pairs, the mapping functions for interpolating the two matrices in the SVD representation for the reconstruction of HR face images can be learned more accurately. Therefore, the final estimation of the high-frequency details of the HR face images will become more reliable and effective. The experimental results demonstrate that our proposed framework can achieve promising results for both face hallucination and recognition. [ABSTRACT FROM AUTHOR]

Face recognition methods, which usually represent face images using holistic or local facial features, rely heavily on alignment. Their performances also suffer a severe degradation under variations in expressions or poses, especially when there is one gallery per subject only. With the easy access to high-resolution (HR) face images nowadays, some HR face databases have recently been developed. However, few studies have tackled the use of HR information for face recognition or verification. In this paper, we propose a pose-invariant face-verification method, which is robust to alignment errors, using the HR information based on pore-scale facial features. A new keypoint descriptor, namely, pore-Principal Component Analysis (PCA)-Scale Invariant Feature Transform (PPCASIFT)—adapted from PCA-SIFT—is devised for the extraction of a compact set of distinctive pore-scale facial features. Having matched the pore-scale features of two-face regions, an effective robust-fitting scheme is proposed for the face-verification task. Experiments show that, with one frontal-view gallery only per subject, our proposed method outperforms a number of standard verification methods, and can achieve excellent accuracy even the faces are under large variations in expression and pose. [ABSTRACT FROM AUTHOR]

Establishing correct correspondences between two faces with different viewpoints has played an important role in 3D face reconstruction and other computer-vision applications. Usually, face images are considered to lack sufficient distinctive features to establish a large number of correspondences on uncalibrated images. In this paper, we investigate pore-scale facial features, which are formed from pores, fine wrinkles, and hair. These features have many characteristics that make them suitable for matching facial images under different variations. Using both biological observation and computer-vision consideration, a new framework is devised for pore-scale facial-feature extraction and matching. The matching difficulty under various skin appearances of different subjects and imaging distortion is also analyzed. For further improving the matching performance and tackling distortions such as varying illuminations and unfocused blurring, a pore-to-pore correspondences dataset is established for training a more distinctive and compact descriptor. Experiments are conducted on a face database containing 105 subjects, and the results prove that the pore-scale features are highly distinctive; face images with a minimum resolution of 600×700 (0.4 mega) pixels contain sufficient details to perform a reliable matching in different poses. Generally, our algorithm can establish between 500 and 2000 correct correspondences on a pair of uncalibrated face images of the same person. Furthermore, the proposed methods can be applied to face recognition, 3D reconstruction, etc. [ABSTRACT FROM AUTHOR]

In this paper, we propose an efficient algorithm to reconstruct the 3D structure of a human face from one or more of its 2D images with different poses. In our algorithm, the nonlinear least-squares model is first employed to estimate the depth values of facial feature points and the pose of the 2D face image concerned by means of the similarity transform. Furthermore, different optimization schemes are presented with regard to the accuracy levels and the training time required. Our algorithm also embeds the symmetrical property of the human face into the optimization procedure, in order to alleviate the sensitivities arising from changes in pose. In addition, the regularization term, based on linear correlation, is added in the objective function to improve the estimation accuracy of the 3D structure. Further, a model-integration method is proposed to improve the depth-estimation accuracy when multiple nonfrontal-view face images are available. Experimental results on the 2D and 3D databases demonstrate the feasibility and efficiency of the proposed methods. [ABSTRACT FROM PUBLISHER]

Abstract: In this paper, an efficient method for human facial expression recognition is presented. We first propose a representation model for facial expressions, namely the spatially maximum occurrence model (SMOM), which is based on the statistical characteristics of training facial images and has a powerful representation capability. Then the elastic shape–texture matching (ESTM) algorithm is used to measure the similarity between images based on the shape and texture information. By combining SMOM and ESTM, the algorithm, namely SMOM–ESTM, can achieve a higher recognition performance level. The recognition rates of the SMOM–ESTM algorithm based on the AR database and the Yale database are 94.5% and 94.7%, respectively. [Copyright &y& Elsevier]

Abstract: In this paper, a new algorithm is proposed to derive the 3D structure of a human face from a group of face images under different poses. Based on the corresponding 2D feature points of the respective images, their respective poses and the depths of the feature points can be estimated based on measurements using the similarity transform. To accurately estimate the pose of and the 3D information about a human face, the genetic algorithm (GA) is applied. Our algorithm does not require any prior knowledge of camera calibration, and has no limitation on the possible poses or the scale of the face images. It also provides a means to evaluate the accuracy of the constructed 3D face model based on the similarity transform of the 2D feature point sets. Our approach can also be extended to face recognition to alleviate the effect of pose variations. Experimental results show that our proposed algorithm can construct a 3D face structure reliably and efficiently. [Copyright &y& Elsevier]

Abstract: In this paper, we propose a new face recognition algorithm based on a single frontal-view image for each face subject, which considers the effect of the face manifold structure. To compare two near-frontal face images, each face is considered a combination of a sequence of local image blocks. Each of the image blocks of one image can be reconstructed according to the corresponding local image block of the other face image. Then an elastic local reconstruction (ELR) method is proposed to measure the similarities between the image block pairs in order to measure the difference between the two face images. Our algorithm not only benefits from the face manifold structure, in terms of being robust to various image variations, but also is computationally simple because there is no need to build the face manifold. We evaluate the performance of our proposed face recognition algorithm with the use of different databases, which are produced under various conditions, e.g. lightings, expressions, perspectives, with/without glasses and occlusions. Consistent and promising experimental results were obtained, which show that our algorithm can greatly improve the recognition rates under all the different conditions. [Copyright &y& Elsevier]

Abstract: In this paper, a novel, elastic, shape-texture matching method, namely ESTM, for human face recognition is proposed. In our approach, both the shape and the texture information are used to compare two faces without establishing any precise pixel-wise correspondence. The edge map is used to represent the shape of an image, while the texture information is characterized by both the Gabor representations and the gradient direction of each pixel. Combining these features, a shape-texture Hausdorff distance is devised to compute the similarity of two face images. The elastic matching is robust to small, local distortions of the feature points such as those caused by facial expression variations. In addition, the use of the edge map, Gabor representations and the direction of the image gradient can all alleviate the effect of illumination to a certain extent. With different databases, experimental results show that our algorithm can always achieve a better performance than other face recognition algorithms under different conditions, except when an image is under poor and uneven illumination. Experiments based on the Yale database, AR database, ORL database and YaleB database show that our proposed method can achieve recognition rates of 88.7%, 97.7%, 78.3% and 89.5%, respectively. [Copyright &y& Elsevier]

Abstract: In this paper, an efficient representation method insensitive to varying illumination is proposed for human face recognition. Theoretical analysis based on the human face model and the illumination model shows that the effects of varying lighting on a human face image can be modeled by a sequence of multiplicative and additive noises. Instead of computing these noises, which is very difficult for real applications, we aim to reduce or even remove their effect. In our method, a local normalization technique is applied to an image, which can effectively and efficiently eliminate the effect of uneven illuminations while keeping the local statistical properties of the processed image the same as in the corresponding image under normal lighting condition. After processing, the image under varying illumination will have similar pixel values to the corresponding image that is under normal lighting condition. Then, the processed images are used for face recognition. The proposed algorithm has been evaluated based on the Yale database, the AR database, the PIE database, the YaleB database and the combined database by using different face recognition methods such as PCA, ICA and Gabor wavelets. Consistent and promising results were obtained, which show that our method can effectively eliminate the effect of uneven illumination and greatly improve the recognition results. [Copyright &y& Elsevier]

This paper proposes a novel illumination compensation algorithm, which can compensate for the uneven illuminations on human faces and reconstruct face images in normal lighting conditions. A simple yet effective local contrast enhancement method, namely block-based histogram equalization (BHE), is first proposed. The resulting image processed using BHE is then compared with the original face image processed using histogram equalization (HE) to estimate the category of its light source. In our scheme, we divide the light source for a human face into 65 categories. Based on the category identified, a corresponding lighting compensation model is used to reconstruct an image that will visually be under normal illumination. In order to eliminate the influence of uneven illumination while retaining the shape information about a human face, a 2D face shape model is used. Experimental results show that, with the use of principal component analysis for face recognition, the recognition rate can be improved by 53.3% to 62.6% when our proposed algorithm for lighting compensation is used. [Copyright &y& Elsevier]

Hausdorff distance is an efficient measure of the similarity of two point sets. In this paper, we propose a new spatially weighted Hausdorff distance measure for human face recognition. The weighting function used in the Hausdorff distance measure is based on an eigenface, which has a large value at locations of importance facial features and can reflect the face structure more effectively. Two modified Hausdorff distances, namely, “spatially eigen-weighted Hausdorff distance” (SEWHD) and “spatially eigen-weighted ‘doubly’ Hausdorff distance” (SEW2HD) are proposed, which incorporate the information about the location of important facial features such as eyes, mouth, and face contour so that distances at those regions will be emphasized. Experimental results based on a combination of the ORL, MIT, and Yale face databases show that SEW2HD can achieve recognition rates of 83%, 90% and 92% for the first one, the first three and the first five likely matched faces, respectively, while the corresponding recognition rates of SEWHD are 80%, 83% and 88%, respectively. [Copyright &y& Elsevier]

The edge map of a facial image contains abundant information about its shape and structure, which is useful for face recognition. To compare edge images, Hausdorff distance is an efficient measure that can determine the degree of their resemblance, and does not require a knowledge of correspondence among those points in the two edge maps. In this paper, a new modified Hausdorff distance measure is proposed, which has a better discriminant power. As different facial regions have different degrees of significance for face recognition, a new modified Hausdorff distance is proposed which is weighted according to a weighted function derived from the spatial information of the human face; hence crucial regions are emphasized for face identification. Experimental results show that the distance measure can achieve recognition rates of 80%, 87%, and 91% for the first, the first five, and the first seven likely matched faces, respectively. [Copyright &y& Elsevier]

• A novel method for Low-resolution (LR) face recognition is proposed. • Features are decomposed into a low-rank feature matrix and a corresponding sparse error matrix. • Learned low-rank features are projected onto a sparse feature space for recognition. • Experiments on five face datasets shows the effectiveness of our proposed method. In this paper, we propose a novel approach for low-resolution face recognition, under uncontrolled settings. Our approach first decomposes a multiple of extracted local features into a set of representative basis (low-rank matrix) and sparse error matrix, and then learns a projection matrix based on our proposed sparse-coding-based algorithm, which preserves the sparse structure of the learned low-rank features, in a low-dimensional feature subspace. Then, a coefficient vector, based on linear regression, is computed to determine the similarity between the projected gallery and query image's features. Furthermore, a new morphological pre-processing approach is proposed to improve the visual quality of images. Our experiments were conducted on five available face-recognition datasets, which contain images with variations in pose, facial expressions and illumination conditions. Experiment results show that our method outperforms other state-of–the-art low-resolution face recognition methods in terms of recognition accuracy. [ABSTRACT FROM AUTHOR]