Search

Facial attractiveness prediction is an important research topic in the computer vision community. It not only contributes to the development of interdisciplinary research in psychology and sociology, but also provides fundamental technical support for applications like aesthetic medicine and social media. With the advances in 3D data acquisition and feature representation, this paper aims to investigate the facial attractiveness from deep learning and three‐dimensional perspectives. The 3D faces are first processed to unwrap the texture images and refine the raw meshes. The feature extraction networks for texture, point cloud, and mesh are then delicately designed, considering the characteristics of different types of data. A more discriminative face representation is derived by feature fusion for the final attractiveness prediction. During network training, the cyclical learning rate with an improved range test is introduced, so as to alleviate the difficulty in hyperparameter setting. Extensive experiments are conducted on a 3D FAP benchmark, where the results demonstrate the significance of deep feature fusion and enhanced learning rate in cooperatively facilitating the performance. Specifically, the fusion of texture image and point cloud achieves the best overall prediction, with PC, MAE, and RMSE of 0.7908, 0.4153, and 0.5231, respectively. [ABSTRACT FROM AUTHOR]

Copyright of Journal of Computer Engineering & Applications is the property of Beijing Journal of Computer Engineering & Applications Journal Co Ltd. and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

The field of fine-grained semantic segmentation for a person's face and head, which includes identifying facial parts and head components, has made significant progress in recent years. However, this task remains challenging due to the difficulty of considering ambiguous occlusions and large pose variations. To address these difficulties, we propose a new framework called Mask-FPAN. Our framework includes a de-occlusion module that learns to parse occluded faces in a semi-supervised manner, taking into account face landmark localization, face occlusion estimations, and detected head poses. Additionally, we improve the robustness of 2D face parsing by combining a 3D morphable face model with the UV GAN. We also introduce two new datasets, named FaceOccMask-HQ and CelebAMaskOcc-HQ, to aid in face parsing work. Our proposed Mask-FPAN framework successfully addresses the challenge of face parsing in the wild and achieves significant performance improvements, with a mIoU increase from 0.7353 to 0.9013 compared to the current state-of-the-art on challenging face datasets. [Display omitted] • Novel Mask-FPAN framework improves face parsing on challenging datasets. • Occ-Autoencoders enhance occlusion handling and enable semi-supervised learning. • 3D morphable model + UV GAN boost robustness, enhancing face parsing. • Introducing high-quality FaceOccMask-HQ and CelebAMaskOcc-HQ datasets. • Sparse relationships addressed via integrated techniques for improved parsing. [ABSTRACT FROM AUTHOR]

Dense vertex-to-vertex correspondence (i.e. registration) between 3D faces is a fundamental and challenging issue for 3D &2D face analysis. While the sparse landmarks are definite with anatomically ground-truth correspondence, the dense vertex correspondences on most facial regions are unknown. In this view, the current methods commonly result in reasonable but diverse solutions, which deviate from the optimum to the dense registration problem. In this paper, we revisit dense registration by a dimension-degraded problem, i.e. proportional segmentation of a line, and employ an iterative dividing and diffusing method to reach an optimum solution that is robust to different initializations. We formulate a local registration problem for dividing and a linear least-square problem for diffusing, with constraints on fixed features on a 3D facial surface. We further propose a multi-resolution algorithm to accelerate the computational process. The proposed method is linked to a novel local scaling metric, where we illustrate the physical significance as smooth adaptions for local cells of 3D facial shapes. Extensive experiments on public datasets demonstrate the effectiveness of the proposed method in various aspects. Generally, the proposed method leads to not only significantly better representations of 3D facial data, but also coherent local deformations with elegant grid architecture for fine-grained registrations. [ABSTRACT FROM AUTHOR]

We proposed a monocular image disentanglement framework based on a compositional model. Our model disentangles the input image into its constituent components of albedo, depth, deformation, pose, and illumination. Instead of relying on any handcrafted priors, we trained our deep neural network to understand the physical meaning of each element by mimicking real-world operations, allowing it to reconstruct images in a self-supervised manner. Our model, trained on multi-frame images of each subject, demonstrates a better understanding of the objects without requiring any supervision or strong model assumptions. We utilized a deformation-free canonical space to align multi-frame images in the same space. This approach enables the understanding of information from multi-frame images in the same space. Our experiments showed that our approach accurately disentangled the physical elements of deformable faces from images with wide variations found in the wild. [ABSTRACT FROM AUTHOR]

This article proposes a facial expression removal method to recover a 3D neutral face from a single 3D expressional or non-neutral face. We treat a 3D non-neutral face as the sum of its neutral one and the residual. This can be satisfied if the correspondence between 3D vertices of expressional faces and those of neutral faces is established. We propose a non-rigid deformation method to establish the correspondence between 3D faces. Then, according to algebra inequality, the minimization of a neutral face model can be replaced by the minimization of its upper bound, i.e., the errors of an expressional face model and a residual model. Thus, we co-optimize the representation errors of the latter two models and build the relationship between the representation coefficients of the two models. Given an expressional face as the input, its corresponding neutral face can be inferred by the associative representation parameters in these two models. In the testing stage, we use an iterative joint fitting scheme to obtain a more accurate recovery. Extensive experiments are conducted to evaluate our method. The results show that our method obtains considerably better performance than existing methods in terms of average root mean square errors and recognition rates, and also better visual effects. [ABSTRACT FROM AUTHOR]

The face attractiveness from 3D representation is still in the infancy stage. In this paper, we investigate the role of 3D face geometry in facial beauty perception. Driven by heuristic rules, the aesthetics-aware geometric features are extracted, including features directly extended from 2D task to the 3D space, i.e., the global shape and Euclidean ratios, as well as features exclusively designed for 3D face structure, i.e., the geodesic ratios and region-based curvatures. A benchmark dataset with face models and attractiveness ratings is developed for facial attractiveness computation. Experimental results demonstrate the significance of 3D geometric features in capturing the face geometry over their 2D counterparts, and the advantages of the exclusive features in characterizing facial attractiveness over the extended features. [ABSTRACT FROM AUTHOR]

We present a fully automatic system that can produce high-fidelity, photo-realistic three-dimensional (3D) digital human heads with a consumer RGB-D selfie camera. The system only needs the user to take a short selfie RGB-D video while rotating his/her head and can produce a high-quality head reconstruction in less than 30 s. Our main contribution is a new facial geometry modeling and reflectance synthesis procedure that significantly improves the state of the art. Specifically, given the input video a two-stage frame selection procedure is first employed to select a few high-quality frames for reconstruction. Then a differentiable renderer-based 3D Morphable Model (3DMM) fitting algorithm is applied to recover facial geometries from multiview RGB-D data, which takes advantages of a powerful 3DMM basis constructed with extensive data generation and perturbation. Our 3DMM has much larger expressive capacities than conventional 3DMM, allowing us to recover more accurate facial geometry using merely linear basis. For reflectance synthesis, we present a hybrid approach that combines parametric fitting and Convolutional Neural Networks (CNNs) to synthesize high-resolution albedo/normal maps with realistic hair/pore/wrinkle details. Results show that our system can produce faithful 3D digital human faces with extremely realistic details. The main code and the newly constructed 3DMM basis is publicly available. [ABSTRACT FROM AUTHOR]

Building an effective representation for 3D face geometry is essential for face analysis tasks, that is, landmark detection, face recognition and reconstruction. This paper proposes to use a Multi‐Channel Facial Shape (MCFS) representation that consists of depth, hand‐engineered feature and attention maps to construct a 3D facial descriptor. And, a multi‐channel adjustment mechanism, named filtered squeeze and reversed excitation (FSRE), is proposed to re‐organize MCFS data. To assign a suitable weight for each channel, FSRE is able to learn the importance of each layer automatically in the training phase. MCFS and FSRE blocks collaborate together effectively to build a robust 3D facial shape representation, which has an excellent discriminative ability. Extensive experimental results, testing on both high‐resolution and low‐resolution face datasets, show that facial features extracted by our framework outperform existing methods. This representation is stable against occlusions, data corruptions, expressions and pose variations. Also, unlike traditional methods of 3D face feature extraction, which always take minutes to create 3D features, our system can run in real time. [ABSTRACT FROM AUTHOR]

In the process of reconstructing a historical event such as a rock concert only from video, the reconstruction of faces and expressions of the musicians is obviously important. However, in the process of rebuilding appearance, because of the low quality of the video of the recorded concert, the result of the reconstruction may be far from the real appearance. In this paper, a robust 3D face reconstruction application is described that can be applied to a video recording. The application first uses DeblurGAN program to run anti-ambiguity calculation and removes the ambiguity in the concert video. Then, the super-resolution program is used to enlarge every frame of the concert video by four times, thus making every frame of the video clearer. Finally, the 3D faces are obtained after 3D reconstruction of the processed video frames via the 3DMM_CNN program. [ABSTRACT FROM AUTHOR]

Emotion recognition is becoming commercially popular due to the major role of analytics in various aspects of marketing and strategy management. Several papers have been proposed in emotion recognition. They are mainly classified in the past under 2D and 3D emotion recognition, out of which 2D emotion recognition has been more popular. Various aspects like facial posture, light intensity variations and sensor-independent recognition have been studied by different authors in the past. However, in reality, 3D emotion recognition has been found to be more efficient which has a broader area of use. In this paper, a 3D tracking plane with 2D feature points has enabled us to recognize emotions by statistical voting method from all planes having over threshold number of points in their respective contour area. The proposed technique's results are comparable to existing methods in terms of time, space complexity and accuracy improvement. [ABSTRACT FROM AUTHOR]

Recent methods for 2D facial landmark localization perform well on close‐to‐frontal faces, but 2D landmarks are insufficient to represent 3D structure of a facial shape. For applications that require better accuracy, such as facial motion capture and 3D shape recovery, 3DA‐2D (2D Projections of 3D Facial Annotations) is preferred. Inferring the 3D structure from a single image is an ill‐posed problem whose accuracy and robustness are not always guaranteed. This paper aims to solve accurate 2D facial landmark localization and the transformation between 2D and 3DA‐2D landmarks. One way to increase the accuracy is to input more precisely annotated facial images. The traditional cascaded regressions cannot effectively handle large or noisy training data sets. In this paper, we propose a Mini‐Batch Cascaded Regressions (MBCR) method that can iteratively train a robust model from a large data set. Benefiting from the incremental learning strategy and a small learning rate, MBCR is robust to noise in training data. We also propose a new Cross‐Dimension Annotations Conversion (CDAC) method to map facial landmarks from 2D to 3DA‐2D coordinates and vice versa. The experimental results showed that CDAC combined with MBCR outperforms the‐state‐of‐the‐art methods in 3DA‐2D facial landmark localization. Moreover, CDAC can run efficiently at up to 110 fps on a 3.4 GHz‐CPU workstation. Thus, CDAC provides a solution to transform existing 2D alignment methods into 3DA‐2D ones without slowing down the speed. Training and testing code as well as the data set can be downloaded from https://github.com/SWJTU‐3DVision/CDAC. [ABSTRACT FROM AUTHOR]

This study proposes a novel occlusions detection and restoration strategy. The aim is to success with 3D face recognition even when faces are partially occluded by external objects. The method, which relies on geometrical facial properties, is designed for managing two types of facial occlusions (eye and mouth occlusions due to hands). First occlusions are detected and (if present) classified, by considering their effects on the 3D points cloud. Then, the occluded regions are progressively removed, and finally, the non-occluded symmetrical regions are used to restore the missing information. After the restoration process, face recognition is performed relying on the restored facial information and on the localized landmarks. The landmarking methodology relies on derivatives and on 12 differential geometry descriptors. The discriminating features adopted for facial comparison include shape index histograms, Euclidean and geodetical distances between landmarks, facial curves, and nose volume. Obtained recognition rates, evaluated on the whole Bosphorus database and on our private dataset, ranging from 92.55 to 97.20% depending on the completeness of data. [ABSTRACT FROM AUTHOR]

Nowadays, facial mimicry studies have acquired a great importance in the clinical domain and 3D motion capture systems are becoming valid tools for analysing facial muscles movements, thanks to the remarkable developments achieved in the 1990s. However, the face analysis domain suffers from a lack of valid motion capture protocol, due to the complexity of the human face. Indeed, a framework for defining the optimal marker set layout does not exist yet and, up to date, researchers still use their traditional facial point sets with manually allocated markers. Therefore, the study proposes an automatic approach to compute a minimum optimized marker layout to be exploited in facial motion capture, able to simplify the marker allocation without decreasing the significance level. Specifically, the algorithm identifies the optimal facial marker layouts selecting the subsets of linear distances among markers that allow to automatically recognizing with the highest performances, through a k-nearest neighbours classification technique, the acted facial movements. The marker layouts are extracted from them. Various validation and testing phases have demonstrated the accuracy, robustness and usefulness of the custom approach. [ABSTRACT FROM AUTHOR]

Highlights • Person re-identification approach from depth cameras. • Cumulating 3D skeletal and 3D face models across temporal sequences. • Weak classification of 3D skeletal features. • 3D face reconstruction and refinement from depth frames. • Weighted combination of 3D skeletal and 3D face features. Graphical abstract Abstract Person re-identification is typically performed using 2D still images or videos, where photometric appearance is the main visual cue used to discover the presence of a target subject when switching from different camera views across time. This invalidates any application where a person may change dress across subsequent acquisitions as can be the case of patients monitoring at home. Differently from RGB data, 3D information as acquired by depth cameras can open the way to person re-identification based on biometric cues such as distinguishing traits of the body or face. However, the accuracy of skeleton and face geometry extracted from depth data is not always adequate to enable person recognition, since both these features are affected by the pose of the subject and the distance from the camera. In this paper, we propose a method to derive a robust skeleton representation from a depth sequence and to complement it with a highly discriminative face feature. This is obtained by selecting skeleton and face samples based on their quality and using the temporal redundancy across the sequence to derive and refine cumulated models for both of them. Extracting skeleton and face features from such cumulated models and combining them for the recognition allow us to improve rank-1 re-identification accuracy compared to individual cues. A comparative evaluation on three benchmark datasets also shows results at the state-of-the-art. [ABSTRACT FROM AUTHOR]

Abstract We investigate the problem of Facial Expression Recognition (FER) using 3D data. Building from one of the most successful frameworks for facial analysis using exclusively 3D geometry, we extend the analysis from a curve-based representation into a spectral representation, which allows a complete description of the underlying surface that can be further tuned to the desired level of detail. Spectral representations are based on the decomposition of the geometry in its spatial frequency components, much like a Fourier transform, which are related to intrinsic characteristics of the surface. In this work, we propose the use of Graph Laplacian Features (GLFs), which result from the projection of local surface patches into a common basis obtained from the Graph Laplacian eigenspace. We extract patches around facial landmarks and include a state-of-the-art localization algorithm to allow for fully-automatic operation. The proposed approach is tested on the three most popular databases for 3D FER (BU-3DFE, Bosphorus and BU-4DFE) in terms of expression and AU recognition. Our results show that the proposed GLFs consistently outperform the curves-based approach as well as the most popular alternative for spectral representation, Shape-DNA, which is based on the Laplace Beltrami Operator and cannot provide a stable basis that guarantee that the extracted signatures for the different patches are directly comparable. Interestingly, the accuracy improvement brought by GLFs is obtained also at a lower computational cost. Considering the extraction of patches as a common step between the three compared approaches, the curves-based framework requires a costly elastic deformation between corresponding curves (e.g. based on splines) and Shape-DNA requires computing an eigen-decomposition of every new patch to be analyzed. In contrast, GLFs only require the projection of the patch geometry into the Graph Laplacian eigenspace, which is common to all patches and can therefore be pre-computed off-line. We also show that 14 automatically detected landmarks are enough to achieve high FER and AU detection rates, only slightly below those obtained when using sets of manually annotated landmarks. Highlights • We present a fully automatic framework for 3D facial expression recognition. • We explore spectral methods as shape descriptors for 3D facial expression analysis. • Our Graph Laplacian Features are shown to outperform alternative local representations. • We show results on the 3 most popular databases for 3D facial expression recognition. [ABSTRACT FROM AUTHOR]

Normally, modeling 3D face is an effective way for pose-invariant recognition, yet its expensive computation significantly discourages potential applications. In this paper, a simple and fully automatic panoramic image-based pose-invariant face recognition method is proposed to present excellent accuracy with low complexity. In this paper, a face shape model with local morphing treatment is first constructed and considered as the alignment standard to deal with all of the possible geometric distortion problems. During the recognition phase, a proposed systematically designed algorithm with morphing and the selection function are both utilized to significantly ease the negative effects of various poses within ±45° in yaw and ±22.5° in pitch. As demonstrated in experimental results, a similar accuracy as that of the 3D start-of-the-arts is achieved with much less computational complexity. [ABSTRACT FROM AUTHOR]

We present an algorithm that automatically establishes dense correspondences between a large number of 3D faces. Starting from automatically detected sparse correspondences on the outer boundary of 3D faces, the algorithm triangulates existing correspondences and expands them iteratively by matching points of distinctive surface curvature along the triangle edges. After exhausting keypoint matches, further correspondences are established by generating evenly distributed points within triangles by evolving level set geodesic curves from the centroids of large triangles. A deformable model (K3DM) is constructed from the dense corresponded faces and an algorithm is proposed for morphing the K3DM to fit unseen faces. This algorithm iterates between rigid alignment of an unseen face followed by regularized morphing of the deformable model. We have extensively evaluated the proposed algorithms on synthetic data and real 3D faces from the FRGCv2, Bosphorus, BU3DFE and UND Ear databases using quantitative and qualitative benchmarks. Our algorithm achieved dense correspondences with a mean localisation error of 1.28 mm on synthetic faces and detected 14 anthropometric landmarks on unseen real faces from the FRGCv2 database with 3 mm precision. Furthermore, our deformable model fitting algorithm achieved 98.5 percent face recognition accuracy on the FRGCv2 and 98.6 percent on Bosphorus database. Our dense model is also able to generalize to unseen datasets. [ABSTRACT FROM AUTHOR]

This study proposes a novel automatic method for facial landmark localization relying on geometrical properties of 3D facial surface working both on complete faces displaying different emotions and in presence of occlusions. In particular, 12 descriptors coming from Differential Geometry including the coefficients of the fundamental forms, Gaussian, mean, principal curvatures, shape index and curvedness are extracted as facial features and their local geometric properties are exploited to localize 13 soft-tissue landmarks from eye and nose areas. The method is deterministic and is backboned by a thresholding technique designed by studying the behaviour of each geometrical descriptor in correspondence to the locus of each landmark. Occlusions are managed by a detection algorithm based on geometrical properties which allows to proceed with the landmark localization avoiding the covered areas. Experimentations were carried out on 3132 faces of the Bosphorus database and of a 230-sized internal database, including expressive and occluded ones (mouth, eye, and eyeglasses occlusions), obtaining 4.75 mm mean localization error. [ABSTRACT FROM AUTHOR]

3D face similarity is a critical issue in computer vision, computer graphics and face recognition and so on. Since Fréchet distance is an effective metric for measuring curve similarity, a novel 3D face similarity measure method based on Fréchet distances of geodesics is proposed in this paper. In our method, the surface similarity between two 3D faces is measured by the similarity between two sets of 3D curves on them. Due to the intrinsic property of geodesics, we select geodesics as the comparison curves. Firstly, the geodesics on each 3D facial model emanating from the nose tip point are extracted in the same initial direction with equal angular increment. Secondly, the Fréchet distances between the two sets of geodesics on the two compared facial models are computed. At last, the similarity between the two facial models is computed based on the Fréchet distances of the geodesics obtained in the second step. We verify our method both theoretically and practically. In theory, we prove that the similarity of our method satisfies three properties: reflexivity, symmetry, and triangle inequality. And in practice, experiments are conducted on the open 3D face database GavaDB, Texas 3D Face Recognition database, and our 3D face database. After the comparison with iso-geodesic and Hausdorff distance method, the results illustrate that our method has good discrimination ability and can not only identify the facial models of the same person, but also distinguish the facial models of any two different persons. [ABSTRACT FROM AUTHOR]