Search

Face parsing aims to assign pixel-wise semantic labels to different facial components (e.g., hair, brows, and lips) in given face images. However, directly predicting pixel-level labels for each facial component over the whole face image would obtain limited accuracy, especially for tiny facial components. To address this problem, some recent works propose to first crop tiny patches from the whole face image and then predict masks for each facial component. However, such cropping-and-segmenting strategy consists of two independent stages, which cannot be jointly optimized. Besides, as one valuable piece of information for parsing the highly structured facial components, context cues are not elaborately explored by the existing works. To address these issues, we propose a component-level refinement network (CLRNet) for precisely segmenting out each facial component. Specifically, we introduce an attention mechanism to bridge the two independent stages together and form an end-to-end trainable pipeline for face parsing. Furthermore, we incorporate the global context information into the refining process for each cropped facial component patch, providing informative cues for accurate parsing. Extensive experiments are carried out on two benchmark datasets, LFW-PL and HELEN. The results demonstrate the superiority of the proposed CLRNet over other state-of-the-art methods, especially for tiny facial components.

Automatic facial emotion recognition in real-world situations like partial occlusions, varying head poses and illumination conditions are challenging to the machine learning community. The main reason is the lack of sufficient samples with the aforementioned conditions in the baseline datasets which throws difficulty in training a well performing machine learning or deep learning model. To overcome this challenge, we have adopted the concept of meta-learning. Meta-learning using prototypical networks (metric-based meta-learning) has been proven to be well-fit for few-shot problems without severe overfitting. We leverage the quick adaptation power of prototypical networks for emotion recognition in the scarcity of such diverse samples. We have used CMU Multi-PIE dataset which contains images with partial occlusions, varying head-poses and illumination levels for training and evaluating the model. For testing the adaptability of the system to intra-class and inter-dataset variations, AffectNet face database images have been used. The proposed method is named as ERMOPI (Emotion Recognition using Meta-learning across Occlusion, Pose and Illumination) which performs emotion recognition from facial expressions using meta-learning approach for still images and it is robust to partial occlusions, varying head poses and illumination levels which is the novelty of this work. The key benefit is the usage of less number of training samples compared to the existing work in emotion recognition and achieved comparable results with the state-of-the-art approaches. The proposed method achieved 90% accuracy for CMU Multi-PIE database images and 68% accuracy for AffectNet database images.

The image-based virtual illumination technology directly changes the illumination effect of objects on the image. It does not require complex light propagation simulation calculations, it is an image-based rendering technology. The image mainly relies on the imaging of the visual sensor, and uses the virtual illumination technology of the image to illuminate the image obtained by the visual sensor. This is a new cross-cutting direction in computer vision, virtual reality and other fields. Face Illumination Swapping via Dense Landmark and Semantic Parsing is a major branch. Keeping the geometrical features of the target images and relighting the entire image instead of the face area are problems to be solved in the research. This paper based on the three-dimensional model to analyze the illumination information of the face images and re-render the illumination of the target face, and finally achieves the illumination swap between the two face images. We designed and implemented a 3DDFA-based face image illumination transfer method. First, 3DDFA is used to reconstruct the target face image. Estimate the surface normal and albedo. Then align and fill the surface normal and face parsing to illuminate the face image for light rendering and illumination transfer of the face images. Finally, the illumination analysis and re-rendering of face images based on 3DDFA are expanded to achieve the swap of illumination between face images. Experimental results show that this method can generate good effects of face image illumination transmission and swap while keeping the geometric features of the target images.

The disclosure of face image features can seriously threaten the security of user information, which limits the application of face recognition technology in Internet of Vehicles. This paper proposes a new method of generating and restoring private face images based on semantic features and adversarial examples. The Segnet network first segments the face images semantically, then a generative adversarial network generates adversarial examples and perturbs the semantic features of the face image. The perturbation position can be accurately controlled through a coefficient matrix as the identity tag of the face image is concealed steganographically. A restoration network is trained to extract the real identity tag from the private face image using a discriminator against the generation network, then it restores the private face image to its original state. Compared to other state-of-the-art methods, private face images generated by the proposed method experimentally show high detection resistance, better quality, and stronger median filtering defense.

The head-mounted displays (HMDs) provide immersive experiences in virtual reality (VR). However, the face interactions are limited due to the serious occlusion of the user’s face. Existed approaches try to recover the user’s facial expression by adding additional sensors to HMD. In this paper, we develop a novel framework to reconstruct the userb’s 3D face in VR only using an RGB camera. Given a reference face, a realistic full-textured avatar is created by fitting a 3D Morphable Model (3DMM). A self-supervised UV map Generative Adversarial Network (GAN) is proposed to make the facial texture look more realistic. Next, we propose a novel landmark detection method to locate the landmark positions under HMD occlusion since facial landmarks are commonly used for driving the avatar. To this end, we synthesize a face dataset with HMD. Our method is easy to build and popularize with low cost. The experiments on the synthetic and real HMD data demonstrate that the proposed method can detect landmark accurately and restore facial expressions faithfully despite the large occlusion of HMD.

Realistic 3D facial modeling and animation have been increasingly used in many graphics, animation, and virtual reality applications. However, generating realistic fine-scale wrinkles on 3D faces, in particular, on animated 3D faces, is still a challenging problem that is far away from being resolved. In this article we propose an end-to-end system to automatically augment coarse-scale 3D faces with synthesized fine-scale geometric wrinkles. By formulating the wrinkle generation problem as a supervised generation task, we implicitly model the continuous space of face wrinkles via a compact generative model, such that plausible face wrinkles can be generated through effective sampling and interpolation in the space. We also introduce a complete pipeline to transfer the synthesized wrinkles between faces with different shapes and topologies. Through many experiments, we demonstrate our method can robustly synthesize plausible fine-scale wrinkles on a variety of coarse-scale 3D faces with different shapes and expressions.

User authentication on smartphones is the key to many applications, which must satisfy both security and convenience. We propose a novel user authentication system EchoPrint, which leverages acoustics and vision for secure and convenient user authentication, without requiring any special hardware. EchoPrint actively emits almost inaudible acoustic signals from the earpiece speaker to illuminate the user's face and authenticates the user by the unique features extracted from the echoes bouncing off the 3D facial contour. To combat changes in phone-holding poses thus echoes, a Convolutional Neural Network (CNN) is trained to extract reliable acoustic features, which are further combined with visual facial features extracted from state-of-the-art face recognition deep models to feed a binary Support Vector Machine (SVM) classifier for final authentication. Because the echo features depend on 3D facial geometries, EchoPrint is not easily spoofed by images or videos like 2D visual face recognition systems. It needs only commodity hardware, thus avoiding the extra costs of special sensors in solutions like FaceID. Experiments with 62 volunteers and non-human objects such as images, photos, and sculptures show that EchoPrint achieves 93.75% balanced accuracy and 93.50% F-score, while the average precision is 98.05% using acoustic features and basic facial landmarks. The precision is further improved to 99.96% with sophisticated visual features.

Facial expression recognition has been studied broadly, and several works using local micro-pattern descriptors have obtained significant results. There are, however, open questions: how to design a discriminative and robust feature descriptor?, how to select expression-related most influential features?, and how to represent the face descriptor exploiting the most salient parts of the face? In this paper, we address these three issues to achieve better performance in recognizing facial expressions. First, we propose a new feature descriptor, namely Local Shape Pattern (LSP), that describes the local shape structure of a pixel's neighborhood based on the prominent directional information by analyzing the statistics of the neighborhood gradient, which allows it to be robust against subtle local noise and distortion. Furthermore, we propose a selection strategy for learning the influential codes being active in the expression affiliated changes by selecting them exhibiting statistical dominance and high spatial variance. Lastly, we learn the size of the salient facial blocks to represent the facial description with the notion that changes in expressions vary in size and location. We conduct person-independent experiments in existing datasets after combining above three proposals, and obtain an improved performance for the facial expression recognition task.

As face editing scenarios have become popular, the face inpainting technique has become a hot topic. Although some existing methods can inpaint faces with preserved identity information, they fail to solve a more flexible inpainting problem that fills the “holes” with identity-specific content from other faces. In this work, we propose a disentangle and subject-agnostic framework that affects both full and partial-face inpainting with the guidance of a reference face image. The framework consists of an identity encoding module, a content inference module and a generative module. The identity encoding module extracts the identity embedding from the reference image, the content inference module learns to predict the content image, and the generative module integrates the content image and the reference identity embedding to generate the identity-specific inpainted result. To minimize the structure and style gap between the incomplete image and inpainted image, we use a double attribute loss to the generative module and a postprocess of blending operation to the swapped result. We compare our method with state-of-the-art works and demonstrate that our method achieves higher identity similarity and better structural correctness.

The three-dimensional morphable model (3DMM) is the most widely used representative model for obtaining a three-dimensional (3-D) face from a target on an image. Although 3DMMs have demonstrated the powerful capability to represent various facial shapes on natural images, they are limited to capturing texture variations of in-the-wild human faces. Based on the fact that fitting a 3-D facial model to an image determines the corresponding UV map, we propose a novel method for facial fitting and synthesis by competitively training two deep learning networks for facial alignment and UV texture completion. When the completion network is trained using well-aligned UV maps, it can model facial textures precisely and, consequently, fill the missing regions more completely. Accordingly, we use a UV completion network, denoted as a UV energy-based generative adversarial network (UV EB-GAN), to discriminate whether a UV map from the alignment network is well aligned by defining the generative loss of the completion network as the energy. Competitive learning facilitates training the completion network without ground-truth facial UV maps and training the alignment network without hard constraints and regularization terms. The proposed network can be trained in an end-to-end manner. The facial texture, albedo, lighting parameters, and 3-D facial shape can be obtained through this network. The results of the experiments on 2-D alignment, 3-D reconstruction, texture synthesis, and illumination estimation verified that the proposed method achieves remarkable improvements over the state-of-the-art methods.

Due to the proliferation of sports-related multimedia content on the WWW, effective visual search and retrieval present interesting research challenges. These are caused by poor image quality, a wide range of possible camera points of view, pose variations on the part of athletes engaged in playing a sport, deformations of text appearing on sports person’s clothing and uniforms in motion, occlusions caused by other objects, etc. To address these challenges, this paper presents a new method for detecting text on human bodies in sports images. Unlike most existing methods, which attempt to exploit locations of a player’s torso, face, and skin, we propose an end-to-end episodic learning approach that employs inductive learning criteria for detecting clothing regions in an image, which are, in turn, then used for text detection. Our method integrates a Residual Network (ResNet) and Pyramidal Pooling Module (PPM) for generating a spatial attention map. The Progressive Scalable Expansion Algorithm (PSE) is adapted for text detection from these regions. Experimental results on our own dataset as well as several benchmarks (like RBNR and MMM which contain images of runners in marathons, and Re-ID which is a person re-identification dataset) demonstrate that the proposed method outperforms existing methods in terms of precision and F1-score. We also present results for sports images chosen from natural scene text detection datasets such as CTW1500 and MS-COCO to show the proposed method is effective and reliable across a range of inputs.

Recently, researchers have shown great interest in using convolutional neural networks (CNNs) for no-reference image quality assessment (NR-IQA). Due to the lack of big training data, the efforts of existing metrics in optimizing CNN-based NR-IQA models remain limited. Furthermore, the diversity of distortions in images result in the generalization problem of NR-IQA models when trained with known distortions and tested on unseen distortions, which is an easy task for human. Hence, we propose a NR-IQA metric via deep meta-learning, which is highly generalizable in the face of unseen distortions. The fundamental idea is to learn the meta-knowledge shared by human when evaluating the quality of images with diversified distortions. Specifically, we define NR-IQA of different distortions as a series of tasks and propose a task selection strategy to build two task sets, which are characterized by synthetic to synthetic and synthetic to authentic distortions, respectively. Based on these two task sets, an optimization-based meta-learning is proposed to learn the generalized NR-IQA model, which can be directly used to evaluate the quality of images with unseen distortions. Extensive experiments demonstrate that our NR-IQA metric outperforms the state-of-the-arts in terms of both evaluation performance and generalization ability.

Face recognition can be installed in a surveillance system so that it can be used for monitoring, tracking and access control. An excellent, intelligent surveillance system should be sensitive to the objects far away from the camera. Unfortunately, due to the long-distance, objects like human faces captured by the camera are too small to identify. As to enhance the subtle color differences in the face image, in this paper we first improve the resolution of the captured image using deep convolution neural networks (DCNNs). Then the efficient features are extracted and used to do classification. As for verifying the effectiveness of the proposed method, we used three databases including AR face database, Georgia Tech face database (GT) database, and Labelled Faces in the Wild (LFW) database, altogether, to conduct the training and testing. Compared to the existing approaches, experimental results show that the identification accuracy of the proposed method outperforms any existing approaches.

Sparse representation-based face recognition has gained considerable attention recently due to its robustness against illumination and occlusion. Recognizing faces from videos has become a topic of importance to alleviate the limit of information content in still images. However, the sparse recognition framework is not applicable to video-based face recognition due to its sensitivity towards pose and alignment changes. In this paper, we propose a video-based face recognition method which improves upon the sparse representation framework. Our key contribution is an intelligent and adaptive sparse dictionary that updates the current probe image into the training matrix based on continuously monitoring the probe video through a novel confidence criterion and a Bayesian inference scheme. Due to this novel approach, our method is robust to pose and alignment and hence can be used to recognize faces from unconstrained videos successfully. Moreover, in a moving scene, camera angle, illumination and other imaging conditions may change quickly leading to performance loss in accuracy. In such situations, it is impractical to re-enroll the individual and re-train the classifiers on a continuous basis. Our novel approach addresses these practical issues. Experimental results on the well known YouTube Face database demonstrates the effectiveness of our method.

Automatic face recognition system is suggested in this work on the basis of appearance based features focusing on the whole image as well as local based features focusing on critical face points like eyes, mouth, and nose for generating further details. Face detection is the major phase in face recognition systems, certain method for face detection (Viola-Jones) has the ability to process images efficiently and achieve high rates of detection in real time systems. Dimension reduction and feature extraction approaches are going to be utilized on the cropped image caused by detection. One of the simple, yet effective ways for extracting image features is the Local Binary Pattern Histogram (LBPH), while the technique of Principal Component Analysis (PCA) was majorly utilized in pattern recognition. Also, the technique of Linear Discriminant Analysis (LDA) utilized for overcoming PCA limitations was efficiently used in face recognition. Furthermore, classification is going to be utilized following the feature extraction. The utilized machine learning algorithms are PART and J48. The suggested system is showing high accuracy for detection with Viola-Jones 98.75, whereas the features which are extracted by means of LDA with J48 provided the best results of (F-measure, Recall, and Precision).

Most existing face image Super‐Resolution (SR) methods assume that the Low‐Resolution (LR) images were artificially downsampled from High‐Resolution (HR) images with bicubic interpolation. This operation changes the natural image characteristics and reduces noise. Hence, SR methods trained on such data most often fail to produce good results when applied to real LR images. To solve this problem, a novel framework for the generation of realistic LR/HR training pairs is proposed. The framework estimates realistic blur kernels, noise distributions, and JPEG compression artifacts to generate LR images with similar image characteristics as the ones in the source domain. This allows to train an SR model using high‐quality face images as Ground‐Truth (GT). For better perceptual quality, a Generative Adversarial Network (GAN) based SR model is used, where the commonly used VGG‐loss [1] is exchanged with LPIPS‐loss [2]. Experimental results on both real and artificially corrupted face images show that our method results in more detailed reconstructions with less noise compared to the existing State‐of‐the‐Art (SoTA) methods. In addition, it is shown that the traditional non‐reference Image Quality Assessment (IQA) methods fail to capture this improvement and demonstrate that the more recent NIMA metric [3] correlates better with human perception via Mean Opinion Rank (MOR).

Along with the performance improvement of deep-learning-based face hallucination methods, various face priors (facial shape, facial landmark heatmaps, or parsing maps) have been used to describe holistic and partial facial features, making the cost of generating super-resolved face images expensive and laborious. To deal with this problem, we present a simple yet effective dual-path deep fusion network (DPDFN) for face image super-resolution (SR) without requiring additional face prior, which learns the global facial shape and local facial components through two individual branches. The proposed DPDFN is composed of three components: a global memory subnetwork (GMN), a local reinforcement subnetwork (LRN), and a fusion and reconstruction module (FRM). In particular, GMN characterize the holistic facial shape by employing recurrent dense residual learning to excavate wide-range context across spatial series. Meanwhile, LRN is committed to learning local facial components, which focuses on the patch-wise mapping relations between low-resolution (LR) and high-resolution (HR) space on local regions rather than the entire image. Furthermore, by aggregating the global and local facial information from the preceding dual-path subnetworks, FRM can generate the corresponding high-quality face image. Experimental results of face hallucination on public face data sets and face recognition on real-world data sets (VGGface and SCFace) show the superiority both on visual effect and objective indicators over the previous state-of-the-art methods.

The facial caricature shows the distinct characteristics of a person via exaggerations of both shape and appearance. This paper presents a novel framework that automatically generates vivid facial caricatures by encoding personalized semantic information. To this end, we first design a part-based scheme for geometry warping, which composes local semantic deformation into a global warping field, equipped with sufficient warping freedom of different facial components. Second, under the scheme of Part-based Warping, we design a photo-to-caricature translation network called PbWarpGAN, and adopt several novel losses to capture the personalized characteristics of each input face and preserve its identity better. Third, based on PbWarpGAN, we develop a user-friendly interface by introducing an attention scheme on each facial component, allowing ordinary users to adjust the automatically generated caricature by PbWarpGAN according to their preference conveniently. Experimental results show that our PbWarpGAN is more effective in capturing personalized characteristics than counterparts, and provides an efficient tool for caricature designing application.

Despite tremendous success has been achieved in faithfully reconstructing face shapes from single images, recovering accurate local details still remains challenging. Previous works propose reprojection-based methods to improve the performance of detail recovering – they render a textured 3D shape into an image and make it approximate to the input during iterations. However, details from textures and shapes are mixed in the rendered image when minimizing the re-projection loss, which leads to limitations in detail recovery. To address this issue, we propose a novel 3D face reconstruction framework that 1) uses a coarse-medium-fine strategy to capture details while preserving the global shape, 2) disentangles details from the texture to enhance local accuracy, and 3) applies a phased optimization to recover details over multiple scales. Experiments demonstrate the capability of our framework to reconstruct high-fidelity face shapes with accurate, fine details.

Emotion recognition is a crucial application in human–computer interaction. It is usually conducted using facial expressions as the main modality, which might not be reliable. In this study, we proposed a multimodal approach that uses 2-channel electroencephalography (EEG) signals and eye modality in addition to the face modality to enhance the recognition performance. We also studied the use of facial images versus facial depth as the face modality and adapted the common arousal–valence model of emotions and the convolutional neural network, which can model the spatiotemporal information from the modality data for emotion recognition. Extensive experiments were conducted on the modality and emotion data, the results of which showed that our system has high accuracies of 67.8% and 77.0% in valence recognition and arousal recognition, respectively. The proposed method outperformed most state-of-the-art systems that use similar but fewer modalities. Moreover, the use of facial depth has outperformed the use of facial images. The proposed method of emotion recognition has significant potential for integration into various educational applications.

Face detection is to search all the possible regions for faces in images and locate the faces if there are any. Many applications including face recognition, facial expression recognition, face tracking and head-pose estimation assume that both the location and the size of faces are known in the image. In recent decades, researchers have created many typical and efficient face detectors from the Viola-Jones face detector to current CNN-based ones. However, with the tremendous increase in images and videos with variations in face scale, appearance, expression, occlusion and pose, traditional face detectors are challenged to detect various "in the wild" faces. The emergence of deep learning techniques brought remarkable breakthroughs to face detection along with the price of a considerable increase in computation. This paper introduces representative deep learning-based methods and presents a deep and thorough analysis in terms of accuracy and efficiency. We further compare and discuss the popular and challenging datasets and their evaluation metrics. A comprehensive comparison of several successful deep learning-based face detectors is conducted to uncover their efficiency using two metrics: FLOPs and latency. The paper can guide to choose appropriate face detectors for different applications and also to develop more efficient and accurate detectors.

Talking face generation is a demanding task to synthesize a high quality video with accurate lip synchronization and rhythmic head motion. Any subtle artifacts could be sensitively captured by humans and lead to poor visual quality. Existing methods tend to employ a conditional generation solution, which introduces facial landmarks to bridge the input information and output videos. However, these methods always suffer from unrealistic facial animations, because 1) they only take single-mode input, but ignore the complementarity of multimodal inputs for lip-sync improvement; 2) they only explore lip movements, but ignore the articulator synergy between lips and jaw; 3) they generate each video frame in a temporal-independent way, but ignore the temporal continuity among the entire video. To address these limitations, in this paper, we present a novel method to generate realistic and temporally coherent talking heads by considering multimodal inputs, articulator synergy, inter-frame consistency and intra-frame consistency. Firstly, for landmark prediction, a novel Multiple Synergy Network (MSN) is proposed to improve the accuracy of landmark prediction by incorporating multimodal inputs (i.e., audio and text inputs). Besides, instead of merely considering lip landmarks, we also explore the jaw movements to ensure articulator synergy among lips and jaw. Secondly, for realistic video generation, a Video Consistency Network (VCN) is proposed conditioned on the predicted landmarks. In VCN, the optical flow is adopted to model the temporal continuity between frames to ensure inter-frame consistency. Meanwhile, a mouth generation branch is proposed to enhance mouth texture and the corresponding mouth mask is employed to ensure intra-frame consistency between the mouth area and the others. Extensive experiments demonstrate that our approach exhibits excellent superiority on lip-sync and can generate photo-realistic facial animations. Project is available at http://imcc.ustc.edu.cn/project/tfgen/.

The human vision system can efficiently recognize multiscale objects in cluttered backgrounds. The scheme can be achieved with a visual attention mechanism by concentrating visual resources to the saliency area while ignoring other task-irrelevant areas. However, in the computer vision community, when recognizing multiscale faces in cluttered backgrounds, object detection modules are necessary to locate face regions and reduce the influence of complex backgrounds on the recognition model, which inevitably increases the computational complexity. Motivated by the human vision system, this study proposes the attention developmental network to recognize multiscale faces without using face detectors. A top-down attention mechanism is used to teach the network to focus on the face areas and ignore the backgrounds. An attention-based synapse maintenance mechanism is also introduced to further suppress the background pixels and improve the accuracy of face recognition. Comparative experiments show that our method can attain at least 13% of accuracy improvement over bionic neural networks and ResNet-based recognition networks on the same model scale with less training epochs.

State-of-the-art face restoration methods employ deep convolutional neural networks (CNNs) to learn a mapping between degraded and sharp facial patterns by exploring local appearance knowledge. However, most of these methods do not well exploit facial structures and identity information, and only deal with task-specific face restoration (e.g., face super-resolution or deblurring). In this paper, we propose cross-tasks and cross-models plug-and-play 3D facial priors to explicitly embed the network with the sharp facial structures for general face restoration tasks. Our 3D priors are the first to explore 3D morphable knowledge based on the fusion of parametric descriptions of face attributes (e.g., identity, facial expression, texture, illumination, and face pose). Furthermore, the priors can easily be incorporated into any network and are very efficient in improving the performance and accelerating the convergence speed. Firstly, a 3D face rendering branch is set up to obtain 3D priors of salient facial structures and identity knowledge. Secondly, for better exploiting this hierarchical information (i.e., intensity similarity, 3D facial structure, and identity content), a spatial attention module is designed for the image restoration problems. Extensive face restoration experiments including face super-resolution and deblurring demonstrate that the proposed 3D priors achieve superior face restoration results over the state-of-the-art algorithms.

ML (machine learning) is consisted of a method of recognizing face. This technique is useful for the attendance system. Two sets are generated for testing and training phases in order to segment the image, to extract the features and develop a dataset. An image is considered as a testing set; the training set is contrasted when it is essential to identify an image. An ensemble classifier is implemented to classify the test images as recognized or non-recognized. The ensemble algorithm fails to acquire higher accuracy as it classifies the data in two classes. Thus, GLCM (Grey Level Co-occurrence Matrix) is projected for analyzing the texture features in order to detect the face. The attendance of the query image is marked after detecting the face. The simulation outcomes revealed the superiority of the projected technique over the traditional methods concerning accuracy. Keywords: DWT, GLCM, KNN, Decision Tree

With the growing needs of practical applications such as security monitoring, partial face recognition is a challenging but important issue, because the captured faces in real-world surveillance videos may be occluded or with variations. Though current face recognition methods perform well in relatively constrained scenes, they may suffer from degradation for partial faces. In this paper, we propose a framework of model-based transfer learning and sparse coding (MTLSC) for partial face recognition. First, due to less information in partial face image, we exploit the mirrored image of an original probe sample as sample augment to provide further information. Considering the inadequacy of training face samples, we obtain face features based on model-based transfer learning VGGNet that is pre-trained on VGGFace dataset. Then we reconstruct face features by sliding window in view of different sizes of partial face hard to extract the same feature dimension. Finally we carry out sparse coding with rectification and calculate the minimum score of the probe and mirrored samples among all classes to get the results. Thus, by model-based transfer learning, sliding window for feature reconstruction and sparse coding with rectification, the proposed framework improves partial face recognition performance. Experimental results on three face databases (LFW, AR and NIR), and two person re-identification databases (iLIDS-VID and PKU-Reid) demonstrate our method is effective for partial face recognition.

Kinship verification using single image of a person is a challenging task for real-world applications. In this paper, we propose a novel robust discriminative feature subspace analysis (RDFSA) method to address single sample per person (SSPP) problem in kinship verification. The proposed RDFSA method takes advantages of facial symmetry and patch-based analysis to extract discriminative features for kinship verification. Each face image is firstly divided into two halves about bilateral symmetry axis, and each halved face is then partitioned into equal sized non-overlapping patches. Multiple image-sets are formed by grouping these patches according to their positions at each halved face. Then, an SSPP is formulated as an RDFSA problem and a feature subspace is learned by maximizing inter-class separation and minimizing intra-class variance for different patches in each image-set. For a given test image pair, similarity is computed for each feature subspace and majority voting strategy is employed to determine if a given image pair is kin related or not. Proposed RDFSA method is extensively evaluated on different publicly available kinship datasets to validate kinship accuracy. Experimental results show that RDFSA achieves competitive accuracy on all kinship datasets while performing kinship verification under unconstrained environment.

Facial Recognition is the biometric technique used in face detection. The task for validating or recognizing a face from the multi-media photographs is done using facial recognition technique. With the evolution of advanced society the requirement for face identification has been really important. Detection and identification of faces has been grown worldwide. It owes the demand for security such as authorization, national safety and other vital circumstances. There are number of algorithms for facial detection. This paper aspires to present the comparison of two face recognition techniques Haar Cascade and Local Binary Pattern edified for the classification. As a result the accuracy of Haar Cascade is more than the Local Binary Pattern but the execution time in Haar Cascade is more than Local Binary Pattern.

The zero-shot semantic segmentation requires models with a strong image understanding ability. The majority of current solutions are based on direct mapping or generation. These schemes are effective in dealing with the zero-shot recognition, but they cannot fully transfer the visual dependence between objects in more complex scenarios of semantic segmentation. More importantly, the predicted results become seriously biased to the seen-category in the training set, which makes it difficult to accurately recognize the unseen-category. In view of the above two problems, we propose a novel zero-shot semantic segmentation model based on meta-learning. It is observed that the pure semantic space expression has certain limitations for the zero-shot learning. Therefore, based on the original semantic migration, we first migrate the shared information in the visual space by adding a context-module, and then migrate it in the visual and semantic dual space. At the same time, in order to solve the problem of biasness, we improve the adaptability of the model parameters by adjusting the parameters of the dual-space through the meta-learning, so that it can successfully complete the segmentation even in the face of new categories without reference samples. Experiments show that our algorithm outperforms the existing best methods in the zero-shot segmentation on three datasets of Pascal-VOC 2012, Pascal-Context and Coco-stuff.

Over the past decades, three-dimensional(3D) face recognition has developed rapidly due to its intrinsic invariance to pose and illumination changes. Yet despite this, deep learning is rarely used in 3D face recognition due to lack of training dat a. Noting that the majority of the recognition processes are based on facial depth map instead of real 3D data, we focus on the generation of facial depth map to surmount the shortage of training data. In this paper, we present a novel paradigm, Facial Depth Descend, which generates facial depth map from existing 3D face data. The key to this generation paradigm is to recombine facial components from existing faces and thereafter generate brand new faces based on them. We then propose a learning framework RPC to generate recognition-friendly faces. First, it extracts three facial components (eyes, nose and mouth) from the 3D data of real faces. Then, with these components as input, a Relative Location Estimator (RLE) is used to predict the relative location among them so that they can be composed in a reasonable manner. Finally, the method feeds a Pix2Pix network with these composed facial components to extrapolate areas surrounding them and output a full facial depth map. We here enforce an identity preserving loss on the generation network to make the facial depth map more discriminative and favorable for recognition tasks. Since we can choose different identities’ different components as input, RPC is theoretically able to generate a vast number of novel 2.5D faces. More specifically, RPC can generate an enlarged face dataset as large as N 3 identities, where N is the number of identities in the original one. Experiments show that, adding the generated depth maps to the training dataset can help improve the recognition rate in 2.5D face recognition.

Face images, as an information carrier, are rich in sensitive information. Direct publication of these images would cause privacy leak, due to their natural weak privacy. Most of the existing privacy protection methods for face images adopt data publication under a non-interactive framework. However, the E-effect under this framework covers the entire image, such that the noise influence is uniform across the image. To solve the problem, this paper proposes region growing publication (RGP), an algorithm for the interactive publication of face images under differential privacy. This innovative algorithm combines the region growing technique with differential privacy technique. The privacy budget E is dynamically allocated, and the Laplace noise is added, according to the similarity between adjacent sub-images. To measure this similarity more effectively, the fusion similarity measurement mechanism (FSMM) was designed, which better adapts to the intrinsic attributes of images. Different from traditional region growing rules, the FSMM fully considers various attributes of images, including brightness, contrast, structure, color, texture, and spatial distribution. To further enhance algorithm feasibility, RGP was extended to atypical region growing publication (ARGP). While RGP limits the region growing direction between adjacent sub-images, ARGP searches for the qualified sub-images across the image, with the aid of the exponential mechanism, thereby expanding the region merging scope of the seed point. The results show that our algorithm can satisfy E-differential privacy, and the denoised image still have a high availability.

Recent works based on deep learning and facial priors have performed well in superresolving severely degraded facial images. However, due to the limitation of illumination, pixels of the monitoring probe itself, focusing area, and human motion, the face image is usually blurred or even deformed. To address this problem, we properly propose Face Restoration Generative Adversarial Networks to improve the resolution and restore the details of the blurred face. They include the Head Pose Estimation Network, Postural Transformer Network, and Face Generative Adversarial Networks. In this paper, we employ the following: (i) Swish-B activation function that is used in Face Generative Adversarial Networks to accelerate the convergence speed of the cross-entropy cost function, (ii) a special prejudgment monitor that is added to improve the accuracy of the discriminant, and (iii) the modified Postural Transformer Network that is used with 3D face reconstruction network to correct faces at different expression pose angles. Our method improves the resolution of face image and performs well in image restoration. We demonstrate how our method can produce high-quality faces, and it is superior to the most advanced methods on the reconstruction task of blind faces for in-the-wild images; especially, our 8 × SR SSIM and PSNR are, respectively, 0.078 and 1.16 higher than FSRNet in AFLW.

Low-resolution face images can be found in many practical applications. For example, faces captured from surveillance videos are typically in small sizes. Existing face recognition deep networks, trained on high-resolution images, perform poorly in recognizing low-resolution faces. In this work, an improved multi-branch network is proposed by combining ResNet and feature super-resolution modules. ResNet is for recognizing high-resolution facial images and extracting features from both high- and low-resolution images. Feature super-resolution modules are inserted before the classifier of ResNet for low-resolution facial images. They are used to increase feature resolution. The proposed method is effective and simple. Experimental results show that the recognition accuracy for high-resolution face images is high, and the recognition accuracy for low-resolution face images is improved.

The development of web cameras and smart phones is mature, and more and more facial recognition-related applications are implemented on embedded systems. The demand for real-time face recognition on embedded systems is also increasing. In order to improve the accuracy of face recognition, most of the modern face recognition systems consist of multiple deep neural network models for recognition. However, in an embedded system, integrating these complex neural network models and execute simultaneously is not easy to achieve the goal of real-time recognition of human faces and their identities. In view of this, this study proposes a new frame analysis mechanism, continuous frames skipping mechanism (CFSM), which can analyze the frame in real time to determine whether it is necessary to perform face recognition on the current frame. Through the analysis of CFSM, the frames that do not need to be re-recognized for face are omitted. In this way, the workload of the face recognition system will be greatly reduced to achieve the goal of real-time face recognition in the embedded system. The experimental results show that the proposed CFSM mechanism can greatly increase the speed of face recognition in the video on the embedded system, achieving the goal of real-time face recognition.

Emotion recognition from facial images is considered as a challenging task due to the varying nature of facial expressions. The prior studies on emotion classification from facial images using deep learning models have focused on emotion recognition from facial images but face the issue of performance degradation due to poor selection of layers in the convolutional neural network model.To address this issue, we propose an efficient deep learning technique using a convolutional neural network model for classifying emotions from facial images and detecting age and gender from the facial expressions efficiently. Experimental results show that the proposed model outperformed baseline works by achieving an accuracy of 95.65% for emotion recognition, 98.5% for age recognition, and 99.14% for gender recognition.

Face recognition systems are widely used for target recognition and identity authentication, such as automated teller machines, mobile phones, and entrance guard systems. However, face recognition systems are vulnerable to presentation attacks, such as photo, replay, and 3D mask attacks. In particular, silicone mask attacks pose a greater threat to face recognition systems because high-quality silicone masks do living properties. To promote the development of face anti-spoofing detection algorithms for silicone mask attacks, this paper constructs a Silicone Mask Face Motion Video Dataset (SMFMVD) containing 200 real face videos and 200 silicone mask face videos. These videos include different facial motions collected from 20 subjects. Moreover, inspired by the observation that the silicone mask face’s facial movement is not so natural as the real face, we propose a novel silicone mask face anti-spoofing detection method based on visual saliency and facial motion characteristics. Specifically, we compute the visual saliency map of a given face image by simulating two kinds of eye movement behaviors, namely “gaze” and “saccade”. Then, we propose a saliency-weighted histogram of local binary pattern operator to extract facial texture features in spatial domain and a saliency-guided histogram of oriented optical flow operator to extract facial motion features in temporal domain. Finally, the support vector machine is used to fuse two groups of facial features to distinguish real and spoof faces. Extensive experiments on public and self-built datasets show its superiority over the state-of-the-art methods.

Illumination normalization is one of the most challenging issues for facial analysis. To be specific, the variation of environmental illumination influences the visual quality of an image and drastically degrades the performance of face recognition, detection, landmark and other related applications. Retinex theory provides an important concept for processing illumination variation, which supposes that a face image can be decomposed into [an invariant reflectance component and a variant illumination component. Enlighten by this theory, in this paper, we put forward a novel deep learning approach which combines self-supervised learning and adversarial training for face illumination normalization (FIN-GAN). The proposed FIN-GAN framework can be implemented by two steps. Firstly, self-supervised learning is employed to decompose the original face image into the illumination component and the illumination-invariant component with Retinex constraint. Then, we employ the conditional generative adversarial network for face image reconstruction. For network optimization, we design the combined loss to ensure visual quality and preserve identity information. Experiments are performed on Extended-YaleB, CAS-PEAL, CMU-PIE and Multi-PIE datasets. Through multiple quantitative criteria, we demonstrate that the proposed FIN-GAN obtains promising performance in face illumination normalization.

A rapidly growing amount of personal sensitive information is being released to the public due to unprotected sharing of face images and videos on social networks. Although some pioneering face de-identification techniques, such as face blurring, have been proposed, there is still a long way towards providing full protection of one’s facial privacy. In this paper, we propose a novel end-to-end privacy protection approach to seamlessly replace a face in an image with a synthesized face that looks as natural as normal photos yet pertaining very different look from the original face. The synthesized face images will prevent potential attackers from de-identifying the users. Specifically, our approach relies on generative adversarial network and considers both the foreground and background constrains with respect to the input face image to achieve the following two goals: Firstly, to make synthesized images perceptually unaltered, we design a new generative model to effectively fuse a synthesized face with the original background. Secondly, to ensure a synthesized face to be much different from the original face, we define multiple losses to distinguish the synthesized face from the original face. The experimental results on public datasets have validated the effectiveness of our approach compared with the-state-of-the-art.

With the hyperspectral sensor technology evolving and becoming more cost-effective, hyperspectral imaging offers new opportunities for robust face recognition. Hyperspectral face cubes contain much more spectral information than face images from common RGB color cameras. Hyperspectral face recognition is robust to the impacts, such as illumination, pose, occlusion, and spoofing, which can heavily avoid the limitations of the visible-image-based face recognition. In this paper, we summarize the spectrum properties of hyperspectral face cubes and survey the hyperspectral face recognition methods in the literature. We categorize them into major groups for better understanding. We overview the existing hyperspectral face datasets, and establish our own dataset. We also discuss efficient neural networks used for mobile face recognition and conduct experiments on mobile hyperspectral face recognition. Results show that under harsh conditions like large illumination changing and pose variation, hyperspectral-cube-based methods have higher recognition accuracy than visible-image-based methods. Finally, we deliver insightful discussions and prospects for future works on mobile hyperspectral face recognition.

Human facial recognition in the context of surveillance, forensics and photo-ID verification is a task for which accuracy is critical. Quite often limitations in the overall quality of facial images reduces individuals' ability in taking decisions regarding a person's identity. To verify the suitability of advanced imaging techniques to improve individuals' performance in face matching we investigate how High Dynamic Range (HDR) imaging compares with traditional low (or standard) dynamic range (LDR) imaging in a facial recognition task. An HDR face dataset with five different lighting conditions is created. Subsequently, this dataset is used in a controlled experiment (N=40) to measure performance and accuracy of human participants when identifying faces in HDR vs LDR. Results demonstrate that face matching accuracy and reaction time are improved significantly by HDR imaging. This work demonstrates scope for realistic image reproduction and delivery in face matching tasks and suggests that security systems could benefit from the adoption of HDR imaging techniques.

Let G be a connected plane graph with vertex set V and edge set E. For X ∈ {V, E, V ∪ E}, two elements of X are facially adjacent in G if they are incident elements, adjacent vertices, or facially adjacent edges (edges that are consecutive on the boundary walk of a face of G). A coloring of G is facial with respect to X if there is a coloring of elements of X such that facially adjacent elements of X receive different colors. A facial coloring of G is odd if for every face f and every color c, either no element or an odd number of elements incident with f is colored by c. In this paper we investigate odd facial colorings of trees. The main results of this paper are the following: (i) Every tree admits an odd facial vertex-coloring with at most 4 colors; (ii) Only one tree needs 6 colors, the other trees admit an odd facial edge-coloring with at most 5 colors; and (iii) Every tree admits an odd facial total-coloring with at most 5 colors. Moreover, all these bounds are tight.

In a criminal investigation, along with processing forensic evidence, different investigative techniques are used to identify the perpetrator of the crime. It includes collecting and analyzing unconstrained face images, mostly with low resolution and various qualities, making identification difficult. Since police organizations have limited resources, in this paper, we propose a novel method that utilizes off-the-shelf solutions (Dlib library Histogram of Oriented Gradients-HOG face detectors and the ResNet faces feature vector extractor) to provide practical assistance in unconstrained face identification. Our experiment aimed to establish which one (if any) of the basic image enhancement techniques should be applied to increase the effectiveness. Results obtained from three publicly available databases and one created for this research (simulating police investigators’ database) showed that resizing the image (especially with a resolution lower than 150 pixels) should always precede enhancement to improve face detection accuracy. The best results in determining whether they are the same or different persons in images were obtained by applying sharpening with a high-pass filter, whereas normalization gives the highest classification scores when a single weight value is applied to data from all four databases.

Stress conditions are manifested in different human body’s physiological processes and the human face. Facial expressions are modelled consistently through the Facial Action Coding System (FACS) using the facial Action Units (AU) parameters. This paper focuses on the automated recognition and analysis of AUs in videos as quantitative indices to discriminate between neutral and stress states. A novel deep learning pipeline for automatic recognition of facial action units is proposed, relying on two publicly available annotated facial datasets for training, the UNBC and the BOSPHORUS datasets. Two types of descriptive facial features are extracted from the input images, geometric features (non-rigid 3D facial deformations due to facial expressions) and appearance features (deep facial appearance features). The extracted facial features are then fed to deep fully connected layers that regress AU intensities and robustly perform AU classification. The proposed algorithm is applied to the SRD’15 stress dataset, which contains neutral and stress states related to four types of stressors. We present thorough experimental results and comparisons, which indicate that the proposed methodology yields particularly promising performance in terms of both AU detection and stress recognition accuracy. Furthermore, the AUs relevant to stress were experimentally identified, providing evidence that their intensity is significantly increased during stress, which leads to a more expressive human face as compared to neutral states.

In recent years, the image deblurring task has attracted more and more researchers’ attention. Many researchers are devoted to eliminating motion blur by using a “coarse-to-fine” architecture. This architecture can effectively eliminate motion blur caused by a simple relative displacement. But when using the architecture directly for the face image deblurring task, there would exist some problems. For example, complex network structures make the model difficult to be trained, and a large number of parameters need to be calculated would result in expensive runtime. Since details of images can’t be completely restored, the quality of images will deteriorate, and deblurring visual effect is poor, so in order to solve these issues, combining the “coarse-to-fine” architecture with the “stacked” architecture, we propose a new network architecture-“Stack-based Scale-recurrent Network” for the face image deblurring task. The ConvLSTM network is employed to build a model. We achieves the purpose of improving the visual clarity and the quality of restored face images. And we use the improved encoder-decoder to enhance network performance. Compared with the other deblurring methods, our method can restore clear face images with higher-quality and clearer vision.

Deepfake network is a prominent topic of research as an application to various systems about security measures. Although there have been many recent advancements in facial reconstruction, the greatest challenge to overcome has been the means of finding an efficient and quick way to compute facial similarities or matches. This work is created utilizing the rationale-augmented convolutional neural network (CNN) on MATLAB R2019a platform using the Kaggle DeepFake Video dataset with an accuracy of 95.77%. Hence, real-time deepfake facial reconstruction for security purposes is difficult to complete concerning limited hardware and efficiency. This research paper looks into rational augmented CNN state-of-the-art technology utilized for deepfake facial reconstruction via hardware such as webcams and security cameras in real time. Additionally, discuss a history of face reconstruction and provide an overview of how it is accomplished.

This article presents solutions to the person's fatigue recognition problem by the face's image analysis based on convolutional neural networks. In the present paper, existing algorithms were considered. A new model's architecture was proposed and implemented. Resultant metrics of the model were evaluated.

The detection of face tampering in videos created by artificial intelligence techniques (commonly known as the Deep Fakes) has become an important and challenging task in network security defense. In this paper, we propose a novel attention-based deep fake video detection method, which captures the sharp changes in terms of the facial features caused by the composite video. We utilize the convolutional long short-term memory to extract both spatial and temporal information of DeeFake videos. Meanwhile, we apply the attention mechanism to emphasize the specific facial area of each video frame. Finally, we design a decoder to further fusion multiple frames information for more accurate detection results. Experimental results and comparisons with state-of-the-art methods demonstrate that our framework achieves superior performance.

Face image captured via surveillance videos in an open environment is usually of low quality, which seriously affects the visual quality and recognition accuracy. Most image super‐resolution methods adopt paired high‐quality and its interpolated low‐resolution version to train the super‐resolution network. It is difficult to achieve contented visual quality and restoring discriminative features in real scenarios. A discriminative self‐attention cycle generative adversarial network is proposed for real‐world face image super‐resolution. Based on the cycle GAN framework, unpaired samples are adopted to train a degradation network and a reconstruction network simultaneously. A self‐attention mechanism is employed to capture the contextual information for details restoring. A Siamese face recognition network is introduced to provide a constraint on identify consistency. In addition, an asymmetric perceptual loss is introduced to handle the imbalance between the degradation model and the reconstruction model. Experimental results show that the observation model achieved more realistic low‐quality face images, and the super‐resolved face images have shown better subjective quality and higher face recognition performance.

Finding an unfamiliar person in a crowd of others is an integral task for police officers, CCTV-operators, and security staff who may be looking for a suspect or missing person; however, research suggests that it is difficult and accuracy in such tasks is low. In two real-world visual-search experiments, we examined whether being provided with four images versus one image of an unfamiliar target person would help improve accuracy when searching for that person through video footage. In Experiment 1, videos were taken from above and at a distance to simulate CCTV, and images of the target showed their face and torso. In Experiment 2, videos were taken from approximately shoulder height, such as one would expect from body-camera or mobile phone recordings, and target images included only the face. Our findings suggest that having four images as exemplars leads to higher accuracy in the visual search tasks, but this only reached significance in Experiment 2. There also appears to be a conservative bias whereby participants are more likely to respond that the target is not in the video when presented with only one image as opposed to 4. These results point to there being an advantage for providing multiple images of targets for use in video visual-search.

Face photo-sketch synthesis aims at generating a facial sketch/photo conditioned on a given photo/sketch. It covers wide applications including digital entertainment and law enforcement. Precisely depicting face photos/sketches remains challenging due to the restrictions on structural realism and textural consistency. While existing methods achieve compelling results, they mostly yield blurred effects and great deformation over various facial components, leading to the unrealistic feeling of synthesized images. To tackle this challenge, in this article, we propose using facial composition information to help the synthesis of face sketch/photo. Especially, we propose a novel composition-aided generative adversarial network (CA-GAN) for face photo-sketch synthesis. In CA-GAN, we utilize paired inputs, including a face photo/sketch and the corresponding pixelwise face labels for generating a sketch/photo. Next, to focus training on hard-generated components and delicate facial structures, we propose a compositional reconstruction loss. In addition, we employ a perceptual loss function to encourage the synthesized image and real image to be perceptually similar. Finally, we use stacked CA-GANs (SCA-GANs) to further rectify defects and add compelling details. The experimental results show that our method is capable of generating both visually comfortable and identity-preserving face sketches/photos over a wide range of challenging data. In addition, our method significantly decreases the best previous Fréchet inception distance (FID) from 36.2 to 26.2 for sketch synthesis, and from 60.9 to 30.5 for photo synthesis. Besides, we demonstrate that the proposed method is of considerable generalization ability.