Search

Being able to recognize emotional intensity is a desirable feature for a facial emotional recognition (FER) system. However, the development of such a feature is hindered by the paucity of intensity-labeled data for model training. To ameliorate the situation, the present study proposes using face morphing as a way of data augmentation to synthesize faces that express different degrees of a designated emotion. Such an approach has been successfully validated on humans and machines. Specifically, humans indeed perceived different levels of intensified emotions in these parametrically synthesized faces, and FER systems based on neural networks indeed showed improved sensitivities to intensities of different emotions when additionally trained on the synthesized faces. Overall, the proposed data augmentation method is not only simple and effective but also useful for building FER systems that recognize facial expressions of mixed emotions.

Facial micro-expression (ME) can disclose genuine and concealed human feelings. It makes MEs extensively useful in real-world applications pertaining to affective computing and psychology. Unfortunately, they are induced by subtle facial movements for a short duration of time, which makes the ME recognition, a highly challenging problem even for human beings. In automatic ME recognition, the well-known features encode either incomplete or redundant information, and there is a lack of sufficient training data. The proposed method, Micro-Expression Recognition by Analysing Spatial and Temporal Characteristics, MERASTC mitigates these issues for improving the ME recognition. It compactly encodes the subtle deformations using action units (AUs), landmarks, gaze, and appearance features of all the video frames while preserving most of the relevant ME information. Furthermore, it improves the efficacy by introducing a novel neutral face normalization for ME and initiating the utilization of gaze features in deep learning based ME recognition. The features are provided to the 2D convolutional neural network that jointly analyses the spatial and temporal behavior for correct ME classification. Experimental results on the publicly available datasets indicate that the proposed method exhibits better performance than the well-known methods.

Face parsing aims to assign pixel-wise semantic labels to different facial components (e.g., hair, brows, and lips) in given face images. However, directly predicting pixel-level labels for each facial component over the whole face image would obtain limited accuracy, especially for tiny facial components. To address this problem, some recent works propose to first crop tiny patches from the whole face image and then predict masks for each facial component. However, such cropping-and-segmenting strategy consists of two independent stages, which cannot be jointly optimized. Besides, as one valuable piece of information for parsing the highly structured facial components, context cues are not elaborately explored by the existing works. To address these issues, we propose a component-level refinement network (CLRNet) for precisely segmenting out each facial component. Specifically, we introduce an attention mechanism to bridge the two independent stages together and form an end-to-end trainable pipeline for face parsing. Furthermore, we incorporate the global context information into the refining process for each cropped facial component patch, providing informative cues for accurate parsing. Extensive experiments are carried out on two benchmark datasets, LFW-PL and HELEN. The results demonstrate the superiority of the proposed CLRNet over other state-of-the-art methods, especially for tiny facial components.

Deep learning techniques have proven effective in many applications, but these implementations mostly apply to data in one or two dimensions. Handling 3D data is more challenging due to its irregularity and complexity, and there is a growing interest in adapting deep learning techniques to the 3D domain. A recent successful approach called MeshCNN consists of a set of convolutional and pooling operators applied to the edges of triangular meshes. While this approach produced superb results in classification and segmentation of 3D shapes, it can only be applied to edges of a mesh, which can constitute a disadvantage for applications where the focuses are other primitives of the mesh. In this study, we propose face-based and vertex-based operators for mesh convolutional networks. We design two novel architectures based on the MeshCNN network that can operate on faces and vertices of a mesh, respectively. We demonstrate that the proposed face-based architecture outperforms the original MeshCNN implementation in mesh classification and mesh segmentation, setting the new state of the art on benchmark datasets. In addition, we extend the vertex-based operator to fit in the Point2Mesh model for mesh reconstruction from clean, noisy, and incomplete point clouds. While no statistically significant performance improvements are observed, the model training and inference time are reduced by the proposed approach by 91% and 20%, respectively, as compared with the original Point2Mesh model.

The effects of the compaction conditions and backfill material behind the face of reinforced soil walls (RSWs) were numerically investigated. The analyses were carried out using a two-dimensional finite-element computer program considering different backfill materials in the zone 1 m behind the wall face and different compaction conditions. The results of the analyses were compared with data from a 16.7 m high well-instrumented, steel-reinforced soil wall constructed in 1999 in the USA. The results illustrate the importance of the compaction conditions and the backfill material near the wall facing on the performance of RSWs both at the end of construction and post-construction.

This paper aims to solve two important issues that frequently occur in existing automatic personality analysis systems: 1. Attempting to use very short video segments or even single frames to infer personality traits; 2. Lack of methods to encode person-specific facial dynamics for personality recognition. Hence, we proposes a novel Rank Loss which utilizes the natural temporal evolution of facial actions, rather than personality labels, for self-supervised learning of facial dynamics. Our approach first trains a generic U-net model that can infer general facial dynamics learned from unlabelled face videos. Then, the generic model is frozen, and a set of intermediate filters are incorporated into this architecture. The self-supervised learning is then resumed with only person-specific videos. This way, the learned filters' weights are person-specific, making them a valuable source for modeling person-specific facial dynamics. We then concatenate the weights of the learned filters as a person-specific representation, which can be directly used to predict the personality traits without needing other parts of the network. We evaluate the proposed approach on both self-reported personality and apparent personality datasets. Besides achieving promising results in personality trait estimation from videos, we show that fusion of tasks reaches highest accuracy, and that multi-scale dynamics are more informative than single-scale dynamics.

While we instantaneously recognize a face as attractive, it is much harder to explain what exactly defines personal attraction. This suggests that attraction depends on implicit processing of complex, culturally and individually defined features. Generative adversarial neural networks (GANs), which learn to mimic complex data distributions, can potentially model subjective preferences unconstrained by pre-defined model parameterization. Here, we present generative brain-computer interfaces (GBCI), coupling GANs with brain-computer interfaces. GBCI first presents a selection of images and captures personalized attractiveness reactions toward the images via electroencephalography. These reactions are then used to control a GAN model, finding a representation that matches the features constituting an attractive image for an individual. We conducted an experiment (N=30) to validate GBCI using a face-generating GAN and producing images that are hypothesized to be individually attractive. In double-blind evaluation of the GBCI-produced images against matched controls, we found GBCI yielded highly accurate results. Thus, the use of EEG responses to control a GAN presents a valid tool for interactive information-generation. Furthermore, the GBCI-derived images visually replicated known effects from social neuroscience, suggesting that the individually responsive, generative nature of GBCI provides a powerful, new tool in mapping individual differences and visualizing cognitive-affective processing.

Self-supervised learning has attracted plenty of recent research interest. However, most works for self-supervision in speech are typically unimodal and there has been limited work that studies the interaction between audio and visual modalities for cross-modal self-supervision. This work (1) investigates visual self-supervision via face reconstruction to guide the learning of audio representations; (2) proposes an audio-only self-supervision approach for speech representation learning; (3) shows that a multi-task combination of the proposed visual and audio self-supervision is beneficial for learning richer features that are more robust in noisy conditions; (4) shows that self-supervised pretraining can outperform fully supervised training and is especially useful to prevent overfitting on smaller sized datasets. We evaluate our learned audio representations for discrete emotion recognition, continuous affect recognition and automatic speech recognition. We outperform existing self-supervised methods for all tested downstream tasks. Our results demonstrate the potential of visual self-supervision for audio feature learning and suggest that joint visual and audio self-supervision leads to more informative audio representations for speech and emotion recognition., Accepted for publication in IEEE Transactions on Affective Computing; v3: Publication-ready version including additional experiments and discussion

Although recent emotion recognition methods (based on facial expression cues) achieve excellent performance in controlled scenarios, the recognition of emotion in the wild remains a challenging problem because of occlusion, large head poses, illumination variations, etc. Recent advances in deep learning show that combining an ensemble of deep learning models can considerably outperform the approach of using only a single deep learning model for challenging recognition problems. This paper presents a novel ensemble deep learning method, deep convolutional neural network (DCNN) ensemble classifier, for improved facial expression recognition (FER) in the wild. Our proposed DCNN ensemble classifier is novel in terms of the following aspects: (1) the process of finding ensemble weights for combining DCNN decision outputs is formulated as a stochastic optimization problem (via simulated annealing) in which the energy to be minimized represents the generalized (test) classification error of the DCNN ensemble classifier and (2) for the creation of DCNN members in an ensemble, we propose the combined use of different types of face representations and bagging [16], which is quite useful in increasing the diversity of the DCNN ensemble. Extensive and comparative experiments on three wild FER datasets, namely FER2013, SFEW2.0, and RAF-DB, show that the proposed DCNN ensemble classifier achieves competitive FER performances when compared with other recently developed methods76.69%, 58.68%, and 87.13% of FER accuracy under the FER2013, SFEW2.0, and RAF-DB evaluation protocols, respectively.

Detecting salient objects in videos is a very challenging task. Current state-of-the-art methods are dominated by motion based deep neural networks, among which optical flow is often leveraged as motion representation. Though with robust performance, these optical flow-based video salient object detection methods face at least two problems that may hinder their generalization and application. First, computing optical flow as a pre-processing step does not support direct end-to-end learning; second, little attention has been given to the quality of visual features due to high computational cost of spatiotemporal feature encoding. In this paper we propose a novel self-sufficient feature enhancing network (SFENet) for video salient object detection, which leverages optical flow estimation as an auxiliary task while being end-to-end trainable. With a joint training scheme of both salient object detection and optical flow estimation, its multi-task architecture can be totally self-sufficient for achieving good performance without any pre-processing. Furthermore, for improving feature quality, we design four lightweight modules in spatial and temporal domains, including cross-layer fusion, multi-level warping, spatial-channel attention and boundary-aware refinement. The proposed method is evaluated through extensive experiments on five video salient object detection datasets. Experimental results show that our SFENet can be easily trained with fast convergence speed. It significantly outperforms previous methods in terms of various evaluation metrics. Moreover, with optical flow estimation and unsupervised video object segmentation as example applications, our method also yields state-of-the-art results on standard datasets.

For face-related applications like face recognition and face verification, face detection is vital. Law enforcement agencies extensively use face recognition applications for the detection of known offenders. Security cameras are increasingly deployed everywhere, and these cameras’ captured data can be used for tagging offenders. This is done by offering the face detection system with input images, which are then processed. Later on, attempts will be done to tag the security cameras’ captured data with this processed input data. Effective utilization of deep learning techniques in order to efficiently carry out this task. Computer vision’s various tasks have been taken over by deep Convolutional Neural Networks (CNNs). Of late, the key paradigms for face detection are the region-based CNN (R-CNN) detection models due to their quicker processing speed and progressively better performance. Nevertheless, the huge feature set causes these models’ to be inefficient for real-time face detection. A Grammatical Evolution (GE) Region-Based Fully Conventional Neural Network (R-FCN) for face detection has been proposed in this work in which the GE heuristics are utilized for tuning the hyper-parameters. The WIDER face dataset and Face Detection Dataset and Benchmark (FDDB) dataset were employed for conducting the assessments. In comparison to other advanced paradigms, the proposed technique’s superiority is evident from the experiments conducted.

The Micro-expressions (MEs) are very important nonverbal communication clues. However, due to their local and short nature, spotting them is challenging. In this paper, we address this problem by using a dedicated local and temporal pattern (LTP) of facial movement. This pattern has a specific shape (S-pattern) when ME are displayed. Thus, by using a classical classification algorithm (SVM), MEs are distinguished from other facial movements. We also propose a global final fusion analysis on the whole face to improve the distinction between ME (local) and head (global) movements. However, the learning of S-patterns is limited by the small number of ME databases and the low volume of ME samples. Hammerstein models (HMs) are known to be a good approximation of muscle movements. By approximating each S-pattern with a HM, we can both filter outliers and generate new similar S-patterns. By this way, we perform a data augmentation for S-pattern training dataset and improve the ability to differentiate micro-expressions from other facial movements. The spotting results, performed on CASME and CASME, show that our proposed LTP outperforms the most popular spotting method in terms of F1-score. Adding the fusion process and data augmentation improve even more the spotting performance.

Emotional expressivity has always been a simple job for people, but computer programming is much harder to accomplish. Image emotions may be recognised by recent developments in computer vision and machine learning. In this article, we present a new method to detect face emotion. Use neural networks convolutionary (FERC). The FERC is based on a CNN network of two parts: the first portion removed the backdrop of the image, the second part removed the face vector. The expressional vector (EV) is utilised in the FERC model to detect the fve different kinds of regular facial expressions. The double-level CNN is continuous and the weights and exponent values of the final perception layer vary with each iteration. In that it improves accuracy, FERC varies from widely utilised CNN single-level technology. Moreover, EV generation prevents the development of a number of issues before a new background removal process is used (for example distance from the camera).

Face recognition is one of the useful tasks and can be used for many applications as security systems, it is necessary to find effective and low complexity facial classifier methods. In this paper, we proposed a new one-dimensional CNN deep convolutional neural network (1D-DCNN) classifier was combined with linear discriminative analysis (LDA) techniques to produce a new face recognition methodology. The contribution of this paper is generated of one-dimensional face feature set by LDA from original image database to training of 1D-DCNN classifier, that it contributed in the improvement of facial recognition performance. The model has been tested on MCUT dataset consisted of 3755 images for 276 classes. The results of the implementation of face recognition were accuracy of 100%, precision of 100%, recall of 100%, and F-measure of 100%.

This paper elucidates diverse algorithms that are used in drones for face identification. Face identification is the thriving technology that utilizes image processing for the identification of human faces. The main reasons for the booming interest in face recognition include the rise in population which demands high security and surveillance systems, requisite for identity verification in the digital world, for the combat in rural areas, alleviation during disasters, and so on. The goal of this paper is to explore, compare and contrast numerous face identification algorithms such as Linear Discriminant Analysis (LDA), Local Binary Pattern Histogram (LBPH), Principal Component Analysis (PCA), Elastic Bunch Graph Matching (EBGM) and neural networks. This study would assist the technologists in the field of face recognition to frame a hybrid algorithm based on the needs of the real time applications.

Learning discriminative features is of vital importance for automatic Facial Expression Recognition (FER) in the wild. In this paper, we propose a novel Slide-Patch and Whole-Face Attention model with SE blocks (SPWFA-SE), which jointly perceives the discriminative locality characteristics and informative global features of the face for effective FER. Specifically, the well-designed slide patches are proposed to extract local features. Different from the existing methods, our slide patches not only can maintain the information at the edge area of patches, but also do not need to detect facial landmarks. Moreover, to make the model adaptively focus on the distinguishable regions, an attention module is proposed in the patch level to learn the weight of each patch. Furthermore, squeeze-and-excitation blocks are explored in the channel level to learn the weight of each channel. As such, the proposed multi-level feature extraction and attention mechanisms can enhance the representative ability of the learned features. Extensive experiments on five challenging datasets demonstrate that our method can achieve state-of-the-art performance. Cross database experiments on another three databases show the superior generalization performance of our model. Furthermore, complexity analysis results show that our model contains fewer parameters with fast training advantages than other competing models.

Nowadays criminal activities are growing at an exponential rate. Crime prevention by effective identification of criminals is the main issue before the police and on the other hand, the availability of police officers is inadequate. There are various technological solutions for detecting criminals but they are not up to the mark. In this project, a face detection and recognition system for criminal identification is developed using the multi-task cascade neural network. This system will be able to detect faces and recognize faces of criminals automatically in real-time. This system would also just require a single image of the criminal to recognize him, also known as one-shot learning. The purpose is to identify the criminal face, retrieve the information stored in the database for the identified criminal and a notification is sent to the police personnel with all the details and the location at which he was under the surveillance of the camera.

Over the last years, many face analysis tasks have accomplished astounding performance, with applications including face generation and 3D face reconstruction from a single "in-the-wild" image. Nevertheless, to the best of our knowledge, there is no method which can produce render-ready high-resolution 3D faces from "in-the-wild" images and this can be attributed to the: (a) scarcity of available data for training, and (b) lack of robust methodologies that can successfully be applied on very high-resolution data. In this work, we introduce the first method that is able to reconstruct photorealistic render-ready 3D facial geometry and BRDF from a single "in-the-wild" image. We capture a large dataset of facial shape and reflectance, which we have made public. We define a fast facial photorealistic differentiable rendering methodology with accurate facial skin diffuse and specular reflection, self-occlusion and subsurface scattering approximation. With this, we train a network that disentangles the facial diffuse and specular BRDF components from a shape and texture with baked illumination, reconstructed with a state-of-the-art 3DMM fitting method. Our method outperforms the existing arts by a significant margin and reconstructs high-resolution 3D faces from a single low-resolution image, that can be rendered in various applications, and bridge the uncanny valley., Project and Dataset page: ( https://github.com/lattas/AvatarMe ). 20 pages, including supplemental materials. Accepted for publishing at IEEE Transactions on Pattern Analysis and Machine Intelligence on 13 November 2021. Copyright 2021 IEEE. Personal use of this material is permitted

Realistic speech-driven 3D facial animation is a challenging problem due to the complex relationship between speech and face. In this paper, we propose a deep architecture, called Geometry-guided Dense Perspective Network (GDPnet), to achieve speaker-independent realistic 3D facial animation. The encoder is designed with dense connections to strengthen feature propagation and encourage the re-use of audio features, and the decoder is integrated with an attention mechanism to adaptively recalibrate point-wise feature responses by explicitly modeling interdependencies between different neuron units. We also introduce a non-linear face reconstruction representation as a guidance of latent space to obtain more accurate deformation, which helps solve the geometry-related deformation and is good for generalization across subjects. Huber and HSIC (Hilbert-Schmidt Independence Criterion) constraints are adopted to promote the robustness of our model and to better exploit the non-linear and high-order correlations. Experimental results on the public dataset and real scanned dataset validate the superiority of our proposed GDPnet compared with state-of-the-art model. The code is available for research purposes at http://cic.tju.edu.cn/faculty/likun/projects/GDPnet.

Face hallucination technologies have been widely developed during the past decades, among which the sparse manifold learning (SML)-based approaches have become the popular ones and achieved promising performance. However, these SML methods always failed in handling noisy images due to the least-square regression (LSR) they used for error approximation. To this end, we propose, in this article, a smooth correntropy representation (SCR) model for noisy face hallucination. In SCR, the correntropy regularization and smooth constraint are combined into one unified framework to improve the resolution of noisy face images. Specifically, we introduce the correntropy induced metric (CIM) rather than the LSR to regularize the encoding errors, which admits the proposed method robust to noise with uncertain distributions. Besides, the fused LASSO penalty is added into the feature space to ensure similar training samples holding similar representation coefficients. This encourages the SCR not only robust to noise but also can well exploit the inherent typological structure of patch manifold, resulting in more accurate representations in noise environment. Comparison experiments against several state-of-the-art methods demonstrate the superiority of SCR in super-resolving noisy low-resolution (LR) face images.

Recently, a popular line of research in face recognition is adopting margins in the well-established softmax loss function to maximize class separability. In this paper, we first introduce an Additive Angular Margin Loss (ArcFace), which not only has a clear geometric interpretation but also significantly enhances the discriminative power. Since ArcFace is susceptible to the massive label noise, we further propose sub-center ArcFace, in which each class contains $K$ sub-centers and training samples only need to be close to any of the $K$ positive sub-centers. Sub-center ArcFace encourages one dominant sub-class that contains the majority of clean faces and non-dominant sub-classes that include hard or noisy faces. Based on this self-propelled isolation, we boost the performance through automatically purifying raw web faces under massive real-world noise. Besides discriminative feature embedding, we also explore the inverse problem, mapping feature vectors to face images. Without training any additional generator or discriminator, the pre-trained ArcFace model can generate identity-preserved face images for both subjects inside and outside the training data only by using the network gradient and Batch Normalization (BN) priors. Extensive experiments demonstrate that ArcFace can enhance the discriminative feature embedding as well as strengthen the generative face synthesis., Comment: ArcFace TPAMI version

Cross aging face recognition ability will decrease to recognize someone's face after a certain time. Adding synthetics face images at a certain age generated from face aging architecture is one way to increase the performance of cross aging face recognition. A synthetics face image can create use the Generative Adversarial Network-based architecture. The current Generative Adversarial Network-based in face aging still needs high computation to create a model. Based on that reason we proposed a new optimal variant of Identity Preserving Conditional Generative Adversarial Network (IPCGAN), to generate a synthetic face image at certain age groups. In the proposed architecture, change made at the structure in the generator module, age classification module, and change the objective function to increasing the accuracy performance when generates a realistic synthetic face image in certain age groups and also speed up the training time. Modification in the age classifier at the proposed network forces our architecture to generate better synthetics face in certain age groups. Evaluation using Facenet, and Age prediction shows our method accuracy has 4.2% better results in k-NN classification, 3.6% better accuracy results in SVM classification, 8.6% better accuracy result in age verification, and 4.5% fewer accuracy results in age prediction.

Automatic facial emotion recognition in real-world situations like partial occlusions, varying head poses and illumination conditions are challenging to the machine learning community. The main reason is the lack of sufficient samples with the aforementioned conditions in the baseline datasets which throws difficulty in training a well performing machine learning or deep learning model. To overcome this challenge, we have adopted the concept of meta-learning. Meta-learning using prototypical networks (metric-based meta-learning) has been proven to be well-fit for few-shot problems without severe overfitting. We leverage the quick adaptation power of prototypical networks for emotion recognition in the scarcity of such diverse samples. We have used CMU Multi-PIE dataset which contains images with partial occlusions, varying head-poses and illumination levels for training and evaluating the model. For testing the adaptability of the system to intra-class and inter-dataset variations, AffectNet face database images have been used. The proposed method is named as ERMOPI (Emotion Recognition using Meta-learning across Occlusion, Pose and Illumination) which performs emotion recognition from facial expressions using meta-learning approach for still images and it is robust to partial occlusions, varying head poses and illumination levels which is the novelty of this work. The key benefit is the usage of less number of training samples compared to the existing work in emotion recognition and achieved comparable results with the state-of-the-art approaches. The proposed method achieved 90% accuracy for CMU Multi-PIE database images and 68% accuracy for AffectNet database images.

We propose a method for detecting face swapping and other identity manipulations in single images. Face swapping methods, such as DeepFake, manipulate the face region, aiming to adjust the face to the appearance of its context, while leaving the context unchanged. We show that this modus operandi produces discrepancies between the two regions (e.g., Fig. 1). These discrepancies offer exploitable telltale signs of manipulation. Our approach involves two networks: (i) a face identification network that considers the face region bounded by a tight semantic segmentation, and (ii) a context recognition network that considers the face context (e.g., hair, ears, neck). We describe a method which uses the recognition signals from our two networks to detect such discrepancies, providing a complementary detection signal that improves conventional real vs. fake classifiers commonly used for detecting fake images. Our method achieves state of the art results on the FaceForensics++, Celeb-DF-v2, and DFDC benchmarks for face manipulation detection, and even generalizes to detect fakes produced by unseen methods.

The image-based virtual illumination technology directly changes the illumination effect of objects on the image. It does not require complex light propagation simulation calculations, it is an image-based rendering technology. The image mainly relies on the imaging of the visual sensor, and uses the virtual illumination technology of the image to illuminate the image obtained by the visual sensor. This is a new cross-cutting direction in computer vision, virtual reality and other fields. Face Illumination Swapping via Dense Landmark and Semantic Parsing is a major branch. Keeping the geometrical features of the target images and relighting the entire image instead of the face area are problems to be solved in the research. This paper based on the three-dimensional model to analyze the illumination information of the face images and re-render the illumination of the target face, and finally achieves the illumination swap between the two face images. We designed and implemented a 3DDFA-based face image illumination transfer method. First, 3DDFA is used to reconstruct the target face image. Estimate the surface normal and albedo. Then align and fill the surface normal and face parsing to illuminate the face image for light rendering and illumination transfer of the face images. Finally, the illumination analysis and re-rendering of face images based on 3DDFA are expanded to achieve the swap of illumination between face images. Experimental results show that this method can generate good effects of face image illumination transmission and swap while keeping the geometric features of the target images.

Research on facial expression recognition has been moving from the constrained lab scenarios to the in-the-wild situations and has made progress in recent years. However, it is still very challenging to deal with facial expression in the wild due to large poses and occlusion as well as illumination and intensity variations. Generally, existing methods mainly take the whole face as a uniform source of features for facial expression analysis. Actually, physiology and psychology research shows that some crucial regions such as the eye and mouth reflect the differences of different facial expressions, which have close relationships with emotion expression. Inspired by this observation, a novel relation-aware facial expression recognition method called Relation Convolutional Neural Network (ReCNN) is proposed in this paper, which can adaptively capture the relationship between crucial regions and facial expressions leading to the focus on the most discriminative regions for recognition. We have evaluated the proposed ReCNN on two large in-the-wild databases: AffectNet and RAF-DB. Extensive experiments on these databases show that our method has superior recognition accuracy compared with state-of-the-art methods and the relationship between crucial regions and facial expressions is beneficial to improve the performance of facial expression recognition.

Typical censoring models have mass points at the upper or lower tails, or at both tails, of an otherwise continuous outcome distribution. In contrast, we consider a censoring model with a mass point in the interior of the outcome distribution. We refer to this mass point as “bunching” and use it to estimate model parameters. For example, economic theory suggests that, for increasing marginal income tax rates, many taxpayers will report income exactly at the threshold where the tax rate increases. This translates into a censoring model with bunching at the threshold. The size of this mass point of taxpayers can be used to estimate an elasticity parameter that summarizes taxpayers’ responses to taxes. In this article, we introduce the command bunching, which implements new nonparametric and semiparametric identification methods for estimating elasticities developed by Bertanha, McCallum, and Seegert (2021, Technical Report 2021-002, Board of Governors of the Federal Reserve System). These methods rely on weaker assumptions than what are currently made in the literature and result in meaningfully different estimates of the elasticity.

The disclosure of face image features can seriously threaten the security of user information, which limits the application of face recognition technology in Internet of Vehicles. This paper proposes a new method of generating and restoring private face images based on semantic features and adversarial examples. The Segnet network first segments the face images semantically, then a generative adversarial network generates adversarial examples and perturbs the semantic features of the face image. The perturbation position can be accurately controlled through a coefficient matrix as the identity tag of the face image is concealed steganographically. A restoration network is trained to extract the real identity tag from the private face image using a discriminator against the generation network, then it restores the private face image to its original state. Compared to other state-of-the-art methods, private face images generated by the proposed method experimentally show high detection resistance, better quality, and stronger median filtering defense.

The head-mounted displays (HMDs) provide immersive experiences in virtual reality (VR). However, the face interactions are limited due to the serious occlusion of the user’s face. Existed approaches try to recover the user’s facial expression by adding additional sensors to HMD. In this paper, we develop a novel framework to reconstruct the userb’s 3D face in VR only using an RGB camera. Given a reference face, a realistic full-textured avatar is created by fitting a 3D Morphable Model (3DMM). A self-supervised UV map Generative Adversarial Network (GAN) is proposed to make the facial texture look more realistic. Next, we propose a novel landmark detection method to locate the landmark positions under HMD occlusion since facial landmarks are commonly used for driving the avatar. To this end, we synthesize a face dataset with HMD. Our method is easy to build and popularize with low cost. The experiments on the synthetic and real HMD data demonstrate that the proposed method can detect landmark accurately and restore facial expressions faithfully despite the large occlusion of HMD.

Realistic 3D facial modeling and animation have been increasingly used in many graphics, animation, and virtual reality applications. However, generating realistic fine-scale wrinkles on 3D faces, in particular, on animated 3D faces, is still a challenging problem that is far away from being resolved. In this article we propose an end-to-end system to automatically augment coarse-scale 3D faces with synthesized fine-scale geometric wrinkles. By formulating the wrinkle generation problem as a supervised generation task, we implicitly model the continuous space of face wrinkles via a compact generative model, such that plausible face wrinkles can be generated through effective sampling and interpolation in the space. We also introduce a complete pipeline to transfer the synthesized wrinkles between faces with different shapes and topologies. Through many experiments, we demonstrate our method can robustly synthesize plausible fine-scale wrinkles on a variety of coarse-scale 3D faces with different shapes and expressions.

Privacy concerns of using sensitive biometric data as credentials arise with the wide adoption of user-friendly face authentication. To protect the facial features of users, two important functions, i.e. revocability and reusability, are anticipated to be realized in a privacy-preserving face authentication design. Revocability requires an effective approach to deregister or replace user credentials when the authentication server is compromised; For reusability, the same credentials should appear independently t

Traffic accident may occur at any time, and most of the time is because while driving a car, the driver is tired. Road wreck can be prevented if pilot of the vehicle is conscious of the state and circumstance of caution on the lane. There are many technologies for development today, one of which is Technique of detecting the objects. The technology for detecting the object requires to detect a detailed facial recognition. Researchers develop a system that is capable of detecting the drowsiness to alert the pilot when the pilot appears sleepy while driving a vehicle, this may avoid crash. Pilot that is the consumer in this case, is they shout-off the eye in seconds, the sensors that is the smartphone's front camera can catch and analyze this occurrence and then activate the device to warn the user to speech. Authors use OpenCV Library to process the detected file. OpenCV Library uses the Haar Cascade Classifier to classify features like face and eyes. In this method, the eye and facial recognition are targetted. On the Android Operating System, this program will be introduced.

The topology of isosurfaces changes at isovalues of critical points, making such points an important feature when building contour trees or Morse-Smale complexes. Hexahedral elements with linear interpolants can contain additional off-vertex critical points in element bodies and on element faces. Moreover, a point on the face of a hexahedron which is critical in the element-local context is not necessarily critical in the global context. In ‘`Exploring Scalar Fields Using Critical Isovalues’' Weber et al. introduce a method to determine whether critical points on faces are also critical in the global context, based on the gradient of the asymptotic decider in each element that shares the face. However, as defined, the method of Weber et al. contains an error, and can lead to incorrect results. In this work we correct the error.

User authentication on smartphones is the key to many applications, which must satisfy both security and convenience. We propose a novel user authentication system EchoPrint, which leverages acoustics and vision for secure and convenient user authentication, without requiring any special hardware. EchoPrint actively emits almost inaudible acoustic signals from the earpiece speaker to illuminate the user's face and authenticates the user by the unique features extracted from the echoes bouncing off the 3D facial contour. To combat changes in phone-holding poses thus echoes, a Convolutional Neural Network (CNN) is trained to extract reliable acoustic features, which are further combined with visual facial features extracted from state-of-the-art face recognition deep models to feed a binary Support Vector Machine (SVM) classifier for final authentication. Because the echo features depend on 3D facial geometries, EchoPrint is not easily spoofed by images or videos like 2D visual face recognition systems. It needs only commodity hardware, thus avoiding the extra costs of special sensors in solutions like FaceID. Experiments with 62 volunteers and non-human objects such as images, photos, and sculptures show that EchoPrint achieves 93.75% balanced accuracy and 93.50% F-score, while the average precision is 98.05% using acoustic features and basic facial landmarks. The precision is further improved to 99.96% with sophisticated visual features.

In this paper, ultra-sonic sensors along with kinematics and dynamics equations of a towing vehicle are used to develop three approaches for hitch angle estimation. The first approach is based on direct calculation of hitch angle using certain a priori geometric information and distance measurements of four ultra-sonic sensors. An angle estimate is generated for each of the six possible sensor-pair combinations, and these six estimates are passed through a voting algorithm to produce a single estimate. As the second and third approaches, kinematic and dynamic models of the tractor-trailer system are used to develop least-squares and Kalman filtering based recursive hitch angle estimations. A more reliable hitch angle estimation scheme is then proposed as the integration of the algorithms developed following each of the three approaches via a switching data fusion logic. It is shown that the proposed integrated hitch angle estimation scheme can be used for any ball type box trailer with a flat or symmetric V-nose frontal face without any priori information on the trailer parameters. Based on the validity of the assumptions, the proposed scheme can estimate the hitch angle in both low-speed using the kinematic model, and high-speed using the dynamic model of the tractor-trailer system. Experimental results corroborate the algorithms in estimating the hitch angle estimates in various cases conducted in this study.

Facial expression recognition has been studied broadly, and several works using local micro-pattern descriptors have obtained significant results. There are, however, open questions: how to design a discriminative and robust feature descriptor?, how to select expression-related most influential features?, and how to represent the face descriptor exploiting the most salient parts of the face? In this paper, we address these three issues to achieve better performance in recognizing facial expressions. First, we propose a new feature descriptor, namely Local Shape Pattern (LSP), that describes the local shape structure of a pixel's neighborhood based on the prominent directional information by analyzing the statistics of the neighborhood gradient, which allows it to be robust against subtle local noise and distortion. Furthermore, we propose a selection strategy for learning the influential codes being active in the expression affiliated changes by selecting them exhibiting statistical dominance and high spatial variance. Lastly, we learn the size of the salient facial blocks to represent the facial description with the notion that changes in expressions vary in size and location. We conduct person-independent experiments in existing datasets after combining above three proposals, and obtain an improved performance for the facial expression recognition task.

Facial authentication has become more and more popular on personal devices. Due to the ease of use, it has great potential to be widely deployed for web-service authentication in the near future whereby people can easily log on to online accounts from different devices without memorizing lengthy passwords. However, the growing number of attacks on machine learning especially the Deep Neural Networks (DNN) which is commonly used for facial recognition, imposes big challenges on the successful roll-out of such web-service face authentication. Although there have been studies on defending some machine learning attacks, we are not aware of any specific effort devoted to the web-service facial authentication setting. In this paper, we first demonstrate a new data poisoning attack which does not require to have any knowledge of the server-side and just needs a handful of malicious photo injections to enable an attacker to easily impersonate the victim in the existing facial authentication systems. We then propose a novel defensive approach called DEFEAT that leverages deep learning techniques to automatically detect such attacks. We have conducted extensive experiments on real datasets and our experimental results show that our defensive approach achieves more than 90% detection accuracy.

Head pose estimation is an important problem as it facilitates tasks such as gaze estimation and attention modeling. In the automotive context, head pose provides crucial information about the driver's mental state, including drowsiness, distraction and attention. It can also be used for interaction with in-vehicle infotainment systems. While computer vision algorithms using RGB cameras are reliable in controlled environments, head pose estimation is a challenging problem in the car due to sudden illumination changes, occlusions and large head rotations that are common in a vehicle. These issues can be partially alleviated by using depth cameras. Head rotation trajectories are continuous with important temporal dependencies. Our study leverages this observation, proposing a novel temporal deep learning model for head pose estimation from point cloud. The approach extracts discriminative feature representation directly from point cloud data, leveraging the 3D spatial structure of the face. The frame-based representations are then combined with bidirectional long short term memory (BLSTM) layers. We train this model on the newly collected multimodal driver monitoring (MDM) dataset, achieving better results compared to non-temporal algorithms using point cloud data, and state-of-the-art models using RGB images. We further show quantitatively and qualitatively that incorporating temporal information provides large improvements not only in accuracy, but also in the smoothness of the predictions.

This paper proposes a method for biometric-driven facial image recognition, based on multivariate correlation analysis, which extracts geometrical feature points and low-level visual features. The low-level visual features, such as colour and texture, are extracted at local from the selected prominent regions of the facial image. The geometrical features are captured using the Active Shape Model (ASM). The colour features are extracted from the YCbCr colour model, and the autocorrelation method is deployed to extract the texture features. The extracted features are formed as a feature tensor matrix. The feature matrix of the key face image is compared to the feature matrices of the target face image that are stored in the feature vector database using the Canonical Correlation method. The correlation between the key and target feature matrices is tested, whether it is highly significant or not. If it is significantly correlated, then it is inferred that the key and target face images are the same; otherwise, it is concluded that they are different. The benchmark facial image datasets, GT, LFW, and Pointing ’04 datasets, considered for the experiments; in addition to the datasets, a facial image database has been constructed with celebrities on our interest which has also been subjected to experiments. The proposed method resulted in a mean precision score (mP@α) of 95.27%, 94.20%, 96.19%, and 96.05 for GT, LFW, Pointing ‘04, and our datasets, respectively. Also, the F-score was calculated that are, 96.78%, 95.15%, 97.08%, and 96.96% for GT, LFW, Pointing ‘04, and our datasets, respectively. The results obtained by the proposed method are comparable to the existing methods.

As face editing scenarios have become popular, the face inpainting technique has become a hot topic. Although some existing methods can inpaint faces with preserved identity information, they fail to solve a more flexible inpainting problem that fills the “holes” with identity-specific content from other faces. In this work, we propose a disentangle and subject-agnostic framework that affects both full and partial-face inpainting with the guidance of a reference face image. The framework consists of an identity encoding module, a content inference module and a generative module. The identity encoding module extracts the identity embedding from the reference image, the content inference module learns to predict the content image, and the generative module integrates the content image and the reference identity embedding to generate the identity-specific inpainted result. To minimize the structure and style gap between the incomplete image and inpainted image, we use a double attribute loss to the generative module and a postprocess of blending operation to the swapped result. We compare our method with state-of-the-art works and demonstrate that our method achieves higher identity similarity and better structural correctness.

Various structural defects deteriorate tunnel operation status and threaten public safety. Current tunnel inspection methods face problems of low efficiency, high equipment expense, and difficult data management. Combining the deep learning model and the 3D reconstruction method based on structure from motion (SfM), this paper proposes a novel SfM-Deep learning method for tunnel inspection. The high-quality 3D tunnel model is constructed by using images taken every 1 m along the longitudinal direction. The instance segmentation of leakage in longitudinal images is realized using the mask region-based convolutional neural network deep learning model. The SfM-Deep learning method projects the texture of the images after defect recognition to the 3D model and realizes the visualization of leakage defects. By projecting the model to the design cylindrical surface and expanding it, the tunnel leakage area is quantified. Through its practical application in a Shanghai metro shield tunnel, the reliability of the proposed method was verified. The novel SfM-Deep learning method can help engineers efficiently carry out intelligent tunnel detection.

Face manipulation techniques improve fast with the development of powerful image generation models. Two particular face manipulation methods, namely face swap and expression reenactment attract much attention for their flexibility and ease to generate high quality synthesis results. Recently, these two subjects are actively studied. However, most existing methods treat the two tasks separately, ignoring their underlying similarity. In this paper, we propose to tackle the two problems within a unified framework that achieves high quality synthesis results. The enabling component for our unified framework is the clean disentanglement of 3D pose, shape, and expression factors and then recombining them for different tasks accordingly. We then use the same set of 2D representations for face swap and expression reenactment tasks that are input to a common image translation model to directly generate the final synthetic images. Once trained, the proposed model can accomplish both face swap and expression reenactment tasks for previously unseen subjects. Comprehensive experiments and comparisons show that the proposed method achieves high fidelity results in multiple aspects, and it is especially good at faithfully preserving source facial shape in the face swap task, and accurately transferring facial movements in the expression reenactment task.

Nowadays, “aging” becomes a challenging problem for face recognition systems. The challenge is particularly due to age-related biological changes, which can lead to significant variations in facial characteristics between two images taken at different ages of the same person. As the face is the most heavily part affected by aging, there is a growing need to extracting robust face features for age-invariant face recognition, particularly, in the presence of large age differences of the same person face images. The aim of this paper is to examine effectiveness of deep-learning based methods as features extraction tool for age-invariant face recognition. In this study, we evaluate five popular pre-trained deep-convolutional neural network (CNN) models that are AlexNet, GoogleNet, Inception V3, ResNet50 and SqueezeNet, on a widely used face-aging database, namely FG-NET, using K-nearest neighbors (K-NN), discriminant analysis, and support vector machines (SVM) classifiers. Further, a statistical analysis test is performed to confirm the statistical significance of the obtained results. Experimental results on this database show the promise of using Convolutional Neural Networks(CNN) for face recognition across age progression. Also the AlexNet model appears to be most promising for age-invariant face recognition, as highest mean accuracy rate is always achieved with feature extraction using the AlexNet model. These results are more significant, according to a 95% confidence level hypothesis test.

With increasing usage of face recognition algorithms, it is well established that external artifacts and makeup accessories can be applied to different facial features such as eyes, nose, mouth, and cheek, to obfuscate one’s identity or to impersonate someone else’s identity. Recognizing faces in the presence of these artifacts comprises the problem of disguised face recognition, which is one of the most arduous covariates of face recognition. The challenge becomes exacerbated when disguised faces are captured in real-time environment, with low resolution images. To address the challenge of disguised face recognition, this paper first proposes a novel multi-objective encoder-decoder network, termed as DED-Net. DED-Net attempts to learn the class variations in the feature space generated by both disguised as well non-disguised images, using a combination of Mahalanobis and Cosine distance metrics, along with Mutual Information based supervision. The DED-Net is then extended to learn from the local and global features of both disguised and non-disguised face images for efficient face recognition, and the complete framework is termed as Disguise Resilient (D-Res) framework. The efficacy of the proposed framework has been demonstrated on two real-world benchmark datasets: Disguised Faces in the Wild (DFW) 2018 and DFW2019 competition datasets. In addition, this research also emphasizes on the importance of recognizing disguised faces in low resolution settings and proposes three experimental protocols to simulate the real-world surveillance scenario. To this effect, benchmark results have been shown on seven protocols for three low resolution settings (32×32, 24×24, and 16×16) of the two DFW benchmark datasets. The results demonstrate superior performance of the D-Res framework, in comparison with benchmark algorithms. For example, an improvement of around 3% is observed on the Overall protocol of the DFW2019 dataset, where the D-Res framework achieves 96.3%. Experiments have also been performed on benchmark face verification datasets (LFW, YTF, and IJB-B), where the D-Res framework achieves improved verification accuracy.

In this paper, we face the reference tracking control problem for a system of heterogeneous Multi Unmanned Vehicles (MUVs) moving in a 2-D planar environment. We consider a scenario where each vehicle follows a trajectory imposed by a local planner and where each UV can have different linear dynamics as well as different constraints and disturbances. In this contest, we design a novel control architecture where a centralized traffic manager, in conjunction with ad-hoc designed local vehicle controllers, is capable of ensuring the absence of collisions. The proposed solution is obtained by exploiting, for the local vehicles' controllers, a dual-mode Model Predictive Controller (MPC) and, for the traffic manager, set-theoretic and controllability properties. Moreover, after modeling the potential vehicle collisions with a graph, connectivity arguments are used to obtain an optimal collision resolution which minimizes the number of vehicles that need to be stopped. The resulting control scheme ensures collision-free signal tracking. Simulation results, conducted on a MUV system are shown to provide tangible evidence of the features of the proposed framework.

Physical attractiveness has an important and great influence in human social life as it was widely linked to the possession of a variety of positive qualities that makes an individual better than the others, as well as offering many advantages to the individual’s life especially in a society obsessed with beauty and attractiveness. As consequences to the attractiveness influence, people tend to look more and more attractive using makeup or in most of the time plastic surgeries. In parallel to the increased need to plastic surgeries, the need to simplify and show how the face will look like after the surgery is recommended, to plan the procedures, verify the satisfaction of the patient, and reduce the risks during the surgery as well. In this paper, we introduce a novel system to analyze and enhance the attractiveness of 3D faces using a Free Form Deformation technique based on the Bezier function. This system was tested on two 3D face datasets and the experimental results show that the edited 3D faces are more attractive compared to the original faces in terms of respecting the beauty canons which are: facial symmetry, golden ration, neoclassical proportions, and the angular profile.

Modifying facial attributes without the paired dataset proves to be a challenging task. Previously, approaches either required supervision from a ground-truth transformed image or required training a separate model for mapping every pair of attributes. These limit the scalability of the models to accommodate a larger set of attributes since the number of models that we need to train grows exponentially large. Another major drawback of the previous approaches is the unintentional gain of the identity of the person as they transform the facial attributes. We propose a method that allows for controllable and identity-aware transformations across multiple facial attributes using only a single model. Our approach is to train a generative adversarial network (GAN) with a multitask conditional discriminator that recognizes the identity of the face, distinguishes real images from fake, as well as identifies facial attributes present in an image. This guides the generator into producing an output that is realistic while preserving the person’s identity and facial attributes. Through this framework, our model also learns meaningful image representations in a lower dimensional latent space and semantically associate separate parts of the encoded vector with both the person’s identity and facial attributes. This opens up the possibility of generating new faces and other transformations such as making the face thinner or chubbier. Furthermore, our model only encodes the image once and allows for multiple transformations using the encoded vector. This allows for faster transformations since it does not need to reprocess the entire image for every transformation. We show the effectiveness of our proposed method through both qualitative and quantitative evaluations, such as ablative studies, visual inspection, and face verification. Competitive results are achieved compared to the main competition (CycleGAN), however, at great space and extensibility gain by using a single model.

Existing face anti-spoofing models using deep learning for multi-modality data suffer from low generalization in the case of using variety of presentation attacks such as 2D printing and high-precision 3D face masks. One of the main reasons is that the non-linearity of multi-spectral information used to preserve the intrinsic attributes between a real and a fake face are not well extracted. To address this issue, we propose a multi-modility data based two-stage cascade framework for face anti-spoofing. The proposed framework has two advantages. Firstly, we design a two-stage cascade architecture that can selectively fuse low-level and high-level features from different modalities to improve feature representation. Secondly, we use multi-modality data to construct a distance-free spectral on RGB and infrared (IR) to augment the non-linearity of data. The presented data fusion strategy is different from popular fusion approaches, since it can strengthen discrimination ability of network models on physical attribute features than identity structure features under certain constraints. In addition, a multi-scale patch based weighted fine-tuning strategy is designed to learn each specific local face region. Experimental results show that the proposed framework achieves better performance than other state-of-the-art methods on both benchmark datasets and self-established datasets, especially on multi-material masks spoofing.

Near-infrared and visual (NIR-VIS) face matching, as the most typical task in Heterogeneous Face Recognition (HFR), has attracted increasing attention in recent years. However, due to the large within-class discrepancies, including domain differences and residual discrepancies (i.e., lighting, expressions, occlusion, blurry, pose, etc), this is still a difficult task. Conventional NIR-VIS FR methods only focus on reducing the modality gap between cross-domain images, while neglecting to eliminate the residual variations. To better solve the above problems, this paper proposes a novel Orthogonal Modality Disentanglement and Representation Alignment (OMDRA) approach, which consists of three key components, including Modality-Invariant (MI) loss, Orthogonal Modality Disentanglement (OMD) and Deep Representation Alignment (DRA). Firstly, the MI loss is designed to learn modality-invariant and identity-discriminative representation, by increasing between-class separability and within-class compactness between NIR and VIS heterogeneous data. Secondly, the high-level Hybrid Facial Feature (HFF) layer of the backbone network is projected into two subspaces: the modality-related and identity-related subspaces. The OMD is designed to decouple modal information via an adversarial process, and we further impose Orthogonal Representation Decorrelation (ORD) to the OMD to decrease the correlation between identity representations and domain representations, as well as enhancing their representation capabilities. Finally, the DRA aims to eliminate the residual variations by performing a high-level representation alignment between non-neutral face and neutral face, which can effectively guides the network to learn discriminative and residual-invariant face representation. The joint scheme enables the disentanglement of modality variations, elimination of residual discrepancies, and the purification of identity information. Extensive experiments on challenging cross-domain databases indicate that our OMDRA method is superior to the state-of-the-art methods.