Search

Facial expression recognition finds extensive applications in human-computer interaction, education, and autonomous driving. Nevertheless, occlusion and pose changes during expression recognition can lead to the loss of information about different facial parts, obfuscation, and the disruption of local relationships, frequently resulting in a decline in recognition accuracy. Existing research methods encounter difficulties in directing attention to affected areas and capturing decisive features and their global relationships. To tackle these issues, this paper presents an Expression Recognition model known as ADMF-ER (Expression Recognition model fusing Adaptive Dropout and Multi-level Features), which integrates hierarchical and global features. Through hierarchical and global feature modeling, the model enhances its ability to recognize facial expressions under complex conditions such as occlusion and pose changes. It constructs a Multi-scale Residual Attention (MSRA) module to extract features at different levels and scales, with the aim of learning rich semantic and geometric features. Additionally, the model incorporates spatial and channel attention mechanisms, enabling the network to selectively focus on and emphasize affected local areas and capture key features. Finally, global features and relationships are learned via the vision Transformer structure and dynamically pruned using Dropout. As training advances, the pruning intensity increases, facilitating a better understanding of overall facial expressions. Experimental results indicate that the method proposed in this paper achieves remarkably superior performance on RAF-DB, FERPlus, and AffectNet datasets. At the same time, the method demonstrates excellent robustness in the evaluation of CK+ across datasets. [ABSTRACT FROM AUTHOR]

In learning environment, instructor's emotions may impact the relationship with students. A wide range of research has been conducted on the facial expression of the student, but the influence of teacher's facial expression remains an unexamined area. Facial expression detection has the ability to detect the impact of an instructor's emotions in a learning environment. During lecture delivery, manual evaluation of the instructor behavior not only improves the learning environment, but can also save time and resources used in manual techniques. Therefore, this research work introduces facial emotion recognition of an instructor in a learning environment with the help of Evolving Hypergraph Manta ray Foraging Convolution Neural Network (EHMFCNN). Initially, the collected lecture videos are converted into frames, and the feature extraction is performed by fast discrete curvelet with wrapping technique. Then, to perform an effective recognition process, the frames are classified into multimedia representation, code walkthrough sessions and instructor presence in the lecture using dense residual dual-shuffle attention network. After classification, the instructor-based frames are selected for further recognition process. Then, evolving hypergraph convolution neural network is utilized to recognize the emotions of the instructor, and manta ray foraging optimization algorithm is employed for weight tuning process. Finally, the proposed system provides appropriate feedback for instructors using recognized emotions. The EHMFCNN system is tested on NPTEL-Indian English speech dataset, and it achieves highest accuracy of 97.5%, which is better than conventional classifiers. Extensive research determines that the introduced scheme improves the performance of the instructor emotion recognition system. [ABSTRACT FROM AUTHOR]

Convolutional neural networks (CNNs) are essentially applicable for feature extraction of images due to their robust mapping ability. However, due to the locality of CNNs, most of the existing methods that utilize CNNs for facial expression recognition (FER) cannot be well-suited for face images with occlusions or pose variations in the wild. Although our previous attention-modulated contextual spatial information network (ACSI-Net) is effective for FER, the loss function in ACSI-Net is not able to sufficiently constrain the network for the perception of discriminative facial features. In this paper, a novel constraining function called joint loss, tailored for FER in the wild, is offered to supervise ACSI-Net learning. In the Taylor expansion of the cross-entropy loss, the relative weights of the polynomial basis are adjusted to increase the learning intensity for categories with fewer samples, which produces our equilibrium loss. In addition, the regularization term is defined as the Euclidean distance between features and their class centers, with attention masking, which is named sparse center loss. Benefiting from the combination of equilibrium loss and sparse center loss, the joint loss can explore features with inter-class separateness and intra-class compactness from unbalanced data, leading to improved recognition performance. Experiments demonstrate that the presented joint loss, constraining ACSI-Net, achieves superior performance to ACSI-Net and is competitive with relevant advanced methods on two benchmark datasets, RAF_DB and AffectNet. [ABSTRACT FROM AUTHOR]

Existing facial expression recognition (FER) methods are mainly devoted to learning discriminative features from normal-light images. However, their performance drops sharply when they are used for low-light images. In this paper, we propose a novel low-light FER framework (termed LL-FER) that can simultaneously enhance the images and recognition tasks of low-light facial expression images. Specifically, we first meticulously design a low-light enhancement network (LLENet) to recover expressions images’ rich detail information. Then, we design a joint loss to train the LLENet with FER network in a cascade manner, so that the FER network can guide the LLENet to gradually perceive and restore discriminative features which are useful for FER during the training process. Extensive experiments show that the LLENet not only achieves competitive results both quantitatively and qualitatively, but also in the LL-FER framework, which can produce results more suitable for FER tasks, further improving the performance of the FER methods. [ABSTRACT FROM AUTHOR]

Face expression recognition in the wild faces challenges such as small data size, low quality images, and noisy labels. In order to solve these problems, this paper proposes a novel structural self-contrast learning method (SSCL) based on adaptively weighted negative samples. Firstly, two augmented variants are generated for each input sample as positive samples, which should be as similar in structure as possible. Secondly, EMD (Earth's Mover Distance) is used to define the structural similarity between the two augmented samples and extract the features in a self-supervised way. Subsequently, the concept of negative samples is further introduced into SSCL, and the augmented variants of other categories are generated as negative samples, which makes the structural similarity between positive samples increase and between negative samples simultaneously decrease. Finally, an adaptive weighting method for the negative is proposed, which aims to solve the imbalance between positive samples and negative samples, mine difficult negative samples, and reduce their similarity with the input image in an adaptive way. The extensive experiments on benchmark datasets demonstrate that SSCL outperforms the state-of-the-art methods. The cross-dataset evaluation shows the superior generalization capability of the proposed method. The code will be available at https://github.com/zjhzhy/SSCL. [ABSTRACT FROM AUTHOR]

In the field of facial expression recognition (FER), two main trends point to the data-driven FER and feature-driven FER exist. The former focused on the data problems (e.g., sample imbalance and multimodal fusion), while the latter explored the facial expression features. As the feature-driven FER is more important than the data-driven FER, for deeper mining of facial features, we propose an expression recognition model based on Local–Global information Reasoning and Landmark Spatial Distributions. Particularly to reason local–global information, multiple attention mechanisms with the modified residual module are designed for the Res18-LG module. In addition, taking the spatial topology of facial landmarks into account, a topological relationship graph of landmarks and a two-layer graph neural network are introduced to extract spatial distribution features. Finally, the experiment results on FERPlus and RAF-DB datasets demonstrate that our model outperforms the state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Facial expression recognition (FER) is a tedious task in image processing for complex real-world scenarios that are captured under different lighting conditions, facial obstructions, and a diverse range of facial orientations. To address this issue, a novel Twinned attention network (Twinned-Att) is proposed in this paper for an efficient FER in occluded images. The proposed Twinned-Att network is designed in two separate modules: Holistic module (HM) and landmark centric module (LCM). The holistic module comprises of dual coordinate attention block (Dual-CA) and the Cross Convolution block (Cross-conv). The Dual-CA block is essential for learning positional, spatial, and contextual information by highlighting the most prominent characteristics in the facial regions. The Cross-conv block learns the spatial inter-dependencies and correlations to identify complex relationships between various facial regions. The LCM emphasizes smaller and distinct local regions while maintaining resilience against occlusions. Vigorous experiments have been undertaken to improve the efficacy of the proposed Twinned-Att. The results produced by the Twinned-Att illustrate the remarkable responses which achieve the accuracies of 86.92%, 85.64%, 78.40%, 69.82%, 64.71%, 85.52%, and 85.83% for the datasets viz., RAF DB, FER PLUS, FER 2013, FED RO, SFEW 2.0, occluded RAF DB and occluded FER Plus respectively. The proposed Twinned-Att network is experimented with various backbone networks, including Resnet-18, Resnet-50, and Resnet-152. It consistently outperforms well and highlights its prowess in addressing the challenges of robust FER in the images captured in complex real-world environments. [ABSTRACT FROM AUTHOR]

This work introduces the representation ensemble learning algorithm, a novel approach for generating diverse unsupervised representations rooted in the principles of self-taught learning. The ensemble comprises convolutional autoencoders (CAEs) learned in an unsupervised manner, fostering diversity via a loss function designed to penalize similar CAEs' latent representations. We employ support vector machines, bagging, and random forest as primary classification methods for the final classification step. Additionally, we incorporate KnoraU, a well-established technique used to dynamically select competent classifiers based on a test sample. We evaluate various fusion strategies, including sum, product, and stacking, to comprehensively assess the ensemble's performance. A robust experimental protocol considering the facial expression recognition problem shows that the proposed approach based on self-taught learning surpasses the accuracy of fine-tuned convolutional neural network (CNN) models. In terms of accuracy, the proposed method is up to 9.9 and 6.3 percentage points better than the CNN-based models fine-tuned for JAFFE and CK+ datasets, respectively. [ABSTRACT FROM AUTHOR]

Noisy labels are unavoidable in facial expression recognition (FER) task, significantly hindering FER performance in real-world scenarios. Recent advances tackle this problem by leveraging uncertainty for sample partitioning or constructing label distributions. However, these approaches primarily depend on labels, leading to confirmation bias issues and performance degradation. We argue that mining both label-independent features and label-dependent information can mitigate the confirmation bias induced by noisy labels. In this paper, we propose MCR, that is, mining simple yet effective label-free consistency regularization (MCR) to learn robust representations against noisy labels. The proposed MCR incorporates three label-free consistency regularizations: instance-level embedding consistency regularization, pairwise distance consistency regularization, and neighbour consistency regularization. Initially, we employ instance-level embedding consistency regularization to learn instance-level discriminative information from identical facial samples under perturbations in an unsupervised manner. This facilitates the efficacy of mitigating inherent noise in data. Subsequently, a pairwise distance consistency regularization is constructed to regularize the classifier and alleviate bias induced by noisy labels. Finally, we use the neighbour consistency regularization to further strengthen the discriminative capability of the model against noise. Benefiting from the advantages of these three label-free consistency regularizations, MCR can learn discriminative and robust representations against noise. Extensive experimental results demonstrate the superior performance of MCR on three popular in-the-wild facial expression datasets, including RAF-DB, FERPlus, and AffectNet. Moreover, MCR demonstrates superior generalization capability on other datasets with noisy labels, such as CIFAR100 and Tiny-ImageNet. [ABSTRACT FROM AUTHOR]

By recognizing students’ facial expressions in actual classroom situations, the students’ emotional states can be quickly uncovered, which can help teachers grasp the students’ learning rate, which allows teachers to adjust their teaching strategies and methods, thus improving the quality and effectiveness of classroom teaching. However, most previous facial expression recognition methods have problems such as missing key facial features and imbalanced class distributions in the dateset, resulting in low recognition accuracy. To address these challenges, this paper proposes LCANet, a model founded on a fused attention mechanism and a joint loss function, which allows the recognition of students’ emotions in real classroom scenarios. The model uses ConvNeXt V2 as the backbone network to optimize the global feature extraction capability of the model, and at the same time, it enables the model to pay closer attention to the key regions in facial expressions. We incorporate an improved Channel Spatial Attention (CSA) module as a way to extract more local feature information. Furthermore, to mitigate the class distribution imbalance problem in the facial expression dataset, we introduce a joint loss function. The experimental results show that our LCANet model has good recognition rates on both the public emotion datasets FERPlus, RAF-DB and AffectNet, with accuracies of 91.43%, 90.03% and 64.43%, respectively, with good robustness and generalizability. Additionally, we conducted experiments using the model in real classroom scenarios, detecting and accurately predicting students’ classroom emotions in real time, which provides an important reference for improving teaching in smart teaching scenarios. [ABSTRACT FROM AUTHOR]

Facial expression recognition (FER) is one of the important research topics in computer vision. It is difficult to obtain high accuracy in FER tasks, especially when the high-quality labeled data are insufficient. Indeed, the facial images with non-frontal faces, occlusions and inaccurate labels heavily affects the training results of FER network models, which causes low recognition accuracy and poor robustness. To this end, we propose a novel strategy for FER tasks through combining transfer learning and generative adversarial network (GAN). First, we enlarge the training datasets by introducing an effective face-cycle GAN to synthesize additional facial expression images. Then, we develop two FER neural networks based on two representative convolutional neural networks (CNN). By transferring the cross-domain knowledge from the two well-trained CNNs to the proposed FER CNNs, it not only obtains more pre-trained knowledge and also accelerates the training process greatly. The experimental results show that the proposed FER CNNs integrated with the new face-cycle GAN achieves high accuracies 98.44%, 95.24% and 91.67% on three widely used datasets CK + , JAFFE, and Oulu-CASIA, respectively. Compared to the results obtained by other state-of-the-art FER methods, the accuracies are improved by 0.34%, 0.24%, and 2.62%, respectively. [ABSTRACT FROM AUTHOR]

This study enhances the generalization ability and recognition accuracy of convolutional neural networks (CNNs) in educational settings, particularly in the task of Facial Expression Recognition (FER), to support effective analysis of classroom teaching behaviors. A novel multimodal teaching behavior analysis model is proposed to achieve this goal, combining the Adaptive Deep Convolutional Neural Network (ADCNN), an Adaptive Entropy Minimization algorithm, and Transfer Learning. The aim is to improve teaching quality through more precise behavior recognition. A Bayesian Network based on a Hybrid Deep Restricted Boltzmann Machine is introduced to enhance the model's performance, enabling effective processing and analysis of multimodal data. The model's performance in the FER task has been extensively validated through experiments, encompassing precise analysis of training and testing data. These methods allow the model to adapt to different datasets automatically and improve overall performance. In addition, the study collects data on classroom engagement and learning interest from Chinese university students through questionnaires, further supporting the model's application in real educational environments. Experimental results show that the proposed model performs exceptionally well in the FER task, with an average training precision of approximately 99.79% and a testing precision of about 99.60%. The maximum difference between training and testing precision is 0.19%. The questionnaire results reveal issues such as low English proficiency and insufficient classroom engagement among students at Heilongjiang Bayi Agricultural University, providing data to support the improvement of teaching methods. The proposed multimodal teaching behavior analysis model provides an effective tool for optimizing classroom teaching by accurately recognizing students' facial expressions and classroom behaviors. [ABSTRACT FROM AUTHOR]

Recognition of facial expressions is a challenging task in computer vision because of the complexity associated with individual facial features and social differences. Early studies classified human facial expressions into six basic categories which are anger, disgust, fear, happiness, sadness and surprise. The neutral expression is also taken into account. Furthermore, compound emotions are explored on human faces which are the representations of the expressions that entail the combination of more than a single basic facial expression. Including at least two expression categories, one is considered as the dominating expression and the other as the complementary expression. In this way, the categorization of compound facial expressions is done. In this study, a novel approach is proposed to recognize compound facial expressions. The main contribution of this paper is the proposed fusion of deep texture and geometric features. The texture features are the deep textures obtained from a deep learning model. The iCV-MEFED dataset is employed. It includes compound facial expressions consisting of all the combinations of basic facial expressions in the sense of dominating and complementary expressions. Therefore, 50 distinct classes of facial expressions are presented. The previous studies carried out on this dataset report high rates of misclassification due to the challenge of the complexity of facial expressions and correlations among the compound expressions. The proposed approach obtained encouraging results and has shown significant improvements in the recognition accuracy of compound facial expressions on the iCV-MEFED dataset. [ABSTRACT FROM AUTHOR]

A customer information system (CIS) is a crucial component of a customer relationship management (CRM) system. The CIS collects, stores, and manages customer data to enhance the customer experience and sales. Deep learning technology has been used to develop facial expression recognition systems that can accurately identify and analyze facial expressions. By integrating facial expression recognition deep learning into a CIS for CRM, businesses can gain a deeper understanding of the emotions and behavior of the customers, allowing them to provide more personalized experiences. The data collected by the CIS can then be used to personalize marketing campaigns, tailor product recommendations, and improve customer service interactions. The integration of facial expression recognition deep learning into a CIS for CRM has the potential to revolutionize the way businesses interact with their customers. This paper constructed a Feature Fusion Deep Multi-Layer Classification (FFDMLC) model for facial expression recognition for the CIS system. With FFDMLC model comprises the feature-fusion model in face recognition. The propsoed FFDMLC approach fuses various features and estimates the feature fusion model. The FFDMLC model employs deep learning for feature computation and expression classification in individuals. The hyperparameters of this proposed method are optimized using the COOT optimization method. This process optimizes reaction conditions, leading to increased yields, minimized by-products, and improved overall efficiency. The CIS model uses the facial expression recognition system for the computation of facial features in the video sequences. The proposed FFDMLC model is evaluated based on the consideration of two datasets such asCK + and the FER2013 dataset, which also exhibits a higher recognition rate of 99.93% for the understanding opinion of the customers. [ABSTRACT FROM AUTHOR]

Facial Expression Recognition (FER) is crucial for human-computer interaction and has achieved satisfactory results on lab-collected datasets. However, occlusion and head pose variation in the real world make FER extremely challenging due to facial information deficiency. This paper proposes a novel Dual Stream Attention Network (DSAN) for occlusion and head pose robust FER. Specifically, DSAN consists of a Global Feature Element-based Attention Network (GFE-AN) and a Multi-Feature Fusion-based Attention Network (MFF-AN). A sparse attention block and a feature recalibration loss designed in GFE-AN selectively emphasize feature elements meaningful for facial expression and suppress those unrelated to facial expression. And a lightweight local feature attention block is customized in MFF-AN to extract rich semantic information from different representation sub-spaces. In addition, DSAN takes into account computation overhead minimization when designing model architecture. Extensive experiments on public benchmarks demonstrate that the proposed DSAN outperforms the state-of-the-art methods with 89.70% on RAF-DB, 89.93% on FERPlus, 65.77% on AffectNet-7, 62.13% on AffectNet-8. Moreover, the parameter size of DSAN is only 11.33M, which is lightweight compared to most of the recent in-the-wild FER algorithms. [ABSTRACT FROM AUTHOR]

Facial expression is the most direct way for human to express emotion and intention, and facial expression recognition (FER) plays a crucial role in machine intelligence. Currently, two common ways to process images includes: viewing facial images as Euclidean structures in space and spliting the images into patches represented with a continuous sequence. However, faces in images are usually irregular, and are not flexible enough to be processed in the form of grid or sequence. Meanwhile, discrete features such as background tend to have a negative impact and interfere with FER when their features are extracted. To address the above problems, an attentional visual graph neural network based FER method (AT-ViG) is proposed in this paper. First, AT-ViG represents the input face images as graphs, and applies pixel-based composition methods to process the input images, to solve the problem of inflexibility data processing methods. Second, AT-ViG employs an attention mechanism to enhance feature extraction, to minimize the influence of irrelevant features such as background and hair in facial images. This further highlights the features that are relevant to FER and suppresses irrelevant features, where the model can effectively capture the correlations between different facial regions, reduce redundant information and improve the accuracy and robustness of expression recognition. Finally, the AT-ViG is experimentally validated on three commonly used expression recognition datasets and achieved the accuracies of 99.60%, 92.03%, 74.35% and 88.15% in CK+, Oulu-CASIA NIR &VIS, FER2013 and RAF-DB, respectively. [ABSTRACT FROM AUTHOR]

Facial expression recognition is vital for human behavior analysis, and deep learning has enabled models that can outperform humans. However, it is unclear how closely they mimic human processing. This study aims to explore the similarity between deep neural networks and human perception by comparing twelve different networks, including both general object classifiers and FER-specific models. We employ an innovative global explainable AI method to generate heatmaps, revealing crucial facial regions for the twelve networks trained on six facial expressions. We assess these results both quantitatively and qualitatively, comparing them to ground truth masks based on Friesen and Ekman's description and among them. We use Intersection over Union (IoU) and normalized correlation coefficients for comparisons. We generate 72 heatmaps to highlight critical regions for each expression and architecture. Qualitatively, models with pre-trained weights show more similarity in heatmaps compared to those without pre-training. Specifically, eye and nose areas influence certain facial expressions, while the mouth is consistently important across all models and expressions. Quantitatively, we find low average IoU values (avg. 0.2702) across all expressions and architectures. The best-performing architecture averages 0.3269, while the worst-performing one averages 0.2066. Dendrograms, built with the normalized correlation coefficient, reveal two main clusters for most expressions: models with pre-training and models without pre-training. Findings suggest limited alignment between human and AI facial expression recognition, with network architectures influencing the similarity, as similar architectures prioritize similar facial regions. [ABSTRACT FROM AUTHOR]

Deep metric-based center loss has been widely used to enhance inter-class separability and intra-class compactness of network features and achieved promising results in facial expression recognition (FER) recently. However, existing center loss does not take the complexity of expression samples into consideration, which deteriorates the representativeness of the generated center vectors resulting in suboptimal performance. To solve this problem, we propose a novel complexity aware center loss for FER. Specifically, a multi-category division module is firstly devised to distinguish simple samples and difficult samples for each category based on the entropy value of sample prediction results. Then, an exact representative center module is employed on simple samples to generate a more representative center vector for each category by encouraging greater differences between different categories. Finally, an adaptive distance adjustment module is proposed to reduce the interference of difficult samples in the model learning process to further improve the accuracy of FER by maintaining a suitable distance between difficult samples and their corresponding center vector. Extensive experimental results on two benchmark datasets demonstrate the effectiveness, universality and superiority of our methods. The code will be available at https://github.com/sanjiaobo/CACL. [ABSTRACT FROM AUTHOR]

To better apply deep convolutional neural networks for expression recognition in UAV-enabled B5G/6G networks, we propose a deep network expression recognition method based on transfer learning and fine-tuning on facial expression datasets. Initially, we train our model on a large-scale facial attribute dataset and subsequently fine-tune it on a facial expression dataset. This strategy effectively lowers the high costs and dependence associated with annotating facial expression datasets, while enhancing the accuracy and training speed of our model. This method is particularly suited for UAVs equipped with on-board cameras and image processing capabilities, enabling real-time expression recognition for various applications such as crowd monitoring, search and rescue, and human-UAV interaction. Compared to traditional methods that train exclusively on facial expression datasets, our method significantly reduces the number of training iterations on facial expression datasets and significantly improves the generalization ability of the model, especially for UAV applications. We use a large-scale facial attribute dataset, which is more closely related to the facial expression recognition task, as our source dataset, forming a contrast with methods that typically use a facial recognition dataset as the transfer learning source dataset. The experimental results on the CK + dataset, integrated with UAV-enabled B5G/6G networks, show that our method achieves a facial expression recognition accuracy of 97.6%, a significant improvement over the 97.3% accuracy rate of methods that only train on facial expression datasets, with less training time as well. [ABSTRACT FROM AUTHOR]

The research on technologies and methodologies for (accurate, real-time, spontaneous, three-dimensional...) facial expression recognition is ongoing and has been fostered in the past decades by advances in classification algorithms like deep learning, which makes them part of the Artificial Intelligence literature. Still, despite its upcoming application to contexts such as human–computer interaction, product and service design, and marketing, only a few literature studies have investigated the willingness of end users to share their facial data with the purpose of detecting emotions. This study investigates the level of awareness and interest of 373 potential consumers towards this technology in the car insurance sector, particularly in the contract drafting phase, with a focus on differentiating the respondents between generation Y and Z. Results show that younger people, individuals with higher levels of education, and social network users feel more confident about this innovative technology and are more likely to share their expressive facial data. [ABSTRACT FROM AUTHOR]

Facial expression recognition (FER) holds significant practical implications in real-world scenarios such as human–computer interaction, fatigue driving detection, and learning engagement analysis. Nonetheless, acquiring large-scale and high-quality annotated facial expression datasets is profoundly challenging due to the inherent ambiguity of facial images and concerns over privacy. Consequently, this paper introduces a self-supervised facial expression recognition method based on mask image modeling. This method can learn multi-level facial feature representations without expensive labels and achieves commendable facial expression recognition performance through further fine-grained feature selection. Specifically, we propose the multi-level feature selector (MFS). The MFS comprises two pivotal components: the multi-level feature combiner and the feature selector. During the pre-training stage, the multi-level feature combiner is employed to integrate multi-level features, effectively addressing the vision transformer's deficiencies in capturing high-frequency facial semantics. Subsequently, in the fine-tuning stage, the feature selector can automatically differentiate highly discriminative regions, extracting fine-grained features. Subsequently, we use graph convolutional networks to further mine the latent connections among fine-grained features, ultimately deriving an integrated feature with enhanced discriminative capabilities. Through such fine-grained facial feature selection, we can mitigate performance degradation induced by inter-class similarities and intra-class variations. Experimental results on the RAF-DB, AffectNet, and FER + datasets demonstrate that our approach significantly outperforms other self-supervised methods in recognition performance and closely approaches the state-of-the-art methods in supervised learning. The code is available at https://github.com/Greysahy/MFS. [ABSTRACT FROM AUTHOR]

Dynamic ensemble pruning (DEP) can select the best set of models for an unseen sample to improve a classification system's prediction accuracy and robustness. However, little work has been done to apply DEP to the facial expression recognition (FER) system and reduce DEP's computational burden. This paper proposes a semi-dynamic ensemble pruning algorithm (SDEP) to implement the FER task. First, a subspace-based classifier sequences selection method is proposed to break DEP's neighborhood construction limitation and avoid the exhaustive similarity calculation. Moreover, a pre-prediction mechanism is introduced to calculate the relationship between an unseen sample and data subspaces. Ultimately, the calculated relationship scores and their credibility will be used as the weights for fusion. SDEP reduces the DEP's computational burden while retaining the dynamic characteristics through changing the classifier selection into the subspace-based classifier sequences combination. SDEP demonstrates superior classification accuracy and efficiency to several state-of-the-art dynamic and static EP algorithms in benchmark FER datasets Fer2013, JAFFE, CK+ using a homogeneous ensemble of 799 CNN classifiers. [ABSTRACT FROM AUTHOR]

Automatic pain intensity estimation from facial expression analysis has important applications in medical and healthcare areas. Most of the existing works tend to directly transfer the typical face recognition models to pain estimation task, which may not obtain good performances because the facial expression of pain is spontaneous with subtle facial variations. Pain estimation from facial video is still challenging because it relies on modeling semantic parts and extraction of fine-grained and dynamic features. In this study, we propose a hierarchical global and local transformer (HGLT) model for pain estimation from facial expression videos. HGLT model consists of an image frame embedding subnetwork and a temporal embedding subnetwork for extraction of spatio-temporal features. In the frame embedding subnetwork, we propose a multi-head local attention mechanism to extract the local fine-grained features related to micro variations of pain, followed by a hierarchical self-attention pooling to integrate the global and local features. In the temporal embedding subnetwork, a transformer encoder with temporal attention is proposed to model the temporal relationships of video frames and capture the dynamic facial variations. A correlation loss is proposed to alleviate the problem of long-tailed imbalance in the distribution of pain intensities. Our proposed method is tested with UNBC-McMaster Shoulder Pain, BioVid Heart Pain dataset, and DAiSEE dataset. Experimental results indicate that our method achieves competitive performances compared with the state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Facial expression recognition (FER) in the wild is one of the most challenging visual tasks owing to various uncontrolled factors such as occlusion, pose, and subtle variation in real scenes. These factors can directly affect the robust performance of current networks, especially as most single-feature learning space methods lack the extraction of potential discriminative features and fail to provide a deeper understanding of expressions. To address the above issues, we propose a novel hybrid attention-aware learning network (HALNet), which comprises a feature compactness network (FCN), a hybrid attention enhancement network (HAEN), and a joint loss optimization strategy. First, FCN performs basic expression feature extraction and optimizes intra- and inter-class distributions simultaneously. Afterward, HAEN constructs a multi-level feature enhancement space by fusing hybrid attention based on CNN and transformer in parallel to effectively improve the profound understanding of expressions. Finally, the expression classification is performed by supervised optimization with joint loss. Extensive experiments are assessed on some of the widest employed wild expression datasets, and results indicate our method is superior to several present state-of-the-art methods, obtaining accuracies of 90.29%, 90.04%, and 61.75% on RAF-DB, FERPlus, and AffectNet, respectively. The cross-dataset and occlusion and pose variation datasets assessment further substantiate our approach's sound generalization and robustness. [ABSTRACT FROM AUTHOR]

Facial expression recognition (FER) plays a vital role for applications based on human–computer interaction. In the past few years, many deep learning models have been proposed for FER, but their performance is limited due to challenges such as variation in head pose, occlusion, illumination, etc. Moreover, existing models consider input image through holistic view without giving attention to features relevant to expressions. In this work, we propose a deep fusion framework for FER which employs self-attention mechanism to resolve this issue. Furthermore, to improve feature representation of the images, the proposed model transforms the input image to wavelet domain through discrete wavelet transform. The framework employs two parallel branches for shallow and deep features which are fused for better feature representation. The proposed model is evaluated on posed and in-the-wild datasets: CK+, JAFFE, MUG, YALE, RAF, and experimental results validate the effectiveness of the model. [ABSTRACT FROM AUTHOR]

Achieving safe collaboration between humans and robots in an industrial work-cell requires effective communication. This can be achieved through a robot perception system developed using data-driven machine learning. The challenge for human–robot communication is the availability of extensive, labelled datasets for training. Due to the variations in human behaviour and the impact of environmental conditions on the performance of perception models, models trained on standard, publicly available datasets fail to generalize well to domain and application-specific scenarios. Thus, model personalization involving the adaptation of such models to the individual humans involved in the task in the given environment would lead to better model performance. A novel framework is presented that leverages robust modes of communication and gathers feedback from the human partner to auto-label the mode with the sparse dataset. The strength of the contribution lies in using in-commensurable multimodes of inputs for personalizing models with user-specific data. The personalization through feedback-enabled human–robot communication (PF-HRCom) framework is implemented on the use of facial expression recognition as a safety feature to ensure that the human partner is engaged in the collaborative task with the robot. Additionally, PF-HRCom has been applied to a real-time human–robot handover task with a robotic manipulator. The perception module of the manipulator adapts to the user's facial expressions and personalizes the model using feedback. Having said that, the framework is applicable to other combinations of multimodal inputs in human–robot collaboration applications. [ABSTRACT FROM AUTHOR]

Artificial intelligence (AI) has been recognised as a promising technology for methodological progress and theoretical advancement in learning sciences. However, there remains few empirical investigations into how AI could be applied in learning sciences research. This study aims to utilize AI facial recognition to inform the learning regulation behaviors in synchronous online collaborative learning environments. By studying groups of university students (N = 36) who participated in their online classes under the COVID-19 social distancing mandates, we strive to understand the interrelation between individual affective states and their collaborative group members. Theoretically underpinned by the socially shared regulation of learning framework, our research features a cutting-edge insight into how learners socially shared regulation in group-based tasks. Findings accentuate fundamental added values of AI application in education, whilst indicating further interesting patterns about student self-regulation in the collaborative learning environment. Implications drawn from the study hold strong potential to provide theoretical and practical contributions to the exploration of AI supportive roles in designing and personalizing learning needs, as well as fathom the motion and multiplicity of collaborative learning modes in higher education. [ABSTRACT FROM AUTHOR]

To solve the problem of poor recognition accuracy due to the complex network structure in the facial expression recognition algorithm, an improved lightweight network based on FasterNet is proposed: FGENet. Firstly, the PConv in the FasterNet block module is replaced with the RFE module to enhance the network's ability to perceive key features of facial expressions. Secondly, the GSConv module is introduced into the residual branch, which helps to strengthen the feature modeling of different regions and improves the expression feature extraction ability of the network. Finally, the ECA attention mechanism is introduced into the Ghost module to design the deep attention mechanism DEA, which is applied to the standard convolution after global pooling to better capture the global context information. FGENet is designed to keep the network lightweight while improving performance. Through a series of experiments, it has been proven that FGENet has achieved performance improvements on the three data sets of Fer2013, CK+ and RAF, reaching recognition accuracy rates of 70.49%, 97.89% and 86.72% respectively, further verifying its effectiveness in facial expression recognition tasks. [ABSTRACT FROM AUTHOR]

Facial features extracted from deep convolutional networks are susceptible to background, individual identity and other factors. It interferes with facial expression recognition when mixed with useless features. Considering that different scale features have rich semantic and texture information respectively, this paper takes VGG-16 as the basic network structure and combines multiscale features to obtain richer feature information. In addition, the input feature map elements are enhanced or suppressed by the attention module in order to extract salient features more accurately. The proposed method was validated on two commonly used expression data sets CK+ and RAF-DB, and the recognition rates were 98.77 and 82.83%, respectively. Experimental results show the superiority of this method. [ABSTRACT FROM AUTHOR]

Facial expression recognition has made a significant progress as a result of the advent of more and more convolutional neural networks (CNN). However, with the improvement of CNN, the models continues to get deeper and larger so as to a greater focus on the high-level features of the image and the low-level features tend to be lost. Because of the reason above, the dependence of low-level features between different areas of the face often cannot be summarized. In response to this problem, we propose a novel network based on the CNN model. To extract long-range dependencies of low-level features, multiple attention mechanisms has been introduced into the network. In this paper, the patch attention mechanism is designed to obtain the dependence between low-level features of facial expressions firstly. After fusion, the feature maps are input to the backbone network incorporating convolutional block attention module (CBAM) to enhance the feature extraction ability and improve the accuracy of facial expression recognition, and achieve competitive results on three datasets CK+ (98.10%), JAFFE (95.12%) and FER2013 (73.50%). Further, according to the PA Net designed in this paper, a hardware friendly implementation scheme is designed based on memristor crossbars, which is expected to provide a software and hardware co-design scheme for edge computing of personal and wearable electronic products. [ABSTRACT FROM AUTHOR]

The limitation of the small-scale expression samples generally causes the performance degradation for facial expression recognition-based methods. Also, the correlation between different expression is always ignored when performing feature extraction process. Given above, we propose a novel approach that develops multi-class differentiation feature representation guided joint dictionary learning for FER. The proposed approach mainly includes two steps: firstly, we construct multi-class differentiation feature dictionaries corresponding to different expressions of training samples, aiming to enlarge inter-expression distance to mitigate the problem of nonlinear distribution in training samples. Secondly, we joint learn the multiple feature dictionaries by optimizing the resolutions of each feature dictionary, aiming to establish the strong relationship and enhance the representation ability among multiple feature dictionaries. To sum up, the proposed approach has more discriminative ability from the representation perspective. Comprehensive experiments carried out using three public datasets, including JAFFE, CK+ , and KDEF datasets, demonstrate that the proposed approach has strong performance for small-scale samples compared to several state-of-the-art methods. Multi-class differentiation feature representation guided joint dictionary learning for facial expression recognition. [ABSTRACT FROM AUTHOR]

This paper presents an image descriptor based on local texture and orientation patterns. The texture information is embedded in the orientation component. Each pixel gradient orientation is considered for extracting the D-EOLP features. The image is divided into several blocks, from each of which the D-EOLP features are extracted and concatenated. For redundancy reduction, local mean-based nearest neighbor discriminant analysis (LM-NNDA) is employed and a low-dimensional potential, enhanced image descriptor is obtained. The performance of the proposed approach is evaluated on two prominent application areas namely, handwritten recognition and facial expression recognition(FER). The robustness of this approach is analyzed in terms of accuracy and computational time by comparing it with state-of-the-art algorithms. Results of handwritten numeral recognition on standard numeral datasets like ISI Kolkata, CMaterdb3, and MNIST and of FER on CK+ and Jaffe show better performance with the proposed approach. [ABSTRACT FROM AUTHOR]

The human brain can effectively perform Facial Expression Recognition (FER) with a few samples by utilizing its cognitive ability. However, unlike the human brain, even the well-trained deep neural network is data-dependent and lacks cognitive ability. To tackle this challenge, this paper proposes a novel framework, Brain Machine Generative Adversarial Networks (BM-GAN), which utilizes the concept of brain's cognitive ability to guide a Convolutional Neural Network to generate LIKE-electroencephalograph (EEG) features. More specifically, we firstly obtain EEG signals triggered from facial emotion images, then we adopt BM-GAN to carry out the mutual generation of image visual features and EEG cognitive features. BM-GAN intends to use the cognitive knowledge learnt from EEG signals to instruct the model to perceive LIKE-EEG features. Thereby, BM-GAN has a superior performance for FER like the human brain. The proposed model consists of VisualNet, EEGNet, and BM-GAN. More specifically, VisualNet can obtain image visual features from facial emotion images and EEGNet can obtain EEG cognitive features from EEG signals. Subsequently, the BM-GAN completes the mutual generation of image visual features and EEG cognitive features. Finally, the predicted LIKE-EEG features of test images are used for FER. After learning, without the participation of the EEG signals, an average classification accuracy of 96.6 % is obtained on Chinese Facial Affective Picture System dataset using LIKE-EEG features for FER. Experiments demonstrate that the proposed method can produce an excellent performance for FER. [ABSTRACT FROM AUTHOR]

Real-time monitoring of students' classroom engagement level is of paramount importance in modern education. Facial expression recognition has been extensively explored in various studies to achieve this goal. However, conventional models often grapple with a high number of parameters and substantial computational costs, limiting their practicality in real-time applications and real-world scenarios. To address this limitation, this paper proposes "Light_Fer," a lightweight model designed to attain high accuracy while reducing parameters. Light_Fer's novelty lies in the integration of depthwise separable convolution, group convolution, and inverted bottleneck structure. These techniques optimize the models' architecture, resulting in superior accuracy with fewer parameters. Experimental results demonstrate that Light_Fer, with just 0.23M parameters, achieves remarkable accuracies of 87.81% and 88.20% on FERPLUS and RAF-DB datasets, respectively. Furthermore, by establishing a correlation between facial expressions and students' engagement levels, we extend the application of Light_Fer to real-time detection and monitoring of students' engagement during classroom activities. In conclusion, the proposed Light_Fer model, with its lightweight design and enhanced accuracy, offers a promising solution for real-time student engagement monitoring through facial expression recognition. [ABSTRACT FROM AUTHOR]

Facial expression recognition (FER) suffers from insufficient label information, as human expressions are complex and diverse, with many expressions ambiguous. Using low-quality labels or low-quantity labels will aggravate ambiguity of model predictions and reduce the accuracy of FER. How to improve the robustness of FER to ambiguous data with insufficient information remains challenging. To this end, we propose the Suppressing Ambiguity Self-Training (SAST) framework which is the first attempt to address the problem of insufficient information both label quality and label quantity containing, simultaneously. Specifically, we design an Ambiguous Relative Label Usage (ARLU) strategy that mixes hard labels and soft labels to alleviate the information loss problem caused by hard labels. We also enhance the robustness of the model to ambiguous data by means of Self-Training Resampling (STR). We further use the landmarks and Patch Branch (PB) to enhance the ability of suppressing ambiguity. Experiments on RAF-DB, FERPlus, SFEW, and AffectNet datasets show that our SAST outperforms 6 semi-supervised methods with fewer annotations, and achieves competitive accuracy to State-Of-The-Art (SOTA) FER methods. Our code is available at https://github.com/Liuxww/SAST. [ABSTRACT FROM AUTHOR]

Student disengagement in higher education offline classroom learning has become a critical concern because of digital distractions and insufficient student engagement monitoring. As the student count increases, it becomes harder for teachers to monitor individual students' engagement and maintain class engagement throughout the lecture. Computer vision, affective computing, and deep learning technologies can potentially build vision-based intelligent applications that can automatically estimate student engagement in real-time. Existing methods have limitations like scalability, high computation, non-real-time performance, and inability to track individual students' engagement. This paper attempts to address this research gap by proposing a real-time vision-based multi-cues approach. The approach obtained three kinds of scores: First, an affect score is based on the positive and negative academic affective states. Second, an attention score is based on frontal, and non-frontal head pose estimation using Perspective-n-Point technique. Third, a head movement score is based on head displacement distance. A Facial Expression Recognition model has been developed for academic affective state classification. This was achieved by fine-tuning MobileNetV2 through an incremental unfreezing approach on our newly developed dataset named Classroom Spontaneous Facial Expression Dataset. The model achieved training, validation, and testing accuracies of 95.7%, 94.9%, and 76%, respectively. The three obtained scores are combined using weighted fusion method to estimate individual students' engagement labels: engaged or non-engaged. Subsequently, every student's labels are aggregated to estimate the overall class engagement. Additionally, when validated with students' engagement self-reports the proposed approach yielded a 75% and 80% correlation for individual students and overall class engagement, respectively. [ABSTRACT FROM AUTHOR]

The real-world facial expression recognition (FER) datasets suffer from noisy annotations due to crowd-sourcing, ambiguity in expressions, the subjectivity of annotators, and inter-class similarity. However, the recent deep networks have a strong capacity to memorize the noisy annotations leading to corrupted feature embedding and poor generalization. Recent works handle the problem by selecting samples with clean labels based on loss values using a fixed threshold for all the classes which may not always be reliable. They also depend upon the noise rate in the data which may not always be available. In this work, we propose a novel FER framework (DNFER) in which samples with clean labels are selected based on a class-specific threshold, computed dynamically in each mini-batch. Specifically, DNFER uses supervised training on selected clean samples and unsupervised consistent training on all the samples. This threshold is independent of noise rate and does not need any clean data, unlike other methods. In addition, to effectively learn from noisy annotated samples, the posterior distributions between weakly-augmented image and strongly-augmented image are aligned using an unsupervised consistency loss. We demonstrate the robustness of DNFER on both synthetic as well as on real noisy annotated FER datasets. In addition, DNFER obtains state-of-the-art performance on popular benchmark datasets, with 90.41% on RAFDB, 57.77 % on SFEW, 89.32% on FERPlus, and 65.22% on AffectNet-7. Our source codes are made publicly available at https://github.com/1980x/DNFER. [ABSTRACT FROM AUTHOR]

In recent times, group emotion recognition has attracted considerable research attention, and diverse applications such as situation-aware models and security surveillance models that require this capability have emerged. Group emotion recognition is a complex task that depends on human visual stimuli and correlation inference. We propose a novel group emotion recognition model using a fuzzy system based on psychological principles. Based on psychological factors defined by a fuzzy system, we classify group emotions into universal classes used to classify seven basic single face emotions: happiness, sadness, surprise, disgust, anger, fear, and neutral, and achieve approximately 14% higher accuracy than an average method in qualitative and quantitative experiments. Furthermore, the contribution of each factor to group emotion recognition was determined using the psychological principles commonly used in cinematography, at a low cost. [ABSTRACT FROM AUTHOR]

Facial expression recognition (FER) is one of the popular research topics in the field of computer vision. When most of the deep learning expression recognition methods that achieve satisfactory results with a single dataset are applied to a new dataset, additional costs result from labeling the new data. FER under cross-dataset also suffers from difficulties such as data discrepancy and expression ambiguity. To address these issues, we propose an Unsupervised Self-Training Similarity Transfer (USTST) method for cross-domain FER. The Cross-Swin-Transformer (CST) module is designed to extract features and assign greater attention weight to the similar regions of the source and target domain images. The Self-Training Resampling (STR) and the Knowledge Transfer (KT) modules are then constructed to improve the confidence of the model prediction for the target domain. We also design ambiguity suppression loss and cross-domain loss to improve the ability of the model to discriminate expressions while transferring knowledge across domains. The experimental results with the RAF-DB dataset as the source domain and the CK+, JAFFE, SFEW, FER2013 and ExpW datasets as the target domains, show that our approach achieves much higher performance than the state-of-the-art cross-domain FER methods, while requiring no labels of new datasets. [ABSTRACT FROM AUTHOR]

Facial expression recognition is crucial in analyzing an individual's emotional state. Ensemble pruning becomes essential to enhance the effectiveness of this recognition by selecting appropriate classifiers from a pool of base classifiers. However, outliers and noise within the classifiers can adversely affect the final recognition results and the generalization performance of the selected subset of classifiers. Additionally, effectively combining accuracy and low redundancy of base classifiers remains a challenging problem that requires further investigation. In this paper, we propose a novel algorithm called Pairwise Dependency-based Robust Ensemble Pruning (PDREP) to address these issues. The PDREP algorithm treats the predicted results of classifiers for sample instances as features of the classifier and evaluates their dependencies between pairs of classifiers using mutual information. By incorporating this dependency measure into the regression-based objective equation, we can assess the redundancy of a subset of base classifiers and prune redundant classifiers. We use the l 2 , 1 -norm in PDREP's objective equation to perform robust classifier pruning while considering the base classifiers' dependencies and accuracy. Furthermore, we introduce weight control parameters to balance accuracy and dependencies, facilitating the elimination of ineffective and redundant classifiers. The results show significant improvements when evaluating the proposed method's recognition accuracy on five public face sentiment datasets (FER2013, JAFFE, CK+, RaFD, and KDEF). Specifically, the proposed method achieves 3.15%, 9.39%, 1.72%, 3.70%, and 4.70% higher accuracy than integrating all base classifiers, respectively. Extensive experiments conducted on five facial expression datasets demonstrate that the proposed method consistently outperforms existing state-of-the-art ensemble pruning algorithms in most cases. [ABSTRACT FROM AUTHOR]

Machine learning applications are increasingly challenged by the growing volume of data. In this context, selecting relevant features from the vast pool of extracted data remains a critical issue. This challenge is particularly prominent in facial expression recognition tasks, where the number of features extracted from facial images can be overwhelming. The quality of the selected features plays a pivotal role in determining the accuracy of facial expression recognition. To tackle this challenge effectively, we propose a novel three-phase hybrid feature selection method that leverages the strengths of filter, wrapper, and embedded algorithms. In the first phase, we employ correlation-based feature selection to reduce the search space, thereby enhancing computational efficiency and expediting the feature selection process. Subsequently, in the second phase, we apply recursive feature elimination to further refine feature selection. Finally, in the third phase, we utilize the XGBoost feature selection algorithm to identify the optimal subset of features. To evaluate the effectiveness of the proposed method, we have conducted experiments on five demanding facial expression recognition datasets: DISFA, CK+, DISFA+, JAFFE, and FER2013, while using key metrics, including accuracy, precision, recall, and F1-Score. The obtained results demonstrate the efficiency of our method, achieving an accuracy of 94.68%, 98.82%, 95.8%, 97.7%, and 66.1% on DISFA, CK+, DISFA+, JAFFE, and FER2013 datasets, respectively. [ABSTRACT FROM AUTHOR]

Facial expressions can convey the internal emotions of a person within a certain scenario and play a major role in the social interaction of human beings. In automatic Facial Expression Recognition (FER) systems, the method applied for feature extraction plays a major role in determining the performance of a system. In this regard, by drawing inspiration from the Swastik symbol, three texture based feature descriptors named Symbol Patterns (SP1, SP2 and SP3) have been proposed for facial feature extraction. SP1 generates one pattern value by comparing eight pixels within a 3 × 3 neighborhood, whereas, SP2 and SP3 generates two pattern values each by comparing twelve and sixteen pixels within a 5 × 5 neighborhood respectively. In this work, the proposed Symbol Patterns (SP) have been evaluated with natural, fibonacci, odd, prime, squares and binary weights for determining the optimal recognition accuracy. The proposed SP methods have been tested on MUG, TFEID, CK+, KDEF, FER2013 and FERG datasets and the results from the experimental analysis demonstrated an improvement in the recognition accuracy when compared to the existing FER methods. [ABSTRACT FROM AUTHOR]

Edge computing has shown significant successes in addressing the security and privacy issues related to facial expression recognition (FER) tasks. Although several lightweight networks have been proposed for edge computing, the computing demands and memory access cost (MAC) imposed by these networks hinder their deployment on edge devices. Thus, we propose an edge computing-oriented real-time facial expression recognition network, called EC-RFERNet. Specifically, to improve the inference speed, we devise a mini-and-fast (MF) block based on the partial convolution operation. The MF block effectively reduces the MAC and parameters by processing only a part of the input feature maps and eliminating unnecessary channel expansion operations. To improve the accuracy, the squeeze-and-excitation (SE) operation is introduced into certain MF blocks, and the MF blocks at different levels are selectively connected by the harmonic dense connection. SE operation is used to complete the adaptive channel weighting, and the harmonic dense connection is used to exchange information between different MF blocks to enhance the feature learning ability. The MF block and the harmonic dense connection together constitute the harmonic-MF module, which is the core component of EC-RFERNet. This module achieves a balance between accuracy and inference speed. Five public datasets are used to test the validity of EC-RFERNet and to demonstrate its competitive performance, with only 2.25 MB and 0.55 million parameters. Furthermore, one human–robot interaction system is constructed with a humanoid robot equipped with the Raspberry Pi. The experimental results demonstrate that EC-RFERNet can provide an effective solution for practical FER applications. [ABSTRACT FROM AUTHOR]

Facial expression recognition plays a crucial role in computer vision and human–computer interaction, with applications ranging from emotion analysis to social robotics. However, accurate recognition becomes challenging in the presence of occlusions, such as facial hair, glasses, and self-occlusion. This study addresses the problem of facial expression recognition despite occlusions and proposes a novel approach to overcome this challenge. The aim of this research is to develop a robust facial expression recognition framework that effectively handles occlusions and improves recognition accuracy. To achieve this, Hopfield networks, Deep Belief Networks (DBN), and Lanczos interpolation are integrated into the proposed method. Lanczos interpolation helps preserve image quality and reduces during resizing. The Hopfield network is utilized for feature extraction, capturing facial expression features even in the presence of occlusions. The DBN is employed for representation learning, fine-tuning the network using DenseNet to adapt to occluded facial expressions. To evaluate the proposed approach, extensive experiments were conducted on various datasets, including the Static Facial Expressions in the Wild (SFEW) dataset, AffectNet dataset, Real-world Affective Faces Database (RAF-DB), MMI Facial Expression Database, Oulu-CASIA NIR&VIS Facial Expression Database, and Extended Cohn-Kanade (CK +) dataset. The results demonstrate the superiority of our method in handling occlusions and achieving improved facial expression recognition accuracy compared to existing approaches. The contributions of this work are twofold: First, a novel framework is proposed that integrates Lanczos interpolation, Hopfield networks, DBN, and to effectively address the challenge of occlusions in facial expression recognition. Second, extensive experimental validation is provided on diverse datasets, highlighting the superior performance of our approach in handling occlusions and achieving accurate recognition. In conclusion, the proposed approach demonstrates its efficacy in improving facial expression recognition under occlusions. By effectively handling occlusions and capturing relevant features, our method opens up possibilities for enhanced emotion analysis, human–computer interaction, and social robotics applications. [ABSTRACT FROM AUTHOR]

Facial expression recognition plays an essential role in surveillance videos, anxiety treatment, expression analysis, gesture recognition, computer games, patient monitoring, operator fatigue detection, and robotics. Therefore, facial expression recognition has attracted more and more attention over the years and became a difficult task because emotion can be influenced by several factors. Some approaches based on Deep Convolutional Neural Networks (DCNN) and Transfer Learning have been successful to recognize emotion in video sequences. However, these approaches remain limited because it is difficult to model spatio-temporal interactions between video frames or identify salient features to improve accuracy. In this article, we propose a facial expression recognition system combining the representations of Deep Learning features and dynamic texture features. For the Deep Learning part, we used the VGG19 model to extract facial features, which will feed the LSTM (Long Short Term Memory) cells in order to extract spatio-temporal information between frames. While the HOG-TOP (Histogram of Oriented Gradients from Three Orthogonal Planes) descriptor aims to extract dynamic textures from video sequences to characterize facial appearance changes. Finally, we combine both models with a Multimodal Compact Bilinear (MCB) algorithm to produce a robust descriptor vector. Classification was performed using the SVM (Support Vector Machine) classifier to predict the emotion class. The experimental part was carried out based on the INTERFACE05 dataset that the accuracy of facial expression recognition was increased almost 1% by the fusion method (98.44%) than the baseline approach (97.75%). To summarize, the proposed method obtain a higher accuracy and robust detection meaning. [ABSTRACT FROM AUTHOR]

Facial expressions are an important form of non-verbal communication as they directly reflect the internal emotions of a person. The primary task of automated Facial Expression Recognition (FER) systems lies in extracting salient features related to facial expressions. In this paper, a Cross Connected Convolutional Neural Network (CC-CNN) has been proposed for extracting the facial features. The proposed CC-CNN model contains two levels of input for extracting the features related to facial expressions. Cyclopentane Feature Descriptor (CyFD), inspired by cyclopentane's structure, has been proposed to extract significant features. The feature response map generated by the CyFD method has been given as input to the first level, and in the second level, the features have been extracted directly from the facial image. The input images from both levels are passed through a series of convolutional layers with cross connections for extracting the fused (local and global) features related to facial expressions. Finally, towards the end, the CC-CNN method works by fusing all the features extracted from both levels. To validate the efficiency of the proposed CC-CNN method, the experiments have been performed on benchmark FER datasets such as CK+, MUG, RAF, FER2013 and FERG. The comparison results from the experimental analysis revealed that the proposed model outperformed the recent FER methods. [ABSTRACT FROM AUTHOR]

The generation of a large human-labelled facial expression dataset is challenging due to ambiguity in labelling the facial expression class, and annotation cost. However, facial expression recognition (FER) systems demand discriminative feature representation, and require many training samples to establish stronger decision boundaries. Recently, FER approaches have used data augmentation techniques to increase the number of training samples for model generation. However, these augmented samples are derived from existing training data, and therefore have limitations for developing an accurate FER system. To achieve meaningful facial expression representations, we introduce an augmentation technique based on deep learning and genetic algorithms for FER. The proposed approach exploits the hypothesis that augmenting the feature-set is better than augmenting the visual data for FER. By evaluating this relationship, we discovered that the genetic evolution of discriminative features for facial expression is significant in developing a robust FER approach. In this approach, facial expression samples are generated from RGB visual data from videos considering human face frames as regions of interest. The face detected frames are further processed to extract key-frames within particular intervals. Later, these key-frames are convolved through a deep convolutional network for feature generation. A genetic algorithm's fitness function is gauged to select optimal genetically evolved deep facial expression receptive fields to represent virtual facial expressions. The extended facial expression information is evaluated through an extreme learning machine classifier. The proposed technique has been evaluated on five diverse datasets i.e. JAFFE, CK+, FER2013, AffectNet and our application-specific Instructor Facial Expression (IFEV) dataset. Experimentation results and analysis show the promising accuracy and significance of the proposed technique on all these datasets. [ABSTRACT FROM AUTHOR]

Parkinson's disease (PD) is the most widespread neurodegenerative disorder after Alzheimer's disease. Masked facial expressions in people with PD may have a substantial impact on their social well-being and quality of life. This study investigated the effectiveness of a newly developed Emotion Detection and Grading System (EDGS) software for identifying and grading expressions of normal as well as patients with PD. EDGS is a deep learning-based emotion detection and grading software for the identification of seven basic emotional expressions and grading the intensities of happy and sad emotional expressivity. It was found that the EDGS software successfully identified happy and surprise emotions in normal subjects and patients with PD. It could categorize two intensities of happy and sad emotions. Additionally, the EDGS software was also found to be able to categorise the patients with PD according to their decrease in emotional expressivity, thereby opening possibilities of detecting the progress of PD. [ABSTRACT FROM AUTHOR]

Facial expression recognition has become a hot issue in the field of artificial intelligence. So, we collect literature on facial expression recognition. First, methods based on machine learning are introduced in detail, which include image preprocessing, feature extraction, and image classification. Then, we review deep learning methods in detail: convolutional neural networks, deep belief networks, generative adversarial networks, and recurrent neural networks. Moreover, the advantages and limitations of different facial expression recognition methods are compared. In addition, 20 commonly used facial expression datasets are collected in this paper, and the types of expressions and the number of images contained in each dataset are summarized. Finally, the current problems and future development of facial expression recognition are concluded. [ABSTRACT FROM AUTHOR]

Although great progress has been made in facial expression recognition, it still faces challenges such as occlusion and pose changes in real-world scenario. To address this issue, we propose a simple yet effective multi-granularity and multi-scale feature fusion network (MM-Net) to achieve robust expression recognition without either manually extracting local patches or designing complex sub-networks. Specifically, we use a puzzle generator to divide the image into local regions of different granularity, which are then randomly shuffled and reorganized to form a new input image. By feeding the facial puzzles in order from fine-grained to coarse-grained, the network progressively mines the local fine-grained information, the coarse-grained information, and the global information. Besides, considering the subtle inter-class variation characteristic of different expressions, we use the multi-scale feature fusion strategy in the shallow feature extraction module to obtain global features with detailed information for capturing the subtle differences in facial expression images. Extensive experimental results on three in-the-wild FER benchmarks demonstrate the superiority of the proposed MM-Net compared to state-of-the-art methods. [ABSTRACT FROM AUTHOR]