Search

Your search keyword '"Jiang, Dongmei"' showing total 28 results
Start Over Author "Jiang, Dongmei" Publication Year Range This year
28 results on '"Jiang, Dongmei"'

1. Improving Multimodal Emotion Recognition by Leveraging Acoustic Adaptation and Visual Alignment

Author

Zhao, Zhixian, Chen, Haifeng, Li, Xi, Jiang, Dongmei, and Xie, Lei

Subjects
Computer Science - Human-Computer Interaction, Computer Science - Sound, Electrical Engineering and Systems Science - Audio and Speech Processing
Abstract

Multimodal Emotion Recognition (MER) aims to automatically identify and understand human emotional states by integrating information from various modalities. However, the scarcity of annotated multimodal data significantly hinders the advancement of this research field. This paper presents our solution for the MER-SEMI sub-challenge of MER 2024. First, to better adapt acoustic modality features for the MER task, we experimentally evaluate the contributions of different layers of the pre-trained speech model HuBERT in emotion recognition. Based on these observations, we perform Parameter-Efficient Fine-Tuning (PEFT) on the layers identified as most effective for emotion recognition tasks, thereby achieving optimal adaptation for emotion recognition with a minimal number of learnable parameters. Second, leveraging the strengths of the acoustic modality, we propose a feature alignment pre-training method. This approach uses large-scale unlabeled data to train a visual encoder, thereby promoting the semantic alignment of visual features within the acoustic feature space. Finally, using the adapted acoustic features, aligned visual features, and lexical features, we employ an attention mechanism for feature fusion. On the MER2024-SEMI test set, the proposed method achieves a weighted F1 score of 88.90%, ranking fourth among all participating teams, validating the effectiveness of our approach.

Published
2024

2. ExpLLM: Towards Chain of Thought for Facial Expression Recognition

Author

Lan, Xing, Xue, Jian, Qi, Ji, Jiang, Dongmei, Lu, Ke, and Chua, Tat-Seng

Subjects
Computer Science - Computer Vision and Pattern Recognition, Computer Science - Multimedia
Abstract

Facial expression recognition (FER) is a critical task in multimedia with significant implications across various domains. However, analyzing the causes of facial expressions is essential for accurately recognizing them. Current approaches, such as those based on facial action units (AUs), typically provide AU names and intensities but lack insight into the interactions and relationships between AUs and the overall expression. In this paper, we propose a novel method called ExpLLM, which leverages large language models to generate an accurate chain of thought (CoT) for facial expression recognition. Specifically, we have designed the CoT mechanism from three key perspectives: key observations, overall emotional interpretation, and conclusion. The key observations describe the AU's name, intensity, and associated emotions. The overall emotional interpretation provides an analysis based on multiple AUs and their interactions, identifying the dominant emotions and their relationships. Finally, the conclusion presents the final expression label derived from the preceding analysis. Furthermore, we also introduce the Exp-CoT Engine, designed to construct this expression CoT and generate instruction-description data for training our ExpLLM. Extensive experiments on the RAF-DB and AffectNet datasets demonstrate that ExpLLM outperforms current state-of-the-art FER methods. ExpLLM also surpasses the latest GPT-4o in expression CoT generation, particularly in recognizing micro-expressions where GPT-4o frequently fails., Comment: project page: https://starhiking.github.io/ExpLLM_Page/

Published
2024

3. Optimus-1: Hybrid Multimodal Memory Empowered Agents Excel in Long-Horizon Tasks

Author

Li, Zaijing, Xie, Yuquan, Shao, Rui, Chen, Gongwei, Jiang, Dongmei, and Nie, Liqiang

Subjects
Computer Science - Artificial Intelligence, Computer Science - Computation and Language
Abstract

Building a general-purpose agent is a long-standing vision in the field of artificial intelligence. Existing agents have made remarkable progress in many domains, yet they still struggle to complete long-horizon tasks in an open world. We attribute this to the lack of necessary world knowledge and multimodal experience that can guide agents through a variety of long-horizon tasks. In this paper, we propose a Hybrid Multimodal Memory module to address the above challenges. It 1) transforms knowledge into Hierarchical Directed Knowledge Graph that allows agents to explicitly represent and learn world knowledge, and 2) summarises historical information into Abstracted Multimodal Experience Pool that provide agents with rich references for in-context learning. On top of the Hybrid Multimodal Memory module, a multimodal agent, Optimus-1, is constructed with dedicated Knowledge-guided Planner and Experience-Driven Reflector, contributing to a better planning and reflection in the face of long-horizon tasks in Minecraft. Extensive experimental results show that Optimus-1 significantly outperforms all existing agents on challenging long-horizon task benchmarks, and exhibits near human-level performance on many tasks. In addition, we introduce various Multimodal Large Language Models (MLLMs) as the backbone of Optimus-1. Experimental results show that Optimus-1 exhibits strong generalization with the help of the Hybrid Multimodal Memory module, outperforming the GPT-4V baseline on many tasks., Comment: 30 pages, 13 figures

Published
2024

4. OV-DINO: Unified Open-Vocabulary Detection with Language-Aware Selective Fusion

Author

Wang, Hao, Ren, Pengzhen, Jie, Zequn, Dong, Xiao, Feng, Chengjian, Qian, Yinlong, Ma, Lin, Jiang, Dongmei, Wang, Yaowei, Lan, Xiangyuan, and Liang, Xiaodan

Subjects
Computer Science - Computer Vision and Pattern Recognition
Abstract

Open-vocabulary detection is a challenging task due to the requirement of detecting objects based on class names, including those not encountered during training. Existing methods have shown strong zero-shot detection capabilities through pre-training and pseudo-labeling on diverse large-scale datasets. However, these approaches encounter two main challenges: (i) how to effectively eliminate data noise from pseudo-labeling, and (ii) how to efficiently leverage the language-aware capability for region-level cross-modality fusion and alignment. To address these challenges, we propose a novel unified open-vocabulary detection method called OV-DINO, which is pre-trained on diverse large-scale datasets with language-aware selective fusion in a unified framework. Specifically, we introduce a Unified Data Integration (UniDI) pipeline to enable end-to-end training and eliminate noise from pseudo-label generation by unifying different data sources into detection-centric data format. In addition, we propose a Language-Aware Selective Fusion (LASF) module to enhance the cross-modality alignment through a language-aware query selection and fusion process. We evaluate the performance of the proposed OV-DINO on popular open-vocabulary detection benchmarks, achieving state-of-the-art results with an AP of 50.6% on the COCO benchmark and 40.1% on the LVIS benchmark in a zero-shot manner, demonstrating its strong generalization ability. Furthermore, the fine-tuned OV-DINO on COCO achieves 58.4% AP, outperforming many existing methods with the same backbone. The code for OV-DINO is available at https://github.com/wanghao9610/OV-DINO., Comment: Technical Report

Published
2024

5. Prompt Customization for Continual Learning

Author

Dai, Yong, Hong, Xiaopeng, Wang, Yabin, Ma, Zhiheng, Jiang, Dongmei, and Wang, Yaowei

Subjects
Computer Science - Computer Vision and Pattern Recognition, Computer Science - Machine Learning
Abstract

Contemporary continual learning approaches typically select prompts from a pool, which function as supplementary inputs to a pre-trained model. However, this strategy is hindered by the inherent noise of its selection approach when handling increasing tasks. In response to these challenges, we reformulate the prompting approach for continual learning and propose the prompt customization (PC) method. PC mainly comprises a prompt generation module (PGM) and a prompt modulation module (PMM). In contrast to conventional methods that employ hard prompt selection, PGM assigns different coefficients to prompts from a fixed-sized pool of prompts and generates tailored prompts. Moreover, PMM further modulates the prompts by adaptively assigning weights according to the correlations between input data and corresponding prompts. We evaluate our method on four benchmark datasets for three diverse settings, including the class, domain, and task-agnostic incremental learning tasks. Experimental results demonstrate consistent improvement (by up to 16.2\%), yielded by the proposed method, over the state-of-the-art (SOTA) techniques., Comment: ACM MM

Published
2024

6. CricaVPR: Cross-image Correlation-aware Representation Learning for Visual Place Recognition

Author

Lu, Feng, Lan, Xiangyuan, Zhang, Lijun, Jiang, Dongmei, Wang, Yaowei, and Yuan, Chun

Subjects
Computer Science - Computer Vision and Pattern Recognition, Computer Science - Robotics
Abstract

Over the past decade, most methods in visual place recognition (VPR) have used neural networks to produce feature representations. These networks typically produce a global representation of a place image using only this image itself and neglect the cross-image variations (e.g. viewpoint and illumination), which limits their robustness in challenging scenes. In this paper, we propose a robust global representation method with cross-image correlation awareness for VPR, named CricaVPR. Our method uses the attention mechanism to correlate multiple images within a batch. These images can be taken in the same place with different conditions or viewpoints, or even captured from different places. Therefore, our method can utilize the cross-image variations as a cue to guide the representation learning, which ensures more robust features are produced. To further facilitate the robustness, we propose a multi-scale convolution-enhanced adaptation method to adapt pre-trained visual foundation models to the VPR task, which introduces the multi-scale local information to further enhance the cross-image correlation-aware representation. Experimental results show that our method outperforms state-of-the-art methods by a large margin with significantly less training time. The code is released at https://github.com/Lu-Feng/CricaVPR., Comment: Accepted by CVPR2024

Published
2024

7. Deep Homography Estimation for Visual Place Recognition

Author

Lu, Feng, Dong, Shuting, Zhang, Lijun, Liu, Bingxi, Lan, Xiangyuan, Jiang, Dongmei, and Yuan, Chun

Subjects
Computer Science - Computer Vision and Pattern Recognition, Computer Science - Artificial Intelligence
Abstract

Visual place recognition (VPR) is a fundamental task for many applications such as robot localization and augmented reality. Recently, the hierarchical VPR methods have received considerable attention due to the trade-off between accuracy and efficiency. They usually first use global features to retrieve the candidate images, then verify the spatial consistency of matched local features for re-ranking. However, the latter typically relies on the RANSAC algorithm for fitting homography, which is time-consuming and non-differentiable. This makes existing methods compromise to train the network only in global feature extraction. Here, we propose a transformer-based deep homography estimation (DHE) network that takes the dense feature map extracted by a backbone network as input and fits homography for fast and learnable geometric verification. Moreover, we design a re-projection error of inliers loss to train the DHE network without additional homography labels, which can also be jointly trained with the backbone network to help it extract the features that are more suitable for local matching. Extensive experiments on benchmark datasets show that our method can outperform several state-of-the-art methods. And it is more than one order of magnitude faster than the mainstream hierarchical VPR methods using RANSAC. The code is released at https://github.com/Lu-Feng/DHE-VPR., Comment: Accepted by AAAI2024

Published
2024
Full Text
View/download PDF

8. Enhancing Emotional Generation Capability of Large Language Models via Emotional Chain-of-Thought

Author

Li, Zaijing, Chen, Gongwei, Shao, Rui, Xie, Yuquan, Jiang, Dongmei, and Nie, Liqiang

Subjects
Computer Science - Computation and Language, Computer Science - Artificial Intelligence
Abstract

Large Language Models (LLMs) have shown remarkable performance in various emotion recognition tasks, thereby piquing the research community's curiosity for exploring their potential in emotional intelligence. However, several issues in the field of emotional generation tasks remain unresolved, including human preference alignment and emotional generation assessment. In this paper, we propose the Emotional Chain-of-Thought (ECoT), a plug-and-play prompting method that enhances the performance of LLMs on various emotional generation tasks by aligning with human emotional intelligence guidelines. To assess the reliability of ECoT, we propose an automated model-based evaluation method called Emotional Generation Score (EGS). EGS incorporates Goleman's Emotional Intelligence Theory as a consensus of human experts, providing a new perspective on the evaluation of emotional generation tasks. Extensive experimental results demonstrate the effectiveness of ECoT and EGS. Further, we discuss the promise of LLMs in the field of emotional intelligence and present key insights into the LLMs with the ECoT in emotional generation tasks.

Published
2024

21. Mechanism of improving the quality of table grapes (Vitis. spp.) by exogenous eugenol based on transcriptome.

Author

Jiang, Nan, Wang, Liuqing, Gao, Yuan, Li, Qianqian, Jiang, Dongmei, Yao, Weirong, and Wang, Meng

Subjects
TABLE grapes, BERRIES, EUGENOL, GRAPE quality, TRANSCRIPTOMES, GENE expression, GRAPES
Abstract

Summary: Grape is widely consumed all over the world for its rich nutritious and health‐promoting effects. While grape berries are prone to quality deterioration after harvest. This study aimed to investigate the effect of eugenol treatment on the quality of grape berries based on the transcriptome. Harvested grape berries were treated with eugenol, and then collected for nutrient component determination and transcriptomics analyses. As a results, treatment with eugenol enhanced the berries firmness, increased the contents of soluble sugars, organic acids and phenolic compounds, improved the antioxidant capacity, and altered gene expression levels of related pathways. Transcriptome analysis revealed 699 differentially expressed genes by 100 mmol L−1 eugenol‐treated berries. Among these DEGs, the transcriptional levels involved in the cell wall, sugar and acid metabolism, phenylpropane metabolism, antioxidant enzymes, and WRKY, MYB family transcription factors were significantly regulated. This study will help to promote the application of eugenol for quality improvements in grape berries. [ABSTRACT FROM AUTHOR]

Published
2024
Full Text
View/download PDF

23. Chitosan/Sodium Alginate Hydrogel for the Release of Berberine as an Algae Suppressant: RSM Optimization and Analysis of Sustained Release Characteristics.

Author

Wang Y, Wu M, Tang P, and Jiang D

Abstract

In this study, we used chitosan/sodium alginate hydrogel as a carrier to prepare berberine sustained-release capsule materials that can inhibit algae for a long time and safely. The preparation conditions of the material were optimized by the response surface method, and the optimized capsule material was characterized and the sustained release characteristics were analyzed to study the change of the algae inhibition effect of the material within 30 days. The results showed that the optimum preparation parameters of the material were 0.54% chitosan content, 2.46% sodium alginate content and 1.09% anhydrous calcium chloride content by response surface optimization design, which was consistent with the parameters set by each factor at the central point. The algae inhibition rate of the material under this preparation condition was 93.75 ± 1.01%, which was similar to the predicted value. The release characteristics analysis showed that the material continuously released up to 90% of berberine within 24 days, and its release characteristics were sustained release after burst release, with good sustained release effect. The results of material characterization showed that chitosan/sodium alginate hydrogel could effectively load berberine and was beneficial to the loading and release of berberine. The results of algae inhibition experiments showed that low concentration materials could control the outbreak of cyanobacterial blooms in a short time, while under high concentration conditions, the materials could inhibit Microcystis aeruginosa efficiently and for a long time.

Published
2024
Full Text
View/download PDF

24. Effects of wheat germ diet on intestinal antioxidant capacity, immunological function and gut microbiota of Sichuan white geese.

Author

Wang X, Jiang D, An X, Li S, Qi Y, Yang Y, Wang Z, Sun Q, Ling W, Ji C, Qi Y, Xu H, Han C, Zhao H, and Kang B

Abstract

Background: Wheat germ is known for its antioxidant, anti-inflammatory, and disease resistance properties in animals. However, its effect on the gut of Sichuan white geese remains unclear., Method: In this study, thirty 250-day-old geese were divided into three equal groups, the control group, LWG group (21.8% wheat germ) and HWG group (43.6% wheat germ), the experiment lasted 12 weeks. We assessed various aspects of geese intestinal health, including barrier function, digestibility, antioxidant capacity, immunity, microbiota, and metabolism., Results: The study revealed a significant increase in villus height (VH), villus height-to-crypt depth (VH/CD) ratio, amylase, and lipase activities in the duodenum and ileum, increased putrescine levels in the duodenum and jejunum, as well as spermidine levels in the jejunum ( P < 0.05). LWG increased the total antioxidant capacity (T-AOC) in the duodenum, while decreasing levels of intestinal malondialdehyde (MDA), serum lipopolysaccharide (LPS), interleukin-6 (IL-6), and diamine oxidase (DAO) activity ( P < 0.05). Furthermore, LWG increased the relative abundance of Oscillospiraceae&#95;unclassified , Ligilactobacillus , and Roseburia , as well as increased levels of acetic acid, butyric acid, and valeric acid, while decreasing the relative abundance of Subdoligranulum , Flavonifractor , and Klebsiella . Additionally, we observed 17 up-regulated genes and 25 down-regulated genes in the jejunum, which are associated with the cell cycle and immunity. These genes play roles in pathways such as the p53 signaling pathway, cell cycle regulation, and pathways associated with immune modulation. On the other hand, HWG increased intestinal VH and spermidine levels, as well as amylase and lipase activities in the duodenum ( P < 0.05). It also elevated ileal T-AOC and sIgA levels ( P < 0.05), while reducing intestinal MDA content, serum LPS levels, DAO activity, and propionic acid in cecum contents ( P < 0.05). Moreover, HWG increased the relative abundance of Ligilactobacillus , Oscillospiraceae&#95;unclassified , and Roseburia ( P < 0.05)., Conclusion: Overall, wheat germ diets, particularly the LWG diet demonstrated the ability to enhance antioxidant capacity, digestibility, immunity, and barrier properties of the intestinal tract, while modulating the gut microbiota and metabolism. Therefore, wheat germ diets hold promise in improving intestinal health by preserving barrier function and regulating flora structure., Competing Interests: The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest., (Copyright © 2024 Wang, Jiang, An, Li, Qi, Yang, Wang, Sun, Ling, Ji, Qi, Xu, Han, Zhao and Kang.)

Published
2024
Full Text
View/download PDF

25. An Investigation of the Correlation Between Retinal Nerve Fiber Layer Thickness with Blood Biochemical Indices and Cognitive Dysfunction in Patients with Type 2 Diabetes Mellitus.

Author

Zhu X, Jiang D, Zhang H, Cai R, Wang Y, and Hua F

Abstract

Objective: The study aimed to explore the correlation between retinal nerve fiber layer thickness (RNFLT) with blood biochemical indicators and cognitive dysfunction in patients with type 2 diabetes mellitus (T2DM) and the possible mechanism, thereby providing more theoretical basis for the occurrence and prevention of diabetes related complications., Methods: Eighty T2DM patients treated in our hospital from March 2022 to September 2022 were selected as the study subjects, and the clinical data of the patients were retrospectively analyzed. All patients underwent fundus fluorescein angiography (FFA) to analyze the changes in retinal blood vessels. Patients who met the inclusion criteria were divided as the diabetic retinopathy (DR) group (n=46) and simple diabetes group (n=34). The RNFLT, blood biochemical indexes and changes in cognitive functions of the patients were detected. The correlation between RNFLT with blood biochemical indexes and cognitive dysfunction was analyzed., Results: Compared with the simple diabetes group, patients in the DR group had much lower mean, nasal, inferior and superior thicknesses ( P <0.01). There existed no significant difference in blood pressure, body mass index (BMI), blood lipids (triglycerides, cholesterol, low-density lipoprotein, high-density lipoprotein) between the two groups ( P >0.05). Compared with the simple diabetes group, patients in the DR group had much higher fasting blood glucose (FBG), hemoglobin A1c (HbA1c), fasting insulin (FINS), insulin resistance (HOMA-IR) index, apolipoprotein B (ApoB)/apolipoprotein A1 (ApoA1) ( P <0.001). Besides, the DR group had sharply lower scores on the Mini-Mental State Examination (MMSE) scale and higher levels of the Trail Making Test-A (TMT-A) and TMT-B ( P <0.001). Spearman correlation analysis confirmed that the mean RNFLT was negatively correlated with the levels of FBG, HbA1c, HOMA-IR index, TMT-A and TMT-B ( P <0.05), positively correlated with the score of mini-mental state examination (MMSE) ( P <0.05), and was no significant correlation with FINS and ApoB/ApoA1 ( P >0.05)., Conclusion: DR patients had significantly reduced RNFLT, elevated levels of blood glucose related indicators, and cognitive dysfunction. There existed a correlation between RNFLT and FBG, HbA1c, HOMA-IR index, TMT-A, TMT-B and MMSE., Competing Interests: The authors declare that they have no competing interests., (© 2024 Zhu et al.)

Published
2024
Full Text
View/download PDF

26. Metabolic profiles in gestational diabetes mellitus can reveal novel biomarkers for prediction of adverse neonatal outcomes.

Author

Yin X, Yu T, Jiang D, Shan C, Xia J, Su M, Zhang M, Chen L, Zhong H, Cui X, and Ji C

Abstract

Background: Gestational diabetes mellitus (GDM) significantly affects the fetal metabolic environment, elevating risks of neonatal hypoglycemia and macrosomia. Metabolomics offers promising avenues for early prediction and diagnosis of GDM and associated adverse offspring outcomes., Methods: This study analyzed serum samples from pregnant women diagnosed with GDM at 24 to 28 weeks of gestation using untargeted metabolomics. We monitored the health outcomes of their offspring to explore the correlation between initial serum metabolite profiles and subsequent health outcomes, to uncover the predictive markers for hypoglycemia and macrosomia in these offspring., Results: Out of 200 participants, 154 had normal newborns, 33 had offspring with hypoglycemia, and 19 had offspring with macrosomia. From 448 identified metabolites, 66 showed significant differences in cases of hypoglycemia, and 45 in macrosomia. A panel of serum metabolite biomarkers achieved Area Under the Curve (AUC) values of 0.8712 for predicting hypoglycemia and 0.9434 for macrosomia., Conclusion: The study delineated metabolic disruptions in GDM during 24-28 weeks of gestation and pinpointed biomarkers capable of forecasting adverse neonatal outcomes. These findings could inform GDM management strategies and minimize the incidence of such outcomes., Competing Interests: The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest., (© 2024 Yin, Yu, Jiang, Shan, Xia, Su, Zhang, Chen, Zhong, Cui and Ji.)

Published
2024
Full Text
View/download PDF

27. Visualization of the Extracranial Branches of the Trigeminal Nerve Using Improved Motion-Sensitized Driven Equilibrium-Prepared 3D Inversion Recovery TSE Sequence.

Author

She D, Huang H, Jiang D, Hong J, You P, Li L, Zhao X, and Cao D

Subjects
Humans, Male, Female, Prospective Studies, Adult, Middle Aged, Aged, Young Adult, Contrast Media, Image Enhancement methods, Trigeminal Nerve Diseases diagnostic imaging, Trigeminal Nerve diagnostic imaging, Imaging, Three-Dimensional methods, Magnetic Resonance Imaging methods
Abstract

Background and Purpose: Visualization of the extracranial trigeminal nerve is crucial to detect nerve pathologic alterations. This study aimed to evaluate visualization of the extracranial trigeminal nerve using 3D inversion recovery TSE with an improved motion-sensitized driven equilibrium (iMSDE) pulse., Materials and Methods: In this prospective study, 35 subjects underwent imaging of the trigeminal nerve using conventional 3D inversion recovery TSE, 3D inversion recovery TSE with an iMSDE pulse, and contrast-enhanced 3D inversion recovery TSE. The visibility of 7 extracranial branches of the trigeminal nerve, venous/muscle suppression, and identification of the relationship between nerves and lesions were scored on a 5-point scale system. In addition, SNR, nerve-muscle contrast ratio, nerve-venous contrast ratio, nerve-muscle contrast-to-noise ratio, and nerve-venous contrast-to-noise ratio were calculated and compared., Results: Images acquired with iMSDE 3D inversion recovery TSE had significantly higher nerve-muscle contrast ratio, nerve-venous contrast ratio, and nerve-to-venous contrast-to-noise ratio (all P  < .001); improved venous/muscle suppression and clearer visualization of the trigeminal nerve branches except the ophthalmic nerve than with conventional 3D inversion recovery TSE (all P  < .05). Compared with contrast-enhanced 3D inversion recovery TSE, images acquired with iMSDE 3D inversion recovery TSE had significantly higher SNR, nerve-muscle contrast ratio, and nerve-to-venous contrast-to-noise ratio (all P  < .05) , and demonstrated comparable diagnostic quality (scores ≥3) of the maxillary nerve, mandibular nerve, inferior alveolar nerve, lingual nerve, and masseteric nerve ( P  > .05). As for the identification of the relationship between nerves and lesions, iMSDE 3D inversion recovery TSE showed the highest scores among these 3 sequences (all P  < .05)., Conclusions: The iMSDE 3D inversion recovery TSE is a promising alternative to conventional 3D inversion recovery TSE and contrast-enhanced 3D inversion recovery TSE for visualization of the extracranial branches of trigeminal nerve in clinical practice., (© 2024 by American Journal of Neuroradiology.)

Published
2024
Full Text
View/download PDF

28. Facial Action Unit Representation Based on Self-Supervised Learning With Ensembled Priori Constraints.

Author

Chen H, Zhang P, Guo C, Lu K, and Jiang D

Subjects
Humans, Facial Muscles physiology, Databases, Factual, Facial Expression, Face diagnostic imaging, Face physiology, Supervised Machine Learning, Video Recording methods, Image Processing, Computer-Assisted methods, Algorithms
Abstract

Facial action units (AUs) focus on a comprehensive set of atomic facial muscle movements for human expression understanding. Based on supervised learning, discriminative AU representation can be achieved from local patches where the AUs are located. Unfortunately, accurate AU localization and characterization are challenged by the tremendous manual annotations, which limits the performance of AU recognition in realistic scenarios. In this study, we propose an end-to-end self-supervised AU representation learning model (SsupAU) to learn AU representations from unlabeled facial videos. Specifically, the input face is decomposed into six components using auto-encoders: five photo-geometric meaningful components, together with 2D flow field AUs. By constructing the canonical neutral face, posed neutral face, and posed expressional face gradually, these components can be disentangled without supervision, therefore the AU representations can be learned. To construct the canonical neutral face without manually labeled ground truth of emotion state or AU intensity, two priori knowledge based assumptions are proposed: 1) identity consistency, which explores the identical albedos and depths of different frames in a face video, and helps to learn the camera color mode as an extra cue for canonical neutral face recovery. 2) average face, which enables the model to discover a 'neutral facial expression' of the canonical neutral face and decouple the AUs in representation learning. To the best of our knowledge, this is the first attempt to design self-supervised AU representation learning method based on the definition of AUs. Substantial experiments on benchmark datasets have demonstrated the superior performance of the proposed work in comparison to other state-of-the-art approaches, as well as an outstanding capability of decomposing input face into meaningful factors for its reconstruction. The code is made available at https://github.com/Sunner4nwpu/SsupAU.

Published
2024
Full Text
View/download PDF

Catalog

Books, media, physical & digital resources
See catalog results