Search

Emotion recognition has recently attracted extensive interest due to its significant applications to human–computer interaction. The expression of human emotion depends on various verbal and non-verbal languages like audio, visual, text, etc. Emotion recognition is thus well suited as a multimodal rather than single-modal learning problem. Owing to the powerful feature learning capability, extensive deep learning methods have been recently leveraged to capture high-level emotional feature representations for multimodal emotion recognition (MER). Therefore, this paper makes the first effort in comprehensively summarize recent advances in deep learning-based multimodal emotion recognition (DL-MER) involved in audio, visual, and text modalities. We focus on: (1) MER milestones are given to summarize the development tendency of MER, and conventional multimodal emotional datasets are provided; (2) The core principles of typical deep learning models and its recent advancements are overviewed; (3) A systematic survey and taxonomy is provided to cover the state-of-the-art methods related to two key steps in a MER system, including feature extraction and multimodal information fusion; (4) The research challenges and open issues in this field are discussed, and promising future directions are given. [ABSTRACT FROM AUTHOR]

Deep learning-based automatic sleep staging methods have been widely applied for sleep scoring and sleep diagnosis. However, most methods consider only a single temporal scale when dealing with sleep signals. Furthermore, these methods are limited to target only a single-age group or single dataset. In this paper, we propose a multi-scale temporally focused sleep staging model, MAGSleepNet, which can be used for multi-age groups simultaneously. MAGSleepNet consists of (1) a group age classification (GAC) module that can offer a preliminary screening on epidemic estimation of multiple age, (2) a dimensional expansion module (DEM) that can expand the dimension of the input signals, (3) a sequential multi-scale convolutional neural network (SMCNN) that extracts multi-scale features and short-time temporal information, and (4) sequence temporal encoder (STE) that extracts sequential temporal information. In addition, two auxiliary tasks are used to complement the short-time temporal information and to reassign the probabilities of different age groups to enhance the model robustness, respectively. The MAGSleepNet is evaluated in adult, child and infant, tested on MASS, CHAT, and CHFU datasets with accruacy of 86.7%, 80.1% and 66.5%, outperforming the state-of-the-art methods. Based on the excellent performance of the proposed method, it is expected to pave the way for automatic sleep staging methods with strong generalizability for multiple age groups. • A novel model is proposed for simultaneous sleep staging in multi-age groups. • The hierarchical architecture of model ensures the effective feature extraction. • An auxiliary task is used to provide temporal information on different scales. • Numerous experiments are conducted to evaluate the proposed model. [ABSTRACT FROM AUTHOR]