Search

With the proliferation of user-generated online videos, Multimodal Sentiment Analysis (MSA) has attracted increasing attention recently. Despite significant progress, there are still two major challenges on the way towards robust MSA: 1) inefficiency when modeling cross-modal interactions in unaligned multimodal data; and 2) vulnerability to random modality feature missing which typically occurs in realistic settings. In this paper, we propose a generic and unified framework to address them, named Efficient Multimodal Transformer with Dual-Level Feature Restoration (EMT-DLFR). Concretely, EMT employs utterance-level representations from each modality as the global multimodal context to interact with local unimodal features and mutually promote each other. It not only avoids the quadratic scaling cost of previous local-local cross-modal interaction methods but also leads to better performance. To improve model robustness in the incomplete modality setting, on the one hand, DLFR performs low-level feature reconstruction to implicitly encourage the model to learn semantic information from incomplete data. On the other hand, it innovatively regards complete and incomplete data as two different views of one sample and utilizes siamese representation learning to explicitly attract their high-level representations. Comprehensive experiments on three popular datasets demonstrate that our method achieves superior performance in both complete and incomplete modality settings. [ABSTRACT FROM AUTHOR]

Neuromorphic computing is an emerging research field that aims to develop new intelligent systems by integrating theories and technologies from multiple disciplines, such as neuroscience, deep learning and microelectronics. Various software frameworks have been developed for related fields, but an efficient framework dedicated to spike-based computing models and algorithms is lacking. In this work, we present a Python-based spiking neural network (SNN) simulation and training framework, named SPAIC, that aims to support brain-inspired model and algorithm research integrated with features from both deep learning and neuroscience. To integrate different methodologies from multiple disciplines and balance flexibility and efficiency, SPAIC is designed with a neuroscience-style frontend and a deep learning-based backend. Various types of examples are provided to demonstrate the wide usability of the framework, including neural circuit simulation, deep SNN learning and neuromorphic applications. As a user-friendly, flexible, and high-performance software tool, it will help accelerate the rapid growth and wide applicability of neuromorphic computing methodologies. [ABSTRACT FROM AUTHOR]