Back to Search
Start Over
Multimodal Learning with Transformers: A Survey
- Publication Year :
- 2022
- Publisher :
- arXiv, 2022.
-
Abstract
- Transformer is a promising neural network learner, and has achieved great success in various machine learning tasks. Thanks to the recent prevalence of multimodal applications and big data, Transformer-based multimodal learning has become a hot topic in AI research. This paper presents a comprehensive survey of Transformer techniques oriented at multimodal data. The main contents of this survey include: (1) a background of multimodal learning, Transformer ecosystem, and the multimodal big data era, (2) a theoretical review of Vanilla Transformer, Vision Transformer, and multimodal Transformers, from a geometrically topological perspective, (3) a review of multimodal Transformer applications, via two important paradigms, i.e., for multimodal pretraining and for specific multimodal tasks, (4) a summary of the common challenges and designs shared by the multimodal Transformer models and applications, and (5) a discussion of open problems and potential research directions for the community.<br />Comment: This paper is accepted by IEEE TPAMI
- Subjects :
- FOS: Computer and information sciences
Computer Science - Machine Learning
Computational Theory and Mathematics
Artificial Intelligence
Applied Mathematics
Computer Vision and Pattern Recognition (cs.CV)
Computer Science - Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
Software
Machine Learning (cs.LG)
Subjects
Details
- Database :
- OpenAIRE
- Accession number :
- edsair.doi.dedup.....e6ba7dacaa0aa60ca2d78a1d00948936
- Full Text :
- https://doi.org/10.48550/arxiv.2206.06488