Back to Search
Start Over
Image-Text Surgery: Efficient Concept Learning in Image Captioning by Generating Pseudopairs.
- Source :
-
IEEE Transactions on Neural Networks & Learning Systems . Dec2018, Vol. 29 Issue 12, p5910-5921. 12p. - Publication Year :
- 2018
-
Abstract
- Image captioning aims to generate natural language sentences to describe the salient parts of a given image. Although neural networks have recently achieved promising results, a key problem is that they can only describe concepts seen in the training image-sentence pairs. Efficient learning of novel concepts has thus been a topic of recent interest to alleviate the expensive manpower of labeling data. In this paper, we propose a novel method,Image-Text Surgery, to synthesize pseudoimage-sentence pairs. The pseudopairs are generated under the guidance of a knowledge base, with syntax from a seed data set (i.e., MSCOCO) and visual information from an existing large-scale image base (i.e., ImageNet). Via pseudodata, the captioning model learns novel concepts without any corresponding human-labeled pairs. We further introduce adaptive visual replacement, which adaptively filters unnecessary visual features in pseudodata with an attention mechanism. We evaluate our approach on a held-out subset of the MSCOCO data set. The experimental results demonstrate that the proposed approach provides significant performance improvements over state-of-the-art methods in terms of F1 score and sentence quality. An ablation study and the qualitative results further validate the effectiveness of our approach. [ABSTRACT FROM AUTHOR]
- Subjects :
- *ARTIFICIAL intelligence
*ARTIFICIAL neural networks
*MACHINE learning
Subjects
Details
- Language :
- English
- ISSN :
- 2162237X
- Volume :
- 29
- Issue :
- 12
- Database :
- Academic Search Index
- Journal :
- IEEE Transactions on Neural Networks & Learning Systems
- Publication Type :
- Periodical
- Accession number :
- 133211358
- Full Text :
- https://doi.org/10.1109/TNNLS.2018.2813306