Back to Search
Start Over
Novel Object Captioning with Semantic Match from External Knowledge
- Source :
- Applied Sciences; Volume 13; Issue 13; Pages: 7868
- Publication Year :
- 2023
- Publisher :
- Multidisciplinary Digital Publishing Institute, 2023.
-
Abstract
- Automatically describing the content of an image is a challenging task that is on the edge between natural language and computer vision. The current image caption models can describe the objects that are frequently seen in the training set very well, but they fail to describe the novel objects that are rarely seen or never seen in the training set. Despite describing novel objects being important for practical applications, only a few works investigate this issue. Furthermore, those works only investigate rarely seen objects, but ignore the never-seen objects. Meanwhile, the number of never-seen objects is more than the number of frequently seen and rarely seen objects. In this paper, we propose two blocks that incorporate external knowledge into the captioning model to solve this issue. Initially, in the encoding phase, the Semi-Fixed Word Embedding block is an improvement for the word embedding layer that enables the captioning model to understand the meaning of the arbitrary visual words rather than a fixed number of words. Furthermore, the Candidate Sentences Selection block chooses candidate sentences by semantic matching rather than probability, avoiding the influence of never-seen words. In experiments, we qualitatively analyze the proposed blocks and quantitatively evaluate several captioning models with the proposed blocks on the Nocaps dataset. The experimental results show the effectiveness of the proposed blocks for novel objects, especially when describing never-seen objects, CIDEr and SPICE improved by 13.1% and 12.0%, respectively.
- Subjects :
- image captioning
novel object captioning
semantic match
Subjects
Details
- Language :
- English
- ISSN :
- 20763417
- Database :
- OpenAIRE
- Journal :
- Applied Sciences; Volume 13; Issue 13; Pages: 7868
- Accession number :
- edsair.multidiscipl..b4f18b053b38619b18fc8df67a2c05e5
- Full Text :
- https://doi.org/10.3390/app13137868