Search

Recent captioning models are limited in their ability to describe concepts unseen in paired image–sentence pairs. This study presents a framework of multi‐task learning for describing novel words not present in existing image‐captioning datasets. The authors’ framework takes advantage of external sources‐labelled images from image classification datasets, and semantic knowledge extracted from the annotated text. They propose minimising a joint objective which can learn from these diverse data sources and leverage distributional semantic embeddings. When in the inference step they change the BeamSearch step by considering both the caption model and language model enabling the model to generalise novel words outside of image‐captioning datasets. They demonstrate that in the framework by adding an annotated text data which can help the image captioning model to describe images with the right corresponding novel words. Extensive experiments are conducted on both AI Challenger and Microsoft coco (MSCOCO) image captioning datasets of two different languages, demonstrating the ability of their framework to describe novel words such as scenes and objects.

This study reveals associative processes for novel words in a second language (L2) and their referents. Thirty Japanese participants learned associative pairs for novel words in Chinese and pictorial referents (CP), as well as novel words in Chinese and words in Japanese (CJ), against a condition in which they learned only novel words in Chinese (C). After the learning phase, participants conducted two learning condition retrieval tasks for word recognition and three recognition tasks for the source-monitoring of the referents. The correct answers for each recognition task were provided to participants after each trial. Although the correct answers in all conditions increased in both the recognition and learning condition retrieval tasks, there was no significant difference among conditions. In contrast, the response times of the correct trials in all recognition tasks and the first learning condition retrieval tasks were faster for the CP condition than the CJ condition. Additionally, in the second learning condition retrieval task, missed items in associative conditions (CP and CJ conditions) were judged to be learned items more often than unlearned items, whereas missed items in the non-associative condition (C condition) were judged to be unlearned items more than learned items. These findings suggest that pictures contribute to the recognition and retrieval speeds of associations between novel words in L2 and referents, and that associative learning of L2 words and referents could enhance more familiarity effects than the learning of L2 words only.

The concreteness effect (CE) describes a processing advantage for concrete over abstract words. Electrophysiologically, the CE manifests in higher N400 and N700 amplitudes for concrete words. The contribution of the stimulus-inherent imageability to the electrophysiological correlates of the CE is not yet fully unraveled. This EEG study focused on the role of imageability irrespective of concreteness by examining the effects of training-induced visual imageability on the processing of novel words. In two training sessions, 21 healthy participants learned to associate novel words with pictures of novel objects as well as electron-microscopical structures and were additionally familiarized with novel words without any picture association. During a post-training EEG session, participants categorized trained novel words with or without picture association, together with real concrete and abstract words. Novel words associated with novel object pictures during the training elicited a higher N700 than familiarized novel words without picture-association. Crucially, this training-induced N700 effect resembled the CE found for real words. However, a CE on the N400 was found for real words, but no effect of imageability in novel words. The results suggest that the N400 CE for real words depends on the integration of multiple semantic features, while mere visual imageability might contribute to the CE in the N700 time window.