Search

Image captioning is one essential work in the multi-modal area, which employs computer vision and natural language processing technology together to describe image content. Most current methods employ the encoder–decoder framework to achieve satisfactory results. Recently, transformers have been extensively utilized in image captioning tasks and earned satisfactory results. Nevertheless, transformers pay more attention to the global features in images and divide images into fixed sizes and process them separately. Those models can not capture the relationship between the internal components in image, such as the relationships between objects, regions and object attributes. In this article, we introduce a novel Residual Swin Transformer and Actor-Critic (RSTAC) in the image captioning task. RSTAC consists of two modules: Residual Swin Transformer and Actor-Critic modules. In particular, we employ a residual network to preserve the vanilla feature in the first place and several residual Swin Transformer blocks with convolution operations to obtain the mid-level features, which are also composed of a residual network and several Swin Transformer blocks and convolution operations. It aims to explore the internal correlation between image content and obtain multi-view features in the image. Then, a policy network is utilized to explore possible words and forecast the future generation word. A value network is employed to calculate the reward of generated sentences whose goal is directly optimizing non-differentiable quality metrics and enhancing the performance of generated sentences. Experiments reveal that our model surpasses other competitive models and performs better on the MSCOCO and Flickr30k datasets. [ABSTRACT FROM AUTHOR]

Most current multi-modal summarization methods follow a cascaded manner, where an off-the-shelf object detector is first used to extract visual features. After that, these visual features are fused with language representations for the decoder to generate the text summary. However, the cascaded way employs separate encoders for different modalities, which makes it hard to learn the joint vision and language representation. In addition, they also ignore the semantics alignment between paragraphs and images for multi-modal summarization tasks, which are crucial to a precise summary. To tackle these issues, in this paper, we propose ViL-Sum to jointly model paragraph-level Vision-Language Semantic Alignment and Multi-Modal Summarization. Our ViL-Sum contains two components for better learning multi-modal semantics and aims to align them. The first one is a joint multi-modal encoder. The other one is two well-designed tasks for multi-task learning, including image reordering and image selection. Specifically, the joint multi-modal encoder converts images into visual embeddings and attaches them with text embedding as the input of the encoder. The reordering task guides the model to learn paragraph-level semantic alignment, and the selection task guides the model to select summary-related images in the final summary. Experimental results show that our proposed ViL-Sum outperforms current state-of-the-art methods on most automatic and manual evaluation metrics. In further analysis, we find that two well-designed tasks and a joint multi-modal encoder can effectively guide the model to learn reasonable paragraph-image and summary-image relations. [ABSTRACT FROM AUTHOR]