Search

Multimodal fusion algorithm is one of the solutions to the problem of urban road video caption which only considers visual information and ignores the equally important audio information. Existing multimodal fusion algorithms based on Transformer all have the problem of low fusion performance between modes and high computational complexity. In order to improve the interaction between multimodal information, This paper recently proposed a new Transformer based model called Multimodal Attention Bottleneck for Video Captioning (MABVC). Firstly, this paper uses pre-trained I3 D and VGGish networks to extract visual and audio features of video and input the extracted features into Transformer model. Then, the decoder part will train the information of the two modes respectively and perform multimodal fusion. Finally, the model processes the results of the decoder and generates text captions that people can understand. This paper conducted a comparison experiments using data sets MSR-VTT, MSVD and self-built data sets BUUISE, and validated model results using evaluation metrics the model. The experimental results show that the video caption model based on multimodal attention fusion has obvious improvement in all indicators. The model can still achieve good results on traffic scene data sets, and has great application prospects/can be promoted and applied in intelligent driving industry. [ABSTRACT FROM AUTHOR]

Aiming at the problems of inaccurate large target segmentation and loss of small target information in current real-time semantic segmentation methods, this paper proposed a real-time semantic segmentation algorithm based on multi-branch networks. First of all, this paper optimized the bilateral segmentation network, and designed pyramid branches to expand the receptive field to cover large objects in the field of view and fully combine context information; secondly, designed a bilateral guidance fusion module to map deep and shallow features Provide guidance information to make up for the loss of small target information. Finally, this paper verified the proposed method on the Cityscapes dataset. The experimental results show that the proposed model achieves an average intersection ratio of 77.8% at an inference speed of 51.3 FPS, and the accuracy is increased by 2.5 percentage points compared with the baseline. The proposed method adopts the pyramid branch to obtain the characteristics of semantic edge regions at different scales while expanding the receptive field, and enhance the modeling ability of semantic boundaries, and the proposed bilateral guidance fusion module can more effectively integrate features of different levels, Compensating for the information loss caused by downsampling can better guide model learning. [ABSTRACT FROM AUTHOR]