基于多模态预训练模型的水稻病虫害图像描述生成研究.

[Objectives] Aiming at the lack of disease description in rice diseases and pests image classification technology, a lightweight rice diseases and pests image description model was proposed in this paper to describe rice diseases and pests image more specifically. [Methods] Ten common rice pests and diseases, such as rice bacterial blight, rice bacterial streak disease, rice bakanae disease, rice three chemical borers, rice blast, rice false smut, rice sheath blight, rice planthopper, rice thrip and rice brown spot were studied, and Chinese description data set of rice pests and diseases image was constructed. Firstly, the multimodal pre-training model CLIP was used to generate image vectors, which contained basic image information and rich semantic information. The mapping network was used to map the image vectors into the text space to generate text prompt vectors. Finally, the language model GPT-2 generates image descriptions according to the prompt vectors. [Results] The test results showed that the indexes of the model in this paper were significantly superior to other models in the image description data set of rice pests and diseases. Compared with the traditional CNN_ LSTM model, the indexes of BLEU-1, BLEU-2, BLEU-3, BLEU-4, ROUGE and METEOR improved 0.26, 0.27, 0.24, 0.22, 0.22 and 0.14, respectively. And the generated image description had the advantages of accurate, detailed and rich semantics. The model was tested by using actual rice field pictures. The actual field scenes were more complex and diverse, and the generated image description index only slightly decreased compared with the data set index, which was still higher than other comparison models. The overall recognition accuracy of the model was 97.28%. [Conclusions] The image description method of rice diseases and pests based on multimodal pre-training model can accurately describe the rice diseases and pests, and provide a new idea for the detection of rice diseases and pests. [ABSTRACT FROM AUTHOR]