Algorithm design of large model of belt tearing based on multi-modality

The AI mine model is a mine intelligent solution based on artificial intelligence. Using big data, deep learning, machine learning and other technologies, it can help mining companies improve production efficiency and safety. In terms of conveyor belt tear detection, a network structure based on Transformer to process multi-modal data was designed based on the large AI mine model, and the DETR-Audio model was proposed to splicing and fusing multi-modal data of video and audio, using the DERT model to encode the video, use the short-time Fourier transform to analyze the time-spectrum of the audio signal, then splice and fuse the feature vectors of the two, and finally pass them into the decoder for fusion decoding. After being trained and tested on 3,000 pictures of underground conveyor belts in coal mines and corresponding audio data, the model performed well, with higher detection accuracy and robustness than models using video or audio information alone.