Search

In order to accurately track progress and improve efficiency in complex construction projects, it's important to effectively monitor individual tasks and measure the time taken to complete a cycle of tasks. Tunnel construction involves a variety of activities, where multiple pieces of equipment are engaged in different actions that occur simultaneously or sequentially during a single activity. This study introduces a contextual audio-visual (multimodal) approach to better recognize multi-equipment activities in a tunnel construction site for monitoring purposes. By incorporating both audio and visual data, and by integrating both spatial and cyclical temporal contexts, the model accurately recognizes the activity being performed by multiple pieces of equipment more often than single-mode models. Tested against real-world operation data, the model achieved a remarkable F-score of 96.3% in recognizing construction activities, demonstrating its superiority over traditional methods in scenarios involving multiple, simultaneously operating pieces of equipment. The results emphasize the potential of contextual multimodal models in enhancing operational efficiency in complex construction sites. • An audio-visual approach is introduced for equipment monitoring in tunnel. • Concurrent activities with multiple pieces of equipment are recognized. • The model integrates spatial and cyclical temporal contexts of activities. • The model achieved a F-score of 96.3% in recognizing activities. [ABSTRACT FROM AUTHOR]

In this paper, the estimation of the ideal ratio mask (IRM) has been carried out based on speech cochleagram and visual cues using Audio-Visual Multichannel Convolutional Neural Network (AVMCNN) to enhance the speech signal. Recently several researchers have shown that speech enhancement using visual data as an additional input along with the audio data is more effective in minimizing the acoustic noise present in the speech signal. This work proposes a novel CNN-based audio-visual IRM estimation model. In the proposed audio-visual IRM estimation model, the dynamics of both audio and visual signal features are extracted using multichannel CNN and contextually combined for speech enhancement. The enhanced speech obtained using the proposed model is evaluated based on speech quality and intelligibility. The evaluation results signify that the proposed audio-visual mask estimation model shows improved performance over the audio-only, visual-only, and existing audio-visual mask estimation models. In turn, the proposed AVMCNN model proves its effectiveness in combining the dynamics of the audio features with the visual speech features for speech enhancement. [ABSTRACT FROM AUTHOR]