Deep multimodal fusion model for moisture content measurement of sand gravel using images, NIR spectra, and dielectric data.

• A deep multimodal fusion (DMF) model is developed for sand gravel moisture content prediction. • The DMF model effectively extracts and fuses information from image, spectral, and dielectric data. • The DMF model shows superior performance on sand gravel multimodal datasets. • The DMF model maintains good robustness when the dataset was disturbed by general noise. A fast and accurate moisture content (MC) measurement of sand gravel is essential for hydraulic engineering project sites. Most existing measurement methods are unimodal, facing non-robust against external interference. To address this issue, a deep multimodal fusion (DMF) model for measuring the MC of sand gravel using images, near-infrared (NIR) spectra, and dielectric data, is proposed. A modified bottleneck transformer network (BoTNet) added with an extremely efficient spatial pyramid (EESP) block is first proposed to extract image features from different receptive fields. The improved convolutional neural network with attention blocks added (A-CNN) and gated recurrent unit with attention blocks added (A-GRU) networks are then adopted to extract local and sequential features from NIR spectra, respectively. The square root of dielectric data and above multimodal features are effectively fused according to their contribution to the target indicator in the Fusion module. Among other comparative models, the DMF model yielded the best performance (R2 = 0.962, RMSE = 0.645, RPD = 5.124) on the original sand gravel dataset, and still maintained the best accuracy (the average R2 and RPD mostly exceeded 0.85 and 2.5, respectively) when against general external noise. [ABSTRACT FROM AUTHOR]