Language guided 3D object detection in point clouds for MEP scenes

Abstract In recent years, contrastive language‐image pre‐training (CLIP) has gained popularity for processing 2D data. However, the application of cross‐modal transferable learning to 3D data remains a relatively unexplored area. In addition, high‐quality, labelled point cloud data for Mechanical, Electrical, and Plumbing (MEP) scenarios are in short supply. To address this issue, the authors introduce a novel object detection system that employs 3D point clouds and 2D camera images, as well as text descriptions as input, using image‐text matching knowledge to guide dense detection models for 3D point clouds in MEP environments. Specifically, the authors put forth the proposition of a language‐guided point cloud modelling (PCM) module, which leverages the shared image weights inherent in the CLIP backbone. This is done with the aim of generating pertinent category information for the target, thereby augmenting the efficacy of 3D point cloud target detection. After sufficient experiments, the proposed point cloud detection system with the PCM module is proven to have a comparable performance with current state‐of‐the‐art networks. The approach has 5.64% and 2.9% improvement in KITTI and SUN‐RGBD, respectively. In addition, the same good detection results are obtained in their proposed MEP scene dataset.