Affiliation:
1. China Coal Shaanxi Yulin Energy & Chemical Co., Ltd. of China National Coal Group Co. Yulin Shanxi China
2. China Coal Electric Co., Ltd of China National Coal Group Co. Beijing China
3. IOT Perception Mine Research Center China University of Mining and Technology Xuzhou Jiangsu China
4. Institute of Electrodynamics and Microelectronics University of Bremen Bremen Germany
Abstract
AbstractIn recent years, contrastive language‐image pre‐training (CLIP) has gained popularity for processing 2D data. However, the application of cross‐modal transferable learning to 3D data remains a relatively unexplored area. In addition, high‐quality, labelled point cloud data for Mechanical, Electrical, and Plumbing (MEP) scenarios are in short supply. To address this issue, the authors introduce a novel object detection system that employs 3D point clouds and 2D camera images, as well as text descriptions as input, using image‐text matching knowledge to guide dense detection models for 3D point clouds in MEP environments. Specifically, the authors put forth the proposition of a language‐guided point cloud modelling (PCM) module, which leverages the shared image weights inherent in the CLIP backbone. This is done with the aim of generating pertinent category information for the target, thereby augmenting the efficacy of 3D point cloud target detection. After sufficient experiments, the proposed point cloud detection system with the PCM module is proven to have a comparable performance with current state‐of‐the‐art networks. The approach has 5.64% and 2.9% improvement in KITTI and SUN‐RGBD, respectively. In addition, the same good detection results are obtained in their proposed MEP scene dataset.
Funder
National Key Research and Development Program of China
Publisher
Institution of Engineering and Technology (IET)
Subject
Computer Vision and Pattern Recognition,Software
Reference51 articles.
1. Lidar
2. 3-D Mapping With an RGB-D Camera
3. Debiased contrastive learning;Chuang C.‐Y.;Adv. Neural Inf. Process. Syst.,2020
4. Chen X. et al.:Improved Baselines with Momentum Contrastive Learning(2020). arXiv preprint arXiv:2003.04297