Search

Intelligent analysis of 3D point clouds has become a frontier in emerging fields such as autonomous driving, digital twins, and the metaverse. Precise segmentation of 3D point clouds is particularly important within these domains; however, it faces several challenges: (1) point cloud data inherently lacks structured topological information; (2) point cloud shapes are complex and highly variable, making it difficult to utilize semantic priors; and (3) the sampling process of point clouds may result in sparse and uneven data. To address these issues, this paper proposes a novel Point Cloud Segmentation Network based on multi-scale feature fusion and Transformer architecture (MFFTNet). MFFTNet enhances the performance of existing segmentation methods by globally modeling the overall point cloud shape and embedding local point cloud details. Specifically, MFFTNet divides the segmentation task into encoding and decoding stages. The encoder is designed as a hierarchical pyramid structure that extracts relatively sparse local center points and fuses local features during progressive downsampling. It also utilizes a Transformer for global feature modeling to establish multi-scale topological and semantic information of the point cloud. Subsequently, multi-scale feature fusion further enhances the network’s perception of local features and global structure. The decoder progressively upsamples to restore the original point cloud and injects multi-scale feature information to achieve precise segmentation. Based on the aforementioned encoding-decoding structure and multi-scale feature fusion, MFFTNet outperforms existing methods on the point cloud semantic segmentation datasets ShapeNetPart and S3DIS.

Multispectral LiDAR can rapidly acquire 3D and spectral information of objects, providing richer features for point cloud semantic segmentation. Despite the remarkable performance of existing graph neural networks in point cloud segmentation, extracting local features still poses challenges in multispectral LiDAR point cloud scenes due to the uneven distribution of geometric and spectral information. To address the prevailing challenges, cutting-edge research predominantly focuses on extracting multiscale local features, compensating for feature extraction shortcomings. Thus, we propose a multiscale adjacency matrix convolutional neural network (MS-AMCNN) for multispectral LiDAR point cloud segmentation. In the MS-AMCNN, a local adjacency matrix convolution module was first proposed to efficiently leverage the point cloud's topological relationships and perceive local geometric features. Subsequently, a multiscale feature extraction architecture was adopted to fuse local geometric features and utilize a global self-attention module to globally model the semantic features of multiscale. The network effectively captures global and local representative features of the point cloud by harnessing the capabilities of convolutional neural networks in local feature modeling and the self-attention mechanism in global semantic feature learning. Experimental results on the Titan dataset demonstrate that the proposed MS-AMCNN network achieves a promising multispectral LiDAR point cloud segmentation performance with an overall accuracy of 94.39% and a mean intersection over union (MIoU) of 86.57%. Compared with other state-of-the-art methods, such as DGCNN, which achieved an MIoU of 85.43%, and RandLA-net, with an MIoU of 85.20%, the proposed approach achieves optimal performance in segmentation.

Aiming at the problem of poor edge effect segmentation in point cloud segmentation, which fails to fully utilize the correlation between the local geometric and semantic features of point cloud.We propose an edge-enhanced graph convolution point cloud segmentation model EdgeGCN-Net, aiming at further exploring the method of point cloud segmentation based on graph convolution neural network. The EdgeGCN-Net employs an enhanced adaptive graph convolutional operator (AdaptConv++) for feature extraction. It constructs semantic and edge branches for joint optimization, aiming to enhance the extraction and comprehension abilities of semantic information and edge features in point clouds. In addition, we propose a graph inference module called AGIM (Attention-based Graph Inference Module), which utilizes the attention mechanism and learnable adjacency matrices to enrich the global knowledge propagation of the channel graph, thus compensating for the loss of geometric information due to the superposition of network layers. Experimental results show that EdgeGCN-Net outperforms previous state-of-the-art segmentation models on both ShapeNetPart and S3DIS datasets with better segmentation performance and higher accuracy, demonstrating its excellent performance in point cloud segmentation tasks.

Point cloud segmentation is a crucial technique for object recognition and localization, widely employed in various applications such as point cloud registration, 3D reconstruction, object recognition, and robotic grasping. However, in practical scenarios, challenges such as sparse object features, significant pose variations, or instances of object adhesion, stacking, and occlusion can lead to difficulties and poor stability in point cloud segmentation. In response to these challenges, this paper proposes a novel point cloud segmentation algorithm based on enhanced Euclidean clustering. To address the issue of over-segmentation and under-segmentation in cases of object adhesion and stacking in unordered scenes, a method is introduced for discriminating collision points based on normal angle constraints. Subsequently, an adaptive search radius is employed to determine the clustering distance threshold, enhancing the algorithm’s ability to handle different object arrangements. Finally, the collision points are reintegrated into the clustering results to ensure the completeness of the segmented target point cloud. Experimental results using the ROBI public dataset and real-world point cloud segmentation scenarios demonstrate that the proposed algorithm effectively resolves challenges associated with the segmentation of adhering and stacking objects. The improved algorithm exhibits higher accuracy and robustness in multi-object segmentation across diverse scenes.

Aiming at the problems of slow manual palletizing, high risk and easy damage to products after the assembly of microchannel heat exchanger, a palletizing system based on 3D visual guidance is presented in this paper. Firstly, an automatic calibration method for RGB-D camera extrinsic parameters was adopted: used random sample consensus algorithm RANSAC for plane fitting, and the pallet was idealized as a plane. The camera extrinsic parameters were calculated according to the rotation relationship between the plane normal vector and the camera coordinate system. When it was performing detection, converted depth data into point cloud data and preprocessed, including coordinate transformation, downsampling, etc. After that, according to the characteristics of stacking, a point cloud segmentation algorithm based on depth and the number of points was used, which could not only segment the pallet and sponge strips, but also effectively detect obstacles. For different parts segmented, the pallet was positioned by fitting the minimum bounding rectangle, and the stacking depths were obtained by calculating the centroid of the sponge strips. Finally, a series of constraints were used to determine whether the unloading conditions were met. The experimental results show that the automatic calibration is effective, the time consuming is about 0.43 s, and the depth range of the pallet is 11 mm after calibration. The algorithm can accurately identify various situations above the stacking and the time consuming is about 1.63 s.

With the development of 3D sensors, 3D point cloud data can now be obtained conveniently. Therefore, it is crucial to process point cloud data automatically. Region growing is a commonly used algorithm to segment point clouds, which greatly depends on the accuracy of points’ normals and requires tuning two thresholds; i.e., the increment of curvature ( $\sigma _{th}$ ) and normal angles ( $\theta _{th}$ ). In this paper, we improve the region growing algorithm in two ways: Accurate normal estimation and strengthening the region growing criteria. For the first aspect, principal component analysis (PCA) is utilized to estimate the initial normals of the point cloud. Then, the points are divided into regular points (RP) and sharp feature points (SFP), according to their initial normals. A robust estimator-based PCA is then applied to refine the SFP normals. For the latter aspect, non-connected and non-coplanar points are detected and ignored when region grows. Finally, the segmentation performance of the proposed method is evaluated using internal and external indices. The results indicate that the proposed method can accurately estimate the point normals within an acceptable time, and obtain a better result than the classic PCA-based region growing algorithm and advanced DetMM-based methods.

Tomographic Synthetic Aperture Radar (TomoSAR) has three-dimensional imaging capability. Never- theless, its building structure is full of considerable noise and wrong targets owing to the baseline distribution and the algorithm restrictions. Accordingly, efficiently extracting and segmenting buildings from SAR point clouds with huge data is a critical issue. According to the characteristics of building facade point clouds, the KD-Tree-based Euclidean clustering is introduced as a fast-clustering method for better extracting and segmenting individual building facades. At first, a simple morphological filter (SMRF) is utilized to identify the ground and non-ground point clouds. The building facades are then extracted from the non-ground point clouds based on density information. Subsequently, the KD-Tree-based Euclidean clustering method denoises the building facades and segments them into individual facades. Finally, a second Euclidean clustering is performed for each of these facades to denoise them again. Experimental results indicate that the presented approach can prepare accurate and efficient extraction and segmentation results for complex urban scenes. [ABSTRACT FROM AUTHOR]

Airborne laser scanning (ALS) point cloud segmentation is an essential procedure for 3D data understanding and applications. This task is challenging due to the unstructured, disordered, and sparse distribution of the point cloud. PointNet++ is a well known end-to-end learning network for point cloud segmentation without fully exploring the local and contextual features, which are less efficient and accurate in capturing the complexity of point clouds. On this basis, we design a novel encoder-decoder network architecture to obtain the semantic features of the ALS point cloud at different levels and achieve a better segmentation effect. The improved local feature aggregation module can merge the deep feature of the point cloud, combining local and global self-attention convolutional networks. It can adaptively explore the inherent semantics feature of points and capture more extensive context information of ALS point cloud, termed DSPNet++. Finally, the conditional random field optimization model can be used to refine the segmentation results. We evaluated the performance of our method on the Vaihingen dataset of the International Society for Photogrammetry and Remote Sensing (ISPRS) and the GML(B) 3D dataset. Experimental results show that our method fully exploits the semantic feature of the ALS point cloud and can achieve higher accuracy. A comparative study with established deep learning models also confirms that our proposed method has outstanding performance in the ALS point cloud segmentation task.

Self-driving baggage tractors on airport ramps or aprons enable better airport operation procedures and support the expansion of the aviation market. Airport ramps have unique mobility requirements in terms of layout, population, demand, and patterns. Avoiding aircraft movement on an airport apron is a top priority because of critical security and safety issues. Existing aircraft detection approaches use remote-sensing images or surveillance cameras. However, these are not compatible with sensors for low-height equipment at airport ramps. Similarly, public road-based self-driving studies have not considered detecting the massive size and concave contours of movable objects. Camera sensors cannot accurately measure the distance of concave contours, whereas a lidar sensor cannot easily cluster or classify an object among point cloud data. In this paper, we present the fusion of cameras and lidar sensors for aircraft and object detection at airport ramps. We use parallel detection from lidar and camera sensors and then integrate both detection results to compensate for any issues. Using the proposed energy optimization model by adapting a conditional random field, we can handle over- and under-segmentation of the point cloud objects caused by the sparse point cloud generated by the aircraft. Our algorithm achieves 31.1% improvement on tracking and 5.5% improvement on classification over other fusion algorithms when applied to a dataset acquired from the Cincinnati and Northern Kentucky airport. [ABSTRACT FROM AUTHOR]

Most existing point cloud instance and semantic segmentation methods rely heavily on strong supervision signals, which require point-level labels for every point in the scene. However, such strong supervision suffers from large annotation costs, arousing the need to study efficient annotating. In this paper, we discover that the locations of instances matter for both instance and semantic 3D scene segmentation. By fully taking advantage of locations, we design a weakly-supervised point cloud segmentation method that only requires clicking on one point per instance to indicate its location for annotation. With over-segmentation for pre-processing, we extend these location annotations into segments as seg-level labels. We further design a segment grouping network (SegGroup) to generate point-level pseudo labels under seg-level labels by hierarchically grouping the unlabeled segments into the relevant nearby labeled segments, so that existing point-level supervised segmentation models can directly consume these pseudo labels for training. Experimental results show that our seg-level supervised method (SegGroup) achieves comparable results with the fully annotated point-level supervised methods. Moreover, it outperforms the recent weakly-supervised methods given a fixed annotation budget. Code is available at https://github.com/antao97/SegGroup. [ABSTRACT FROM AUTHOR]

In this paper, we introduce a novel point-to-surface representation for 3D point cloud learning. Unlike the previous methods that mainly adopt voxel, mesh, or point coordinates, we propose to tackle this problem from a new perspective: learn a set of quadratic terms based static and global reference surfaces to describe 3D shapes, such that the coordinates of a 3D point (x, y, z) can be extended to quadratic terms (xy, xz, yz, $\ldots $) and transformed to the relationship between the local point and the global reference surfaces. Then, the static surfaces are changed into dynamic surfaces by adaptive contribution weighting to improve the descriptive capability. Towards this end, we propose our point-to-surface representation, a new representation for 3D point cloud learning that has not been attempted before, which can assemble local and global geometric information effectively by building connections between the point cloud and the learned reference surfaces. Given 3D points, we show how the reference surfaces are constructed, and how they are inserted into the 3D learning pipeline for different tasks. The experimental results confirm the effectiveness of our new representation, which has outperformed the state-of-the-art methods on the tasks of 3D classification and segmentation. [ABSTRACT FROM AUTHOR]

Unsupervised domain adaptation (UDA) significantly reduces the gap between the source domain and the target domain in machine learning and computer vision tasks. Most UDA approaches are applied to images and videos, and only a few methods implement domain adaptation on 3-D computer vision problems. The existing UDA approaches operating on point clouds try to extract domain-invariant features in different domains for feature alignment. However, higher commonality brings less diversity and results a loss of detailed information. In this article, we propose a novel dual-branch feature alignment network (DFAN) architecture for domain adaptation on point cloud visual tasks to better exploit the respective characteristics of local and global features. Our approach specializes in the extraction and alignment of global and local features with different strategies in each branch to complement each other. We also introduce a hierarchical alignment strategy for local feature alignment and a distribution alignment strategy for global feature alignment. Experiments on the PointDA-10 and PointSegDA datasets show that our approach achieves state-of-the-art performance on the UDA of point cloud classification and segmentation tasks. The ablation study demonstrates the effectiveness of the dual-branch design and the feature alignment strategies. [ABSTRACT FROM AUTHOR]

The pose estimation is the critical technology in industrial robot. Nowadays, many machine vision-based approaches have applied the technology and achieved excellent results. However, the rapid detection of the pose estimation in complex multiscene environments is still a challenge, due to the interference of multiangle light and multibackground. To address these issues, this article proposes a practical robotic grasping method by using the 6-D pose estimation with protective correction. In this method, the synthetic dataset by self-production is used to train the improved deep object pose estimation network and then use the standard perspective-n-point algorithm to estimate the 6-DoF pose for each object instance. Meanwhile, in order to prevent grasp collisions cause by misrecognition, we propose the corrected grasping pose algorithm for protective correction by measured translation and predicted translation. Finally, the proposed grasping method has an average grasping success rate of 83.3% for the three objects under normal light, and the network for single-image detection speed has been to 1.490 frames/s. The code is available at https://github.com/aimiplus/Practical_Robotic_Grasping_Method. [ABSTRACT FROM AUTHOR]

Convolutional neural network (CNN), one of the branches of deep neural networks, has been widely used in image recognition, natural language processing, and other related fields with great success recently. This paper proposes a novel framework with CNN to classify objects in a point cloud captured by LiDAR on urban streets. The proposed BA-CNN algorithm is composed of five steps: (i) removing ground points, (ii) clustering objects, (iii) transforming to bearing angle images, (iv) ROI selection, and (V) identifying objects by CNN. In the first step, ground points are removed by the multi-threshold-based ground detection to reduce the processing time. Then, a flood-fill-based clustering method is used for object segmentation. Those individual point cloud objects are converted to bearing angle (BA) images. Then, a well-trained CNN is used to classify objects with BA images. The main contribution of this paper is proposing an efficient recognition method that uses the information from point clouds only. In contrast, because most 3D object classifiers use the fusion of point clouds and color images, their models are very complicated and take a colossal amount of memory to store the parameters. Since the ground point detection and object clustering process all points along with the scanline-major order and layer-major order, the proposed algorithm performs better in terms of time consumption and memory consumption. In the experiment, three scenes from KITTI dataset are used for training and testing the proposed BA-CNN classifier, and the proposed BA-CNN achieves high classification accuracy. [ABSTRACT FROM AUTHOR]

Light detection and ranging (LIDAR) scanning is a common method of substation scene modeling that extracts point clouds of electrical equipment from the point cloud scene of a substation. The extraction effect is limited by uncertainty regarding the noise level, nonuniform point cloud density, and the computational complexity. In this paper, we propose a point cloud extraction solution for electrical equipment models. First, a statistical analysis of substation ground elevation is performed to obtain the point clouds of devices at the feature height and remove large numbers of redundant underground point clouds. Second, based on the statistically derived power equipment feature heights, the point cloud data are sliced according to the featured elevation intervals. Based on voxelization, the point cloud slices are then clustered using horizontal hierarchical clustering. The clustering results at different elevation intervals are then reclustered using vertical hierarchical clustering. Finally, we use filters combined with the DBSCAN algorithm to perform fine segmentation on the point cloud data. The results show that our slice clustering approach reduces the computational burden involved in point cloud processing, and the comprehensive clustering strategy ensures the accuracy of the clustering results.

Most of the existing simultaneous localization and mapping (SLAM) methods are based on the static environment assumption. The presence of moving objects in the scene will lead to much uncertainty in SLAM results, which also hinders the loop-closure detection (LCD). Although moving object tracking (MOT) is necessary for planning and decisions, it is often accomplished separately. To jointly solve SLAM and MOT for complex urban driving scenarios, this paper presents a high performance method named 4D-SLAM based on the fusion of LiDAR and IMU. The integration of SLAM and MOT is formulated as a joint posterior probability problem based on a dynamic Bayesian network (DBN) and is implemented with the following four sequential stages: preprocessing, moving object detection and tracking, odometry estimation and mapping. In the preprocessing stage, the motion distortion uncertainty caused by LiDAR scanning is first compensated for, and the initial LiDAR motion is estimated. In the moving object detection and tracking stage, we exploit a CNN-based segmentation network to detect the potential moving objects first, then the states of the potential moving objects are optimized by an unscented Kalman filter (UKF). In the odometry estimation stage, the distinctive planar and edge features extracted from a static background point cloud are used for the odometry estimation, and a two-step Levenberg-Marquardt optimization method is adopted to solve the 6DOF pose across consecutive scans. In the mapping stage, the mapping based on the pose estimation result and LCD are realized, and graph-based global optimization is exploited to further improve the map consistency for large scale environment. The comprehensive experiments with the open source dataset KITTI and the data collected by us show that the presented method not only outperforms the SOTA SLAM methods in terms of trajectory and mapping accuracy but can also detect and track moving objects efficiently.

6D target object detection is of great importance to many applications such as robotics, industrial automation, and unmanned vehicles and is increasingly influencing broad industries including manufacturing, transportation, and retail industries, to name a few. This paper focuses on detecting the 6D poses of the target objects from the point cloud of a cluttered scene. However, conventional point cloud-based 6D object detection methods rely on the robustness of key-point detection results that are not straightforward for humans to understand. The drawback makes conventional point cloud-based methods require expert knowledge to tune. In this paper, we introduced a 6D target object detection method that uses segmented object point cloud patches instead of key points to predict object 6D poses and identity. Our main contributions are an end-to-end data-driven pose correction model that is enhanced with a novel simple yet efficient basis spanning layer booster. Experiments show that although the proposed model is trained only using object CAD models, its 6D detection performance matches that of the models using view data. Thus, the proposed method is suitable for 6D detection applications that have object CAD models instead of labeled scene data.

In recent years, with the rapid development of multisensor fusion technology, point clouds used are no longer limited to those including 3-D coordinates acquired by visual sensors, such as binocular sensors or structured light sensors. The 4-D or more multidimensional data are needed to analyze information in a more intuitive way. The 3-D point cloud and 2-D image have complementary information, and the point cloud can be colored by the fusion of coordinate data and intensity data. However, due to the limitation of sight occlusion, only some points in the point cloud are visible in a single image, i.e., they have intensity information. Most existing methods rely on surface reconstruction, which has always been a complex problem in theory and implementation. In this article, a new algorithm named region growing based on 2-D–3-D mutual projections is proposed. Based on the idea of regional growing, we select the initial seed points by the geometric information of the point cloud in the 3-D space and projection plane, and then estimate the visibility of each point according to the growth criteria defined by us. The results show that the proposed method successfully divides the visible points and the occlusion points and achieves satisfactory results in the subsequent intensity fusion. Our method is more robust than the large curvature change and large density change regions. For the car model, the false negative rate of our algorithm decreases by 4.4% compared with Katz’s method, and the score of our algorithm is 15.4% higher than that of Biasutti et al.’s method. [ABSTRACT FROM AUTHOR]

This article proposes an automatic high-precision detection method for structure parameters of catenary cantilever devices (SPCCDs) using 3-D point cloud data. The steps of the proposed detection method are: 1) segmenting and recognizing the components of the catenary cantilever devices, 2) extracting the detection plane and backbone component axis of catenary cantilever devices, and 3) detecting the SPCCD. The effective segmentation of components is critical for structure parameter detection. A point cloud segmentation and recognition method based on three-dimensional convolutional neural networks (3-D CNNs) is introduced to determine the different components of the catenary cantilever devices. Compared with traditional unsupervised clustering procedures for point cloud segmentation, the proposed method can improve the segmentation accuracy, does not require complex tuning procedures of parameters, and improves robustness and stability. Additionally, the segmentation method defines a recognition function, facilitating the analysis of the structural relationship between objects. Furthermore, we proposed an improved projection random sample consensus (RANSAC) method, which can effectively divide the detection plane of catenary cantilever devices to solve the multicantilever device occlusion problem. With RANSAC, it is also possible to precisely extract the backbone component axis and enhance parameter detection accuracy. The experimental results show that the structure angle and steady arm slope’s error accuracy can achieve 0.1029° and 1.19%, respectively, which indicates the proposed approach can precisely detect the SPCCD. [ABSTRACT FROM AUTHOR]

This paper proposes a novel algorithm for extracting street light poles from vehicleborne mobile light detection and ranging (LiDAR) point-clouds. First, the algorithm rapidly detects curb-lines and segments a point-cloud into road and nonroad surface points based on trajectory data recorded by the integrated position and orientation system onboard the vehicle. Second, the algorithm accurately extracts street light poles from the segmented nonroad surface points using a novel pairwise 3-D shape context. The proposed algorithm is tested on a set of point-clouds acquired by a RIEGL VMX-450 mobile LiDAR system. The results show that road surfaces are correctly segmented, and street light poles are robustly extracted with a completeness exceeding 99%, a correctness exceeding 97%, and a quality exceeding 96%, thereby demonstrating the efficiency and feasibility of the proposed algorithm to segment road surfaces and extract street light poles from huge volumes of mobile LiDAR point-clouds. [ABSTRACT FROM PUBLISHER]

Accurate and timely updated three-dimensional (3-D) model data of interchange bridges are significant for traffic navigation, landscape design, and many other applications. In this study, we explore the potential of using airborne light detection and ranging (LiDAR) data for 3-D reconstruction of large multilayer interchange bridges. To reduce the technical difficulty in this 3-D reconstruction process, we propose a concept of “structure unit.” We propose a new technical framework based on the structure units for 3-D reconstruction of large multilayer interchange bridge, including point cloud extraction, connectivity-based segmentation, determination of structure units, occlusion detection and restoration, and 3-D modeling. The general idea of this framework is to divide an entire interchange bridge into many simple “structure units,” and then derive a complete 3-D model by reconstructing each “structure unit.” The technical novelties in this framework include: 1) a hierarchal segmentation strategy including connectivity-based segmentation and determination of structure units; and 2) an automatic technique for detection and restoration of occluded structures. The experiments provide specific analysis and evaluation from three aspects: 1) correctness and completeness; 2) horizontal accuracy; and 3) elevation accuracy. The experimental results indicate that the proposed framework can provide satisfactory 3-D models of large complex bridges. [ABSTRACT FROM PUBLISHER]