Search

For the automated robotic picking of bunch-type fruit, the strategy is to roughly determine the location of the bunches, plan the picking route from a remote location, and then locate the picking point precisely at a more appropriate, closer location. The latter can reduce the amount of information to be processed and obtain more precise and detailed features, thus improving the accuracy of the vision system. In this study, a long-close distance coordination control strategy for a litchi picking robot was proposed based on an Intel Realsense D435i camera combined with a point cloud map collected by the camera. The YOLOv5 object detection network and DBSCAN point cloud clustering method were used to determine the location of bunch fruits at a long distance to then deduce the sequence of picking. After reaching the close-distance position, the Mask RCNN instance segmentation method was used to segment the more distinctive bifurcate stems in the field of view. By processing segmentation masks, a dual reference model of "Point + Line" was proposed, which guided picking by the robotic arm. Compared with existing studies, this strategy took into account the advantages and disadvantages of depth cameras. By experimenting with the complete process, the density-clustering approach in long distance was able to classify different bunches at a closer distance, while a success rate of 88.46% was achieved during fruit-bearing branch locating. This was an exploratory work that provided a theoretical and technical reference for future research on fruit-picking robots. [ABSTRACT FROM AUTHOR]

A four-ocular vision system is proposed for the three-dimensional (3D) reconstruction of large-scale concrete-filled steel tube (CFST) under complex testing conditions. These measurements are vitally important for evaluating the seismic performance and 3D deformation of large-scale specimens. A four-ocular vision system is constructed to sample the large-scale CFST; then point cloud acquisition, point cloud filtering, and point cloud stitching algorithms are applied to obtain a 3D point cloud of the specimen surface. A point cloud correction algorithm based on geometric features and a deep learning algorithm are utilized, respectively, to correct the coordinates of the stitched point cloud. This enhances the vision measurement accuracy in complex environments and therefore yields a higher-accuracy 3D model for the purposes of real-time complex surface monitoring. The performance indicators of the two algorithms are evaluated on actual tasks. The cross-sectional diameters at specific heights in the reconstructed models are calculated and compared against laser rangefinder data to test the performance of the proposed algorithms. A visual tracking test on a CFST under cyclic loading shows that the reconstructed output well reflects the complex 3D surface after correction and meets the requirements for dynamic monitoring. The proposed methodology is applicable to complex environments featuring dynamic movement, mechanical vibration, and continuously changing features. [ABSTRACT FROM AUTHOR]

A novel detection algorithm based on color, depth, and shape information is proposed for detecting spherical or cylindrical fruits on plants in natural environments and thus guiding harvesting robots to pick them automatically. A probabilistic image segmentation method is first presented to segment a red–green–blue image as a binary mask. Multiplied by this mask, a filtered depth image is obtained. Region growing, a region-based image segmentation method, is then applied to group the depth image into multiple clusters. Each cluster represents a fruit, leaf, or branch that is later transformed into a point cloud. Next, a 3D shape detection method based on M-estimator sample consensus, a model parameter estimator, is employed to detect potential fruits from each point cloud. Finally, an angle/color/shape-based global point cloud descriptor (GPCD) is developed to extract a feature vector for an entire point cloud, and a support vector machine classifier trained on the GPCD features is used to exclude false positives. Pepper, eggplant, and guava datasets were captured in the field. For the pepper, eggplant, and guava datasets, the detection precision was 0.864, 0.886, and 0.888, and the recall was 0.889, 0.762, and 0.812, respectively. Experiments revealed that the proposed algorithm was universal and robust and hence applicable to an agricultural harvesting robot. [ABSTRACT FROM AUTHOR]

• Multi-vision system is powerful, but unavoidable errors emerge in global calibration. • Global calibration error can be corrected by registration of the common parts. • The features of the targets are more adaptive than those of the standard objects. • The corrected models reach higher accuracy than that without correction. • The mean absolute error is 1.06 mm, and the mean relative error is 0.52% respectively. Multi-camera schemes can effectively increase the perception range of vision systems compared to single-camera schemes and are common in many optical applications. Unavoidable errors emerge in the global multi-camera calibration process, however, such as manufacturing error of the optical devices and computational error from marker detection algorithms, which drive down the accuracy of the camera system correlation. This paper discusses the causes of global calibration errors in detail. A four-camera vision system was built to obtain the visual information of targets including static objects and a dynamic concrete-filled steel tubular (CFST) specimen. Local calibration and global calibration were applied successively to realize multi-camera correlation, followed by filtering and stitching operations to acquire filtered global point clouds. A point cloud correction algorithm is designed accordingly to optimize the stitched point cloud structures and further improve the accuracy of the reconstructed surfaces. Based on the density features of the targets themselves (rather than standard calibration markers), the proposed point cloud correction algorithm is effective for various targets and adaptive under dynamic conditions. The point clouds and corresponding reconstructed models are shown to be more accurate after the proposed enhancement process. The point cloud correction algorithm also has strong adaptability to different static targets with complex surfaces and performs well under uncertain geometric changes and vibration. The results presented here provide both theoretical and practical support for advancements in multi-vision applications such as optical measurement, real-time target tracking, quality monitoring, and surface data acquisition. [ABSTRACT FROM AUTHOR]

In unstructured environments, harvesting robots may collide with disorderly growing branches, thus reducing the success rate of harvesting. This study introduces a fruit and branch detection and three-dimensional (3D) reconstruction method for obstacle avoidance path planning of robots. A new architecture for instance segmentation was developed by replacing the backbone of Mask R-CNN with a tiny network, referred to as "tiny Mask R-CNN". The tiny Mask R-CNN was trained with a small number of images and used to detect guava fruits and branches. Each detected fruit and branch were converted into a 3D point cloud. It was then hypothesized that guava fruits could be represented by 3D spheres and irregular branches can be approximated by a finite number of 3D cylindrical segments. Based on the proposed hypothesis, a random sample consensus-based sphere fitting method and a principal component analysis-based cylindrical segment fitting method were investigated to reconstruct the fruits and branches from the point clouds. A guava dataset with 304 RGB-D images was collected from the fields and used to validate the developed method. The results showed that the detection F1 score of the tiny Mask R-CNN was 0.518; the F1 score for fruit reconstruction was approximately 0.851 and 0.833 under the 2D- and 3D-fruit metrics, respectively; and the F1 score for branch reconstruction was approximately 0.394 and 0.415 under the 2D- and 3D-branch metrics, respectively. These results confirm that the proposed method can effectively reconstruct the fruits and branches and can, therefore, be used to plan an obstacle avoidance path for harvesting robots. [ABSTRACT FROM AUTHOR]

• The multi-vision system is powerful, but interferences always emerge in field. • A multi-geometry based framework is designed to ensure stable visual measurement. • The proposed adaptive stereo matching is much more efficient and reliable than the classic one. • The proposed fine stitching algorithm greatly compensates the unavoidable global calibration error. • This study provides reference for applying multi-vision system in complex agricultural picking tasks. Automatic vision-based picking in orchards and fields is a highly challenging task. The orchard banana central stock, which is large in size, low in color contrast, and falls within a complex background, was taken as the subject in this research. A measurement framework based on multi-vision technology was established, and a set of general methods were utilized to improve the comprehensive performance of multi-view-geometry-based vision modules in orchard picking tasks. Multiple cameras at different angles were deployed to maximize the perception range. The global geometric parameters of the cameras were calibrated and a robust semantic segmentation network was trained to achieve effective image pre-processing. A novel adaptive stereo matching strategy was designed to ensure that the robot reliably completes 3D triangulation at various depths as it moves across the target area. Global calibration errors were corrected via a high-accuracy point cloud stitching algorithm. Experimental results indicated that the proposed adaptive stereo matching strategy was accurate to different sampling depths and showed stable performance, and the proposed point cloud stitching algorithm accurately stitched multi-view point clouds. This work provides theoretical and practical references for the 3D sensing of banana central stocks in complex environments. The proposed technique was designed for adaptability of the multi-vision system for field perception, so it can be easily transferred to similar applications such as the 3D reconstruction of agricultural targets, 3D positioning of fruit clusters, and 3D robotic arm obstacle avoidance. [ABSTRACT FROM AUTHOR]