Search

• A new form of orchard mapping framework integrating eye-in-hand stereovision and SLAM is proposed. • Large-scale, high-accuracy, and detailed global maps supporting high-quality orchard picking are obtained. • The proposed hand-eye calibration method is efficient and beats the compared methods. • The proposed stereo matching method is highly adapted to dynamic and complex orchard environment. • The framework generates a more detailed global map than the commercial products used for comparison. Large-scale, high-accuracy, and adaptive three-dimensional (3D) perception are the basic technical requirements for constructing a practical and stable fruit-picking robot. The latest vision-based fruit-picking robots have been able to adapt to the complex background, uneven lighting and low color contrast of the orchard environment. However, most of them have, until now, been limited to a small field of view or rigid sampling manners. Although the simultaneous localization and mapping (SLAM) methods have the potential to realize large scale sensing, it was critically revealed in this study that the classic SLAM pipeline is not completely adapted to orchard picking tasks. In this study, the eye-in-hand stereo vision and SLAM system were integrated to provide detailed global map supporting long-term, flexible and large-scale orchard picking. To be specific, a mobile robot based on eye-in-hand vision was built and an effective hand-eye calibration method was proposed; a state-of-the-art object detection network was trained and used to establish a dynamic stereo matching method adapted well to complex orchard environments; a SLAM system was deployed and combined with the above eye-in-hand stereo vision system to obtain a detailed, wide 3D orchard map. The main contribution of this work is to build a new global mapping framework compatible to the nature of orchard picking tasks. Compared with the existing studies, this work pays more attention to the structural details of the orchard. Experimental results indicated that the constructed global map achieved both large-scale and high-resolution. This is an exploratory work providing theoretical and technical references for the future research on more stable, accurate and practical mobile fruit picking robots. [ABSTRACT FROM AUTHOR]

In unstructured environments, harvesting robots may collide with disorderly growing branches, thus reducing the success rate of harvesting. This study introduces a fruit and branch detection and three-dimensional (3D) reconstruction method for obstacle avoidance path planning of robots. A new architecture for instance segmentation was developed by replacing the backbone of Mask R-CNN with a tiny network, referred to as "tiny Mask R-CNN". The tiny Mask R-CNN was trained with a small number of images and used to detect guava fruits and branches. Each detected fruit and branch were converted into a 3D point cloud. It was then hypothesized that guava fruits could be represented by 3D spheres and irregular branches can be approximated by a finite number of 3D cylindrical segments. Based on the proposed hypothesis, a random sample consensus-based sphere fitting method and a principal component analysis-based cylindrical segment fitting method were investigated to reconstruct the fruits and branches from the point clouds. A guava dataset with 304 RGB-D images was collected from the fields and used to validate the developed method. The results showed that the detection F1 score of the tiny Mask R-CNN was 0.518; the F1 score for fruit reconstruction was approximately 0.851 and 0.833 under the 2D- and 3D-fruit metrics, respectively; and the F1 score for branch reconstruction was approximately 0.394 and 0.415 under the 2D- and 3D-branch metrics, respectively. These results confirm that the proposed method can effectively reconstruct the fruits and branches and can, therefore, be used to plan an obstacle avoidance path for harvesting robots. [ABSTRACT FROM AUTHOR]

• The multi-vision system is powerful, but interferences always emerge in field. • A multi-geometry based framework is designed to ensure stable visual measurement. • The proposed adaptive stereo matching is much more efficient and reliable than the classic one. • The proposed fine stitching algorithm greatly compensates the unavoidable global calibration error. • This study provides reference for applying multi-vision system in complex agricultural picking tasks. Automatic vision-based picking in orchards and fields is a highly challenging task. The orchard banana central stock, which is large in size, low in color contrast, and falls within a complex background, was taken as the subject in this research. A measurement framework based on multi-vision technology was established, and a set of general methods were utilized to improve the comprehensive performance of multi-view-geometry-based vision modules in orchard picking tasks. Multiple cameras at different angles were deployed to maximize the perception range. The global geometric parameters of the cameras were calibrated and a robust semantic segmentation network was trained to achieve effective image pre-processing. A novel adaptive stereo matching strategy was designed to ensure that the robot reliably completes 3D triangulation at various depths as it moves across the target area. Global calibration errors were corrected via a high-accuracy point cloud stitching algorithm. Experimental results indicated that the proposed adaptive stereo matching strategy was accurate to different sampling depths and showed stable performance, and the proposed point cloud stitching algorithm accurately stitched multi-view point clouds. This work provides theoretical and practical references for the 3D sensing of banana central stocks in complex environments. The proposed technique was designed for adaptability of the multi-vision system for field perception, so it can be easily transferred to similar applications such as the 3D reconstruction of agricultural targets, 3D positioning of fruit clusters, and 3D robotic arm obstacle avoidance. [ABSTRACT FROM AUTHOR]

• Picking robots must operate continuously throughout the orchard for commercial and practical purposes. • Efficient visual servo control methods of locomotion and picking are necessary for full-field harvesting. • Serialized keyframes from SLAM provide high-confidence global crop information. • The multi-map fusion mechanism greatly improves locomotion stability, despite computational costs. • Solely relying on multi-view positioning does not enhance picking accuracy but affects picking efficiency. Fruit-picking robots are crucial for achieving efficient orchard harvesting. To genuinely meet the commercial production needs of farmers, the new generation of fruit-picking robots must be capable of demonstrating complete and continuous observation, movement, and picking behaviors throughout complex orchards, akin to real human employees. This poses systematic challenges, as many prior researches have focused solely on a part of the continuous operation of the entire orchard, such as fruit positioning, navigation, path planning, or grasping. These isolated basic functions are important but insufficient for fulfilling operational requirements on a macro scale and continuous situation. Developing an efficient control method for each basic module and constructing their internal coordination is vital for transitioning a harvesting robot from a functional prototype to a practical machine. In this context, this study tackles the visual servo control problem for efficient locomotion, picking, and their seamless integration. A set of vision algorithms for locomotion destination estimation, real-time self-positioning, and dynamic harvesting is proposed. Additionally, a solid coordination mechanism for continuous locomotion and picking behavior is established. Each method offers distinct advantages, such as improved accuracy, adaptability to varying conditions, and enhanced picking efficiency, enabling the robot to operate autonomously and continuously. Comprehensive field experiments validated the soundness of the methods. The primary contribution of this study lies in addressing the challenge of continuous operation in an entire orchard as a systematic problem and providing new insights into control methods for the future development of highly autonomous, practical, and user-oriented fruit harvesting systems. [ABSTRACT FROM AUTHOR]

Automated pineapple harvesting has emerged as a prominent prospective development within the agricultural domain. Nevertheless, the intricate growth conditions that pineapples encounter in the field, such as inadequate light, overexposure, obstructions caused by fruit leaves, or the overlapping of fruits, pose substantial challenges to the accuracy and robustness of traditional real-time detection algorithms. In recent times, Transformer models, when applied to computer vision, have exhibited commendable performance, underscoring their potential for target detection in smart agricultural applications. In this report, we propose a spatio-temporal convolutional neural network model that leverages the shifted window Transformer fusion region convolutional neural network model for the purpose of detecting pineapple fruits. Our study includes a comparative analysis of these results and those obtained through the utilization of conventional models. Additionally, we investigate the influence of various aspects of data preparation, including image resolution, object size, and object complexity, on the ultimate pineapple detection outcomes. Experimental findings elucidate that, in the case of detecting a single-category target like a pineapple, the employment of 2000 annotated supervised data points yields the optimal detection accuracy. Further augmenting the size of the training dataset does not yield any significant improvement in detection accuracy. Furthermore, images of pineapples captured from greater distances encompass smaller targets and an increased number of pineapple instances, rendering them more intricate and challenging to accurately detect. In summary, our study employs the spatio-temporal convolutional neural network model to attain pineapple detection with an impressive accuracy rate of 92.54% and an average inference time of 0.163 s, thus affirming the efficacy of our developed model in achieving superior detection results. [ABSTRACT FROM AUTHOR]

• An image process pipeline is introduced to locate guava fruits and obstacles. • A recurrent deep reinforcement learning algorithm is proposed to predict collision-free paths. • The robot is trained in a simulation environment and transferred to the real world directly. • The robot only needs 29 ms to plan a collision-free path with a success rate of 90.9%. In unstructured orchard environments, picking a target fruit without colliding with neighboring branches is a significant challenge for guava-harvesting robots. This paper introduces a fast and robust collision-free path-planning method based on deep reinforcement learning. A recurrent neural network is first adopted to remember and exploit the past states observed by the robot, then a deep deterministic policy gradient algorithm (DDPG) predicts a collision-free path from the states. A simulation environment is developed and its parameters are randomized during the training phase to enable recurrent DDPG to generalize to real-world scenarios. We also introduce an image processing method that uses a deep neural network to detect obstacles and uses many three-dimensional line segments to approximate the obstacles. Simulations show that recurrent DDPG only needs 29 ms to plan a collision-free path with a success rate of 90.90%. Field tests show that recurrent DDPG can increase grasp, detachment, and harvest success rates by 19.43%, 9.11%, and 10.97%, respectively, compared to cases where no collision-free path-planning algorithm is implemented. Recurrent DDPG strikes a strong balance between efficiency and robustness and may be suitable for other fruits. [ABSTRACT FROM AUTHOR]