Search

The demand for intelligent agriculture is increasing due to the continuous impact of world food and environmental crises. Focusing on fruit detection, with the rapid development of object detection technology, it is now possible to achieve high efficiency and high accuracy in fruit detection systems. However, detecting fruit with high precision in unstructured orchard environments remains particularly challenging. Such environments, which are composed of varying illumination conditions and degrees of occlusion, can be mitigated by certain strategies. To our knowledge, this is the first time that optimization strategies used in fruit detection have been reviewed. This review aims to explore methods for improving fruit detection in complex environments. First, we describe the common types of complex backgrounds found in outdoor orchard environments. Subsequently, we divide the improvement measures into two categories: optimization before and after image sampling. Next, we compare the test results obtained before and after the application of these improved methods. Finally, we describe the future development trends of fruit detection optimization technology in complex backgrounds. We hope that this review will inspire researchers to design their optimization strategies and help explore lower-cost and more robust fruit detection systems. [ABSTRACT FROM AUTHOR]

Accurate and reliable fruit detection in the orchard environment is an important step for yield estimation and robotic harvesting. However, the existing detection methods often target large and relatively sparse fruits, but they cannot provide a good solution for small and densely distributed fruits. This paper proposes a YOLOv3-Litchi model based on YOLOv3 to detect densely distributed litchi fruits in large visual scenes. We adjusted the prediction scale and reduced the network layer to improve the detection ability of small and dense litchi fruits and ensure the detection speed. From flowering to 50 days after maturity, we collected a total of 266 images, including 16,000 fruits, and then used them to construct the litchi dataset. Then, the k-means++ algorithm is used to cluster the bounding boxes in the labeled data to determine the priori box size suitable for litchi detection. We trained an improved YOLOv3-Litchi model, tested its litchi detection performance, and compared YOLOv3-Litchi with YOLOv2, YOLOv3, and Faster R-CNN on the actual detection effect of litchi and used the F1 value and the average detection time as the assessed value. The test results show that the F1 of YOLOv3-Litchi is higher than that of YOLOv2 algorithm 0.1, higher than that of YOLOv3 algorithm 0.08, and higher than that of Faster R-CNN algorithm 0.05; the average detection time of YOLOv3-Litchi is 29.44 ms faster than that of YOLOv2 algorithm, 19.56 ms faster than that of YOLOv3 algorithm ms, and 607.06 ms faster than that of Faster R-CNN algorithm. And the detection speed of the improved model is faster. The proposed model remits optimal detection performance for small and dense fruits. The work presented here may provide a reference for further study on fruit-detection methods in natural environments. [ABSTRACT FROM AUTHOR]

In-field citrus detection and localisation are highly challenging tasks due to varying illumination conditions, partial occlusion of citrus, and the colour variation of citrus at different stages of maturity. A reliable algorithm based on red-green-blue-depth (RGB-D) images was developed to detect and locate citrus in real, outdoor orchard environments for robotic harvesting. A depth filter and a Bayes-classifier-based image segmentation method were first developed to exclude as many backgrounds as possible. A density clustering method was then used to group adjacent points in the filtered RGB-D images into clusters, where each cluster represents a possible citrus. A colour, gradient, and geometry feature-based support vector machine classifier was trained to remove false positives. To test the method, a dataset with 506 RGB-D images was acquired in a citrus orchard on sunny and cloudy days. Results showed that the proposed algorithm was robust with an F1 score of 0.9197; the positioning errors in the x , y and z directions were 7.0 ± 2.5 mm, −4.0 ± 3.0 mm and 13.0 ± 3.0 mm, respectively, and the sizing error was −1.0 ± 4.0 mm. These excellent performance values demonstrate that the proposed method could be used to guide a citrus-harvesting robot. • An algorithm for citrus detection, localisation and sizing was developed. • The algorithm can detect citrus robustly with an F1 score of 0.9197. • Positioning errors in x , y , z were 7.0 ± 2.5 mm, −4.0 ± 3.0 mm and 13.0 ± 3.0 mm. • The sizing error was −1.0 ± 4.0 mm. [ABSTRACT FROM AUTHOR]

• A new form of orchard mapping framework integrating eye-in-hand stereovision and SLAM is proposed. • Large-scale, high-accuracy, and detailed global maps supporting high-quality orchard picking are obtained. • The proposed hand-eye calibration method is efficient and beats the compared methods. • The proposed stereo matching method is highly adapted to dynamic and complex orchard environment. • The framework generates a more detailed global map than the commercial products used for comparison. Large-scale, high-accuracy, and adaptive three-dimensional (3D) perception are the basic technical requirements for constructing a practical and stable fruit-picking robot. The latest vision-based fruit-picking robots have been able to adapt to the complex background, uneven lighting and low color contrast of the orchard environment. However, most of them have, until now, been limited to a small field of view or rigid sampling manners. Although the simultaneous localization and mapping (SLAM) methods have the potential to realize large scale sensing, it was critically revealed in this study that the classic SLAM pipeline is not completely adapted to orchard picking tasks. In this study, the eye-in-hand stereo vision and SLAM system were integrated to provide detailed global map supporting long-term, flexible and large-scale orchard picking. To be specific, a mobile robot based on eye-in-hand vision was built and an effective hand-eye calibration method was proposed; a state-of-the-art object detection network was trained and used to establish a dynamic stereo matching method adapted well to complex orchard environments; a SLAM system was deployed and combined with the above eye-in-hand stereo vision system to obtain a detailed, wide 3D orchard map. The main contribution of this work is to build a new global mapping framework compatible to the nature of orchard picking tasks. Compared with the existing studies, this work pays more attention to the structural details of the orchard. Experimental results indicated that the constructed global map achieved both large-scale and high-resolution. This is an exploratory work providing theoretical and technical references for the future research on more stable, accurate and practical mobile fruit picking robots. [ABSTRACT FROM AUTHOR]

• The multi-vision system is powerful, but interferences always emerge in field. • A multi-geometry based framework is designed to ensure stable visual measurement. • The proposed adaptive stereo matching is much more efficient and reliable than the classic one. • The proposed fine stitching algorithm greatly compensates the unavoidable global calibration error. • This study provides reference for applying multi-vision system in complex agricultural picking tasks. Automatic vision-based picking in orchards and fields is a highly challenging task. The orchard banana central stock, which is large in size, low in color contrast, and falls within a complex background, was taken as the subject in this research. A measurement framework based on multi-vision technology was established, and a set of general methods were utilized to improve the comprehensive performance of multi-view-geometry-based vision modules in orchard picking tasks. Multiple cameras at different angles were deployed to maximize the perception range. The global geometric parameters of the cameras were calibrated and a robust semantic segmentation network was trained to achieve effective image pre-processing. A novel adaptive stereo matching strategy was designed to ensure that the robot reliably completes 3D triangulation at various depths as it moves across the target area. Global calibration errors were corrected via a high-accuracy point cloud stitching algorithm. Experimental results indicated that the proposed adaptive stereo matching strategy was accurate to different sampling depths and showed stable performance, and the proposed point cloud stitching algorithm accurately stitched multi-view point clouds. This work provides theoretical and practical references for the 3D sensing of banana central stocks in complex environments. The proposed technique was designed for adaptability of the multi-vision system for field perception, so it can be easily transferred to similar applications such as the 3D reconstruction of agricultural targets, 3D positioning of fruit clusters, and 3D robotic arm obstacle avoidance. [ABSTRACT FROM AUTHOR]

• Picking robots must operate continuously throughout the orchard for commercial and practical purposes. • Efficient visual servo control methods of locomotion and picking are necessary for full-field harvesting. • Serialized keyframes from SLAM provide high-confidence global crop information. • The multi-map fusion mechanism greatly improves locomotion stability, despite computational costs. • Solely relying on multi-view positioning does not enhance picking accuracy but affects picking efficiency. Fruit-picking robots are crucial for achieving efficient orchard harvesting. To genuinely meet the commercial production needs of farmers, the new generation of fruit-picking robots must be capable of demonstrating complete and continuous observation, movement, and picking behaviors throughout complex orchards, akin to real human employees. This poses systematic challenges, as many prior researches have focused solely on a part of the continuous operation of the entire orchard, such as fruit positioning, navigation, path planning, or grasping. These isolated basic functions are important but insufficient for fulfilling operational requirements on a macro scale and continuous situation. Developing an efficient control method for each basic module and constructing their internal coordination is vital for transitioning a harvesting robot from a functional prototype to a practical machine. In this context, this study tackles the visual servo control problem for efficient locomotion, picking, and their seamless integration. A set of vision algorithms for locomotion destination estimation, real-time self-positioning, and dynamic harvesting is proposed. Additionally, a solid coordination mechanism for continuous locomotion and picking behavior is established. Each method offers distinct advantages, such as improved accuracy, adaptability to varying conditions, and enhanced picking efficiency, enabling the robot to operate autonomously and continuously. Comprehensive field experiments validated the soundness of the methods. The primary contribution of this study lies in addressing the challenge of continuous operation in an entire orchard as a systematic problem and providing new insights into control methods for the future development of highly autonomous, practical, and user-oriented fruit harvesting systems. [ABSTRACT FROM AUTHOR]