Search

The demand for intelligent agriculture is increasing due to the continuous impact of world food and environmental crises. Focusing on fruit detection, with the rapid development of object detection technology, it is now possible to achieve high efficiency and high accuracy in fruit detection systems. However, detecting fruit with high precision in unstructured orchard environments remains particularly challenging. Such environments, which are composed of varying illumination conditions and degrees of occlusion, can be mitigated by certain strategies. To our knowledge, this is the first time that optimization strategies used in fruit detection have been reviewed. This review aims to explore methods for improving fruit detection in complex environments. First, we describe the common types of complex backgrounds found in outdoor orchard environments. Subsequently, we divide the improvement measures into two categories: optimization before and after image sampling. Next, we compare the test results obtained before and after the application of these improved methods. Finally, we describe the future development trends of fruit detection optimization technology in complex backgrounds. We hope that this review will inspire researchers to design their optimization strategies and help explore lower-cost and more robust fruit detection systems. [ABSTRACT FROM AUTHOR]

Accurate and reliable fruit detection in the orchard environment is an important step for yield estimation and robotic harvesting. However, the existing detection methods often target large and relatively sparse fruits, but they cannot provide a good solution for small and densely distributed fruits. This paper proposes a YOLOv3-Litchi model based on YOLOv3 to detect densely distributed litchi fruits in large visual scenes. We adjusted the prediction scale and reduced the network layer to improve the detection ability of small and dense litchi fruits and ensure the detection speed. From flowering to 50 days after maturity, we collected a total of 266 images, including 16,000 fruits, and then used them to construct the litchi dataset. Then, the k-means++ algorithm is used to cluster the bounding boxes in the labeled data to determine the priori box size suitable for litchi detection. We trained an improved YOLOv3-Litchi model, tested its litchi detection performance, and compared YOLOv3-Litchi with YOLOv2, YOLOv3, and Faster R-CNN on the actual detection effect of litchi and used the F1 value and the average detection time as the assessed value. The test results show that the F1 of YOLOv3-Litchi is higher than that of YOLOv2 algorithm 0.1, higher than that of YOLOv3 algorithm 0.08, and higher than that of Faster R-CNN algorithm 0.05; the average detection time of YOLOv3-Litchi is 29.44 ms faster than that of YOLOv2 algorithm, 19.56 ms faster than that of YOLOv3 algorithm ms, and 607.06 ms faster than that of Faster R-CNN algorithm. And the detection speed of the improved model is faster. The proposed model remits optimal detection performance for small and dense fruits. The work presented here may provide a reference for further study on fruit-detection methods in natural environments. [ABSTRACT FROM AUTHOR]

The utilization of machine vision and its associated algorithms improves the efficiency, functionality, intelligence, and remote interactivity of harvesting robots in complex agricultural environments. Machine vision and its associated emerging technology promise huge potential in advanced agricultural applications. However, machine vision and its precise positioning still have many technical difficulties, making it difficult for most harvesting robots to achieve true commercial applications. This article reports the application and research progress of harvesting robots and vision technology in fruit picking. The potential applications of vision and quantitative methods of localization, target recognition, 3D reconstruction, and fault tolerance of complex agricultural environment are focused, and fault-tolerant technology designed for utilization with machine vision and robotic systems are also explored. The two main methods used in fruit recognition and localization are reviewed, including digital image processing technology and deep learning-based algorithms. The future challenges brought about by recognition and localization success rates are identified: target recognition in the presence of illumination changes and occlusion environments; target tracking in dynamic interference-laden environments, 3D target reconstruction, and fault tolerance of the vision system for agricultural robots. In the end, several open research problems specific to recognition and localization applications for fruit harvesting robots are mentioned, and the latest development and future development trends of machine vision are described. [ABSTRACT FROM AUTHOR]

This paper proposes a novel technique for fruit detection in natural environments which is applicable in automatic harvesting robots, yield estimation systems and quality monitoring systems. As most color-based techniques are highly sensitive to illumination changes and low contrasts between fruits and leaves, the proposed technique, conversely, is based on contour information. Firstly, a discriminative shape descriptor is derived to represent geometrical properties of arbitrary fragment, and applied to a bidirectional partial shape matching to detect sub-fragments of interest that match parts of a reference contour. Then, a novel probabilistic Hough transform is developed to aggregate these sub-fragments for obtaining fruit candidates. Finally, all fruit candidates are verified by a support vector machine classifier trained on color and texture features. Citrus, tomato, pumpkin, bitter gourd, towel gourd and mango datasets were provided. Experiments on these datasets demonstrated that the proposed approach was competitive for detecting most type of fruits, such as green, orange, circular and non-circular, in natural environments. [ABSTRACT FROM AUTHOR]

A novel detection algorithm based on color, depth, and shape information is proposed for detecting spherical or cylindrical fruits on plants in natural environments and thus guiding harvesting robots to pick them automatically. A probabilistic image segmentation method is first presented to segment a red–green–blue image as a binary mask. Multiplied by this mask, a filtered depth image is obtained. Region growing, a region-based image segmentation method, is then applied to group the depth image into multiple clusters. Each cluster represents a fruit, leaf, or branch that is later transformed into a point cloud. Next, a 3D shape detection method based on M-estimator sample consensus, a model parameter estimator, is employed to detect potential fruits from each point cloud. Finally, an angle/color/shape-based global point cloud descriptor (GPCD) is developed to extract a feature vector for an entire point cloud, and a support vector machine classifier trained on the GPCD features is used to exclude false positives. Pepper, eggplant, and guava datasets were captured in the field. For the pepper, eggplant, and guava datasets, the detection precision was 0.864, 0.886, and 0.888, and the recall was 0.889, 0.762, and 0.812, respectively. Experiments revealed that the proposed algorithm was universal and robust and hence applicable to an agricultural harvesting robot. [ABSTRACT FROM AUTHOR]

• Object detection based on deep learning is applied to binocular location. • An improved Camellia oleifera fruit detection model is proposed. • The calculation amount of stereo matching is reduced. • It provides visual technical reference for field picking robots. In the complex environment of an orchard, changes in illumination, leaf occlusion, and fruit overlap make it challenging for mobile picking robots to detect and locate oil-seed camellia fruit. To address this problem, YOLO-Oleifera was developed as a fruit detection model method based on a YOLOv4-tiny model, To obtain clustering results appropriate to the size of the Camellia oleifera fruit, the k-means++ clustering algorithm was used instead of the k-means clustering algorithm used by the YOLOv4-tiny model to determine bounding box priors. Two convolutional kernels of 1 × 1 and 3 × 3 were respectively added after the second and third CSPBlock modules of the YOLOv4-tiny model. This model allows the learning of Camellia oleifera fruit feature information and reduces overall computational complexity. Compared with the classic stereo matching method based on binocular camera images, this method innovatively used the bounding box generated by the YOLO-Oleifera model to extract the region of interest of the fruit, and then adaptively performs stereo matching according to the generation mechanism of the bounding box. This allows the determination of disparity and facilitates the subsequent use of the triangulation principle to determine the picking position of the fruit. An ablation experiment demonstrated the effective improvement of the YOLOv4-tiny model. Camellia oleifera fruit images obtained under sunlight and shading conditions were used to test the YOLO-Oleifera model, and the model robustly detected the fruit under different illumination conditions. Occluded Camellia oleifera fruit decreased precision and recall due to the loss of semantic information. Comparison of this model with deep learning models YOLOv5-s,YOLOv3-tiny, and YOLOv4-tiny, the YOLO-Oleifera model achieved the highest AP of 0.9207 with the smallest data weight of 29 MB. The YOLO-Oleifera model took an average of 31 ms to detect each fruit image, fast enough to meet the demand for real-time detection. The algorithm exhibited high positioning stability and robust function despite changes in illumination. The results of this study can provide a technical reference for the robust detection and positioning of Camellia oleifera fruit by a mobile picking robot in a complex orchard environment. [ABSTRACT FROM AUTHOR]

In unstructured environments, harvesting robots may collide with disorderly growing branches, thus reducing the success rate of harvesting. This study introduces a fruit and branch detection and three-dimensional (3D) reconstruction method for obstacle avoidance path planning of robots. A new architecture for instance segmentation was developed by replacing the backbone of Mask R-CNN with a tiny network, referred to as "tiny Mask R-CNN". The tiny Mask R-CNN was trained with a small number of images and used to detect guava fruits and branches. Each detected fruit and branch were converted into a 3D point cloud. It was then hypothesized that guava fruits could be represented by 3D spheres and irregular branches can be approximated by a finite number of 3D cylindrical segments. Based on the proposed hypothesis, a random sample consensus-based sphere fitting method and a principal component analysis-based cylindrical segment fitting method were investigated to reconstruct the fruits and branches from the point clouds. A guava dataset with 304 RGB-D images was collected from the fields and used to validate the developed method. The results showed that the detection F1 score of the tiny Mask R-CNN was 0.518; the F1 score for fruit reconstruction was approximately 0.851 and 0.833 under the 2D- and 3D-fruit metrics, respectively; and the F1 score for branch reconstruction was approximately 0.394 and 0.415 under the 2D- and 3D-branch metrics, respectively. These results confirm that the proposed method can effectively reconstruct the fruits and branches and can, therefore, be used to plan an obstacle avoidance path for harvesting robots. [ABSTRACT FROM AUTHOR]