Search

Citrus fruits are extensively cultivated fruits with high nutritional value. The identification of distinct ripeness stages in citrus fruits plays a crucial role in guiding the planning of harvesting paths for citrus-picking robots and facilitating yield estimations in orchards. However, challenges arise in the identification of citrus fruit ripeness due to the similarity in color between green unripe citrus fruits and tree leaves, leading to an omission in identification. Additionally, the resemblance between partially ripe, orange-green interspersed fruits and fully ripe fruits poses a risk of misidentification, further complicating the identification of citrus fruit ripeness. This study proposed the YOLO-CIT (You Only Look Once-Citrus) model and integrated an innovative R-LBP (Roughness-Local Binary Pattern) method to accurately identify citrus fruits at distinct ripeness stages. The R-LBP algorithm, an extension of the LBP algorithm, enhances the texture features of citrus fruits at distinct ripeness stages by calculating the coefficient of variation in grayscale values of pixels within a certain range in different directions around the target pixel. The C3 model embedded by the CBAM (Convolutional Block Attention Module) replaced the original backbone network of the YOLOv5s model to form the backbone of the YOLO-CIT model. Instead of traditional convolution, Ghostconv is utilized by the neck network of the YOLO-CIT model. The fruit segment of citrus in the original citrus images processed by the R-LBP algorithm is combined with the background segment of the citrus images after grayscale processing to construct synthetic images, which are subsequently added to the training dataset. The experiment showed that the R-LBP algorithm is capable of amplifying the texture features among citrus fruits at distinct ripeness stages. The YOLO-CIT model combined with the R-LBP algorithm has a Precision of 88.13%, a Recall of 93.16%, an F1 score of 90.89, a mAP@0.5 of 85.88%, and 6.1ms of average detection speed for citrus fruit ripeness identification in complex environments. The model demonstrates the capability to accurately and swiftly identify citrus fruits at distinct ripeness stages in real-world environments, effectively guiding the determination of picking targets and path planning for harvesting robots.

Berries are nutritious and valuable, but their thin skin, soft flesh, and fragility make harvesting and picking challenging. Manual and traditional mechanical harvesting methods are commonly used, but they are costly in labor and can damage the fruit. To overcome these challenges, it may be worth exploring alternative harvesting methods. Using berry fruit-picking robots with perception technology is a viable option to improve the efficiency of berry harvesting. This review presents an overview of the mechanisms of berry fruit-picking robots, encompassing their underlying principles, the mechanics of picking and grasping, and an examination of their structural design. The importance of perception technology during the picking process is highlighted. Then, several perception techniques commonly used by berry fruit-picking robots are described, including visual perception, tactile perception, distance measurement, and switching sensors. The methods of these four perceptual techniques used by berry-picking robots are described, and their advantages and disadvantages are analyzed. In addition, the technical characteristics of perception technologies in practical applications are analyzed and summarized, and several advanced applications of berry fruit-picking robots are presented. Finally, the challenges that perception technologies need to overcome and the prospects for overcoming these challenges are discussed.

As strawberries are a widely grown cash crop, the development of strawberry fruit-picking robots for an intelligent harvesting system should match the rapid development of strawberry cultivation technology. Ripeness identification is a key step to realizing selective harvesting by strawberry fruit-picking robots. Therefore, this study proposes combining deep learning and image processing for target detection and classification of ripe strawberries. First, the YOLOv8+ model is proposed for identifying ripe and unripe strawberries and extracting ripe strawberry targets in images. The ECA attention mechanism is added to the backbone network of YOLOv8+ to improve the performance of the model, and Focal-EIOU loss is used in loss function to solve the problem of imbalance between easy- and difficult-to-classify samples. Second, the centerline of the ripe strawberries is extracted, and the red pixels in the centerline of the ripe strawberries are counted according to the H-channel of their hue, saturation, and value (HSV). The percentage of red pixels in the centerline is calculated as a new parameter to quantify ripeness, and the ripe strawberries are classified as either fully ripe strawberries or not fully ripe strawberries. The results show that the improved YOLOv8+ model can accurately and comprehensively identify whether the strawberries are ripe or not, and the mAP50 curve steadily increases and converges to a relatively high value, with an accuracy of 97.81%, a recall of 96.36%, and an F1 score of 97.07. The accuracy of the image processing method for classifying ripe strawberries was 91.91%, FPR was 5.03%, and FNR was 14.28%. This study demonstrates the program’s ability to quickly and accurately identify strawberries at different stages of ripeness in a facility environment, which can provide guidance for selective picking by subsequent fruit-picking robots.

Reasonably formulating the strawberry harvesting sequence can improve the quality of harvested strawberries and reduce strawberry decay. Growth information based on drone image processing can assist the strawberry harvesting, however, it is still a challenge to develop a reliable method for object identification in drone images. This study proposed a deep learning method, including an improved YOLOv8 model and a new image-processing framework, which could accurately and comprehensively identify mature strawberries, immature strawberries, and strawberry flowers in drone images. The improved YOLOv8 model used the shuffle attention block and the VoV–GSCSP block to enhance identification accuracy and detection speed. The environmental stability-based region segmentation was used to extract the strawberry plant area (including fruits, stems, and leaves). Edge extraction and peak detection were used to estimate the number of strawberry plants. Based on the number of strawberry plants and the distribution of mature strawberries, we draw a growth chart of strawberries (reflecting the urgency of picking in different regions). The experiment showed that the improved YOLOv8 model demonstrated an average accuracy of 82.50% in identifying immature strawberries, 87.40% for mature ones, and 82.90% for strawberry flowers in drone images. The model exhibited an average detection speed of 6.2 ms and a model size of 20.1 MB. The proposed new image-processing technique estimated the number of strawberry plants in a total of 100 images. The bias of the error for images captured at a height of 2 m is 1.1200, and the rmse is 1.3565; The bias of the error for the images captured at a height of 3 m is 2.8400, and the rmse is 3.0199. The assessment of picking priorities for various regions of the strawberry field in this study yielded an average accuracy of 80.53%, based on those provided by 10 experts. By capturing images throughout the entire growth cycle, we can calculate the harvest index for different regions. This means farmers can not only obtain overall ripeness information of strawberries in different regions but also adjust agricultural strategies based on the harvest index to improve both the quantity and quality of fruit set on strawberry plants, as well as plan the harvesting sequence for high-quality strawberry yields.

Blueberry is among the fruits with high economic gains for orchard farmers. Identification of blueberry fruits with different maturities has economic significance to help orchard farmers plan pesticide application, estimate yield, and conduct harvest operations efficiently. Vision systems for automated orchard yield estimation have received growing attention toward fruit identification with different maturity stages. However, due to interfering factors such as varying outdoor illuminations, similar colors with the surrounding canopy, imaging distance, and occlusion in natural environments, it remains a serious challenge to develop reliable visual methods for identifying blueberry fruits with different maturities. This study constructed a YOLO-BLBE (Blueberry) model combined with an innovative I-MSRCR (Improved MSRCR (Multi-Scale Retinex with Color Restoration)) method to accurately identify blueberry fruits with different maturities. The color feature of blueberry fruit in the original image was enhanced by the I-MSRCR algorithm, which was improved based on the traditional MSRCR algorithm by adjusting the proportion of color restoration factors. The GhostNet model embedded by the CA (coordinate attention) mechanism module replaced the original backbone network of the YOLOv5s model to form the backbone of the YOLO-BLBE model. The BIFPN (Bidirectional Feature Pyramid Network) structure was applied in the neck network of the YOLO-BLBE model, and Alpha-EIOU was used as the loss function of the model to determine and filter candidate boxes. The main contributions of this study are as follows: (1) The I-MSRCR algorithm proposed in this paper can effectively amplify the color differences between blueberry fruits of different maturities. (2) Adding the synthesized blueberry images processed by the I-MSRCR algorithm to the training set for training can improve the model’s recognition accuracy for blueberries of different maturity levels. (3) The YOLO-BLBE model achieved an average identification accuracy of 99.58% for mature blueberry fruits, 96.77% for semi-mature blueberry fruits, and 98.07% for immature blueberry fruits. (4) The YOLO-BLBE model had a size of 12.75 MB and an average detection speed of 0.009 s.

Litchi is a highly favored fruit with high economic value. Mechanical automation of litchi picking is a key link for improving the quality and efficiency of litchi harvesting. Our research team has been conducting experiments to develop a visual-based litchi picking robot. However, in the early physical prototype experiments, we found that, although picking points were successfully located, litchi picking failed due to random obstructions of the picking points. In this study, the physical prototype of the litchi picking robot previously developed by our research team was upgraded by integrating a visual system for actively removing obstructions. A framework for an artificial intelligence algorithm was proposed for a robot vision system to locate picking points and to identify obstruction situations at picking points. An intelligent control algorithm was developed to control the obstruction removal device to implement obstruction removal operations by combining with the obstruction situation at the picking point. Based on the spatial redundancy of a picking point and the obstruction, the feeding posture of the robot was determined. The experiment showed that the precision of segmenting litchi fruits and branches was 88.1%, the recognition success rate of picking point recognition was 88%, the average error of picking point localization was 2.8511 mm, and an overall success rate of end-effector feeding was 81.3%. These results showed that the developed litchi picking robot could effectively implement obstruction removal.