Search

The agro-food industry is turning to robots to address the challenge of labour shortage. However, agro-food environments pose difficulties for robots due to high variation and occlusions. In the presence of these challenges, accurate world models, with information about object location, shape, and properties, are crucial for robots to perform tasks accurately. Building such models is challenging due to the complex and unique nature of agro-food environments, and errors in the model can lead to task execution issues. In this paper, MinkSORT, a novel method for generating tracking features using a 3D sparse convolutional network in a deepSORT-like approach, is proposed to improve the accuracy of world models in agro-food environments. MinkSORT was evaluated using real-world data collected in a tomato greenhouse, where it significantly improved the performance of a baseline model that tracks tomato positions in 3D using a Kalman filter and Mahalanobis distance. MinkSORT improved the HOTA from 42.8% to 44.77%, the association accuracy from 32.55% to 35.55%, and the MOTA from 57.63% to 58.81%. Different contrastive loss functions for training MinkSORT were also evaluated, and it was demonstrated that it leads to improved performance in terms of three separate precision and recall detection outcomes. The proposed method improves world model accuracy, enabling robots to perform tasks such as harvesting and plant maintenance with greater efficiency and accuracy, which is essential for meeting the growing demand for food in a sustainable manner. • MinkSORT: A novel method of using 3D sparse convolutions for multi-object tracking. • A novel implementation of the concept of DeepSORT for 3D data like point clouds. • Real world experiments in a challenging tomato greenhouse environment. • An evaluation of MinkSORT showing a tracking accuracy increase up to 7.72 points. [ABSTRACT FROM AUTHOR]

Computer-vision based methods contribute to the availability of high-quality phenotypic datasets. Most computer-vision based methods for plant phenotyping are based on analysis of 2D images. However, previous research showed that for traits related to plant architecture, like internode length, a main limitation of 2D methods was that plants with a curved growing pattern could not be accurately measured. In this work, it was hypothesised that methods based on 3D data can overcome this limitation, while increasing the overall accuracy of the internode length measurements. To test the hypothesis, a method was proposed to estimate internode lengths from 3D point clouds of cucumber plants. First, a deep neural network based on PointNet++ was trained to segment the point clouds into plant parts. The points that were predicted as 'node' were then selected and a clustering algorithm was used to group points belonging to the same node. The Euclidean distance between the detected nodes was used as an estimate of the internode length. The results were compared to the results of a previously published method based on 2D images. The results of the 3D method were significantly more accurate than the results of the 2D method. Moreover, in contrast to the 2D method, the internode length estimates of the 3D method were equally accurate for curved plants as well as for straight plants. The results clearly demonstrated that computer-vision based methods to measure plant architecture in general, and more specifically to measure internode length, greatly benefit from the availability of 3D data. • Digital plant phenotyping greatly benefits from the availability of 3D data. • Internode lengths estimated from 3D point clouds and 2D images were compared. • The accuracy of the estimated internode lengths was higher when using 3D point clouds. • The node detection rate was higher when using 2D images. • Combining 2D images and 3D point clouds could provide the best of both worlds. [ABSTRACT FROM AUTHOR]

The ability to accurately represent and localise relevant objects is essential for robots to carry out tasks effectively. Traditional approaches, where robots simply capture an image, process that image to take an action, and then forget the information, have proven to struggle in the presence of occlusions. Methods using multi-view perception, which have the potential to address some of these problems, require a world model that guides the collection, integration and extraction of information from multiple viewpoints. Furthermore, constructing a generic representation that can be applied in various environments and tasks is a difficult challenge. In this paper, a novel approach for building generic representations in occluded agro-food environments using multi-view perception and 3D multi-object tracking is introduced. The method is based on a detection algorithm that generates partial point clouds for each detected object, followed by a 3D multi-object tracking algorithm that updates the representation over time. The accuracy of the representation was evaluated in a real-world environment, where successful representation and localisation of tomatoes in tomato plants were achieved, despite high levels of occlusion, with the total count of tomatoes estimated with a maximum error of 5.08% and the tomatoes tracked with an accuracy up to 71.47%. Novel tracking metrics were introduced, demonstrating that valuable insight into the errors in localising and representing the fruits can be provided by their use. This approach presents a novel solution for building representations in occluded agro-food environments, demonstrating potential to enable robots to perform tasks effectively in these challenging environments. • A novel method to build world models for robots based on 3D object tracking. • A real world evaluation of our method in highly occluded scenarios. • Detection and association performance decreases with occlusions. • 3D object tracking yields an accuracy up to 71%. • Tracking metrics give more insight than the total count of objects. [ABSTRACT FROM AUTHOR]

Plant scientists require high quality phenotypic datasets. Computer-vision based methods can improve the objectiveness and the accuracy of phenotypic measurements. In this paper, we focus on 3D point clouds for measuring plant architecture of cucumber plants, using spectral data and deep learning (DL). More specifically, the focus of this paper is on the segmentation of the point clouds, such that for each point it is known to which plant part (e.g. leaf or stem) it belongs. It was shown that the availability of spectral data can improve the segmentation, with the mean intersection-over-union rising from 0.90 to 0.95. Furthermore, we analysed the effect of uncertainty in the collection of ground truth data. For this purpose, we hand-labelled 264 point clouds of cucumber plants twice and show that the intra-observer variability between those two annotation sets can be as low as 0.49 for difficult classes, while it was 0.99 for the class with the least uncertainty. Adding the second set of hand-labelled data to the training of the network improved the segmentation performance slightly. Finally, we show the improved performance of a 4-class segmentation over an 8-class segmentation, emphasizing the need for a careful design of plant phenotyping experiments. The results presented in this paper contribute to further development of automated phenotyping methods for complex plant traits. • Adding spectral data boosts 3D point cloud segmentation for plant phenotyping. • High intra-observer variability shows that ground-truth collection is challenging. • Labelling the same dataset multiple times can improve the segmentation performance. • Careful design of phenotyping experiments makes a computer scientist's life easier. [ABSTRACT FROM AUTHOR]