Search

The agro-food industry is turning to robots to address the challenge of labour shortage. However, agro-food environments pose difficulties for robots due to high variation and occlusions. In the presence of these challenges, accurate world models, with information about object location, shape, and properties, are crucial for robots to perform tasks accurately. Building such models is challenging due to the complex and unique nature of agro-food environments, and errors in the model can lead to task execution issues. In this paper, MinkSORT, a novel method for generating tracking features using a 3D sparse convolutional network in a deepSORT-like approach, is proposed to improve the accuracy of world models in agro-food environments. MinkSORT was evaluated using real-world data collected in a tomato greenhouse, where it significantly improved the performance of a baseline model that tracks tomato positions in 3D using a Kalman filter and Mahalanobis distance. MinkSORT improved the HOTA from 42.8% to 44.77%, the association accuracy from 32.55% to 35.55%, and the MOTA from 57.63% to 58.81%. Different contrastive loss functions for training MinkSORT were also evaluated, and it was demonstrated that it leads to improved performance in terms of three separate precision and recall detection outcomes. The proposed method improves world model accuracy, enabling robots to perform tasks such as harvesting and plant maintenance with greater efficiency and accuracy, which is essential for meeting the growing demand for food in a sustainable manner. • MinkSORT: A novel method of using 3D sparse convolutions for multi-object tracking. • A novel implementation of the concept of DeepSORT for 3D data like point clouds. • Real world experiments in a challenging tomato greenhouse environment. • An evaluation of MinkSORT showing a tracking accuracy increase up to 7.72 points. [ABSTRACT FROM AUTHOR]

Computer-vision based methods contribute to the availability of high-quality phenotypic datasets. Most computer-vision based methods for plant phenotyping are based on analysis of 2D images. However, previous research showed that for traits related to plant architecture, like internode length, a main limitation of 2D methods was that plants with a curved growing pattern could not be accurately measured. In this work, it was hypothesised that methods based on 3D data can overcome this limitation, while increasing the overall accuracy of the internode length measurements. To test the hypothesis, a method was proposed to estimate internode lengths from 3D point clouds of cucumber plants. First, a deep neural network based on PointNet++ was trained to segment the point clouds into plant parts. The points that were predicted as 'node' were then selected and a clustering algorithm was used to group points belonging to the same node. The Euclidean distance between the detected nodes was used as an estimate of the internode length. The results were compared to the results of a previously published method based on 2D images. The results of the 3D method were significantly more accurate than the results of the 2D method. Moreover, in contrast to the 2D method, the internode length estimates of the 3D method were equally accurate for curved plants as well as for straight plants. The results clearly demonstrated that computer-vision based methods to measure plant architecture in general, and more specifically to measure internode length, greatly benefit from the availability of 3D data. • Digital plant phenotyping greatly benefits from the availability of 3D data. • Internode lengths estimated from 3D point clouds and 2D images were compared. • The accuracy of the estimated internode lengths was higher when using 3D point clouds. • The node detection rate was higher when using 2D images. • Combining 2D images and 3D point clouds could provide the best of both worlds. [ABSTRACT FROM AUTHOR]

The ability to accurately represent and localise relevant objects is essential for robots to carry out tasks effectively. Traditional approaches, where robots simply capture an image, process that image to take an action, and then forget the information, have proven to struggle in the presence of occlusions. Methods using multi-view perception, which have the potential to address some of these problems, require a world model that guides the collection, integration and extraction of information from multiple viewpoints. Furthermore, constructing a generic representation that can be applied in various environments and tasks is a difficult challenge. In this paper, a novel approach for building generic representations in occluded agro-food environments using multi-view perception and 3D multi-object tracking is introduced. The method is based on a detection algorithm that generates partial point clouds for each detected object, followed by a 3D multi-object tracking algorithm that updates the representation over time. The accuracy of the representation was evaluated in a real-world environment, where successful representation and localisation of tomatoes in tomato plants were achieved, despite high levels of occlusion, with the total count of tomatoes estimated with a maximum error of 5.08% and the tomatoes tracked with an accuracy up to 71.47%. Novel tracking metrics were introduced, demonstrating that valuable insight into the errors in localising and representing the fruits can be provided by their use. This approach presents a novel solution for building representations in occluded agro-food environments, demonstrating potential to enable robots to perform tasks effectively in these challenging environments. • A novel method to build world models for robots based on 3D object tracking. • A real world evaluation of our method in highly occluded scenarios. • Detection and association performance decreases with occlusions. • 3D object tracking yields an accuracy up to 71%. • Tracking metrics give more insight than the total count of objects. [ABSTRACT FROM AUTHOR]

Plant scientists require high quality phenotypic datasets. Computer-vision based methods can improve the objectiveness and the accuracy of phenotypic measurements. In this paper, we focus on 3D point clouds for measuring plant architecture of cucumber plants, using spectral data and deep learning (DL). More specifically, the focus of this paper is on the segmentation of the point clouds, such that for each point it is known to which plant part (e.g. leaf or stem) it belongs. It was shown that the availability of spectral data can improve the segmentation, with the mean intersection-over-union rising from 0.90 to 0.95. Furthermore, we analysed the effect of uncertainty in the collection of ground truth data. For this purpose, we hand-labelled 264 point clouds of cucumber plants twice and show that the intra-observer variability between those two annotation sets can be as low as 0.49 for difficult classes, while it was 0.99 for the class with the least uncertainty. Adding the second set of hand-labelled data to the training of the network improved the segmentation performance slightly. Finally, we show the improved performance of a 4-class segmentation over an 8-class segmentation, emphasizing the need for a careful design of plant phenotyping experiments. The results presented in this paper contribute to further development of automated phenotyping methods for complex plant traits. • Adding spectral data boosts 3D point cloud segmentation for plant phenotyping. • High intra-observer variability shows that ground-truth collection is challenging. • Labelling the same dataset multiple times can improve the segmentation performance. • Careful design of phenotyping experiments makes a computer scientist's life easier. [ABSTRACT FROM AUTHOR]

• The YOLOv3 model was trained on data acquired in 20 arable fields. • The generalization error on 5 different arable fields was evaluated. • The effect of the number of training images was investigated. • The effect of the number of training fields was investigated. • The generalization error was mitigated using incremental training. Lack of generalization in plant-detection models is one of the main challenges preventing the realization of autonomous weed-control systems. This paper investigates the effect of the train and test dataset distribution on the generalization error of a plant-detection model and uses incremental training to mitigate the said error. In this paper, we use the YOLOv3 object detector as plant-detection model. To train the model and test its generalization properties we used a broad dataset, consisting of 25 sub-datasets, sampled from multiple different geographic areas, soil types, cultivation conditions, containing variation in weeds, background vegetation, camera quality and variations in illumination. Using this dataset we evaluated the generalization error of a plant-detection model, assessed the effect of sampling training images from multiple arable fields on the generalization of our plant-detection model, we investigated the relation between the number of training images and the generalization of the plant-detection model and we applied incremental training to mitigate the generalization error of our plant-detection model on new arable fields. It was found that the average generalization error of our plant-detection model was 0.06 mAP. Increasing the number of sub-datasets for training, while keeping the total number of training images constant, increased the variation covered by the training set and improved the generalization of our plant-detection model. Adding more training images sampled from the same datasets increased the generalization further. However, this effect is limited and only holds when the new images cover new variation. Naively adding more images does not prepare the model for specific scenarios outside the training distribution. Using incremental training the model can be adapted to such scenarios and the generalization error can be mitigated. Depending on the discrepancy between the training set and the new field, finetuning on as little as 25 images can already mitigate the generalization error. [ABSTRACT FROM AUTHOR]

Classification of weeds amongst cash crops is a core procedure in automated weed control. Addressing volunteer potato control in sugar beets, in the EU Smartbot project the aim was to control more than 95% of volunteer potatoes and ensure less than 5% of undesired control of sugar beet plants. A promising way to meet these requirements is deep learning. Training an entire network from scratch, however, requires a large dataset and a substantial amount of time. In this situation, transfer learning can be a promising solution. This study first evaluates a transfer learning procedure with three different implementations of AlexNet and then assesses the performance difference amongst the six network architectures: AlexNet, VGG-19, GoogLeNet, ResNet-50, ResNet-101 and Inception-v3. All nets had been pre-trained on the ImageNet Dataset. These nets were used to classify sugar beet and volunteer potato images taken under ambient varying light conditions in agricultural environments. The highest classification accuracy for different implementations of AlexNet was 98.0%, obtained with an AlexNet architecture modified to generate binary output. Comparing different networks, the highest classification accuracy 98.7%, obtained with VGG-19 modified to generate binary output. Transfer learning proved to be effective and showed robust performance with plant images acquired in different periods of the various years on two types of soils. All scenarios and pre-trained networks were feasible for real-time applications (classification time < 0.1 s). Classification is only one step in weed detection, and a complete pipeline for weed detection may potentially reduce the overall performance. Highlights • Transfer learning provided very promising performance for weed/crop classification. • The highest classification accuracy of 98.7% was obtained with VGG-19. • All scenarios and pre-trained networks were feasible for real-time applications. • Data augmentation may improve the classification accuracy. [ABSTRACT FROM AUTHOR]

• MaskAL is software that can reduce the annotation effort for training Mask R-CNN. • MaskAL uses active learning to select images about which Mask R-CNN is uncertain. • On a broccoli dataset, MaskAL performed significantly better than random sampling. • By using MaskAL, 1400 image annotations were saved compared to random sampling. • The software is available on https://github.com/pieterblok/maskal. The generalisation performance of a convolutional neural network (CNN) is influenced by the quantity, quality, and variety of the training images. Training images must be annotated, and this is time consuming and expensive. The goal of our work was to reduce the number of annotated images needed to train a CNN while maintaining its performance. We hypothesised that the performance of a CNN can be improved faster by ensuring that the set of training images contains a large fraction of hard-to-classify images. The objective of our study was to test this hypothesis with an active learning method that can automatically select the hard-to-classify images. We developed an active learning method for Mask Region-based CNN (Mask R-CNN) and named this method MaskAL. MaskAL involved the iterative training of Mask R-CNN, after which the trained model was used to select a set of unlabelled images about which the model was most uncertain. The selected images were then annotated and used to retrain Mask R-CNN, and this was repeated for a number of sampling iterations. In our study, MaskAL was compared to a random sampling method on a broccoli dataset with five visually similar classes. MaskAL performed significantly better than the random sampling. In addition, MaskAL had the same performance after sampling 900 images as the random sampling had after 2300 images. Compared to a Mask R-CNN model that was trained on the entire training set (14,000 images), MaskAL achieved 93.9% of that model's performance with 17.9% of its training data. The random sampling achieved 81.9% of that model's performance with 16.4% of its training data. We conclude that by using MaskAL, the annotation effort can be reduced for training Mask R-CNN on a broccoli dataset with visually similar classes. Our software is available on https://github.com/pieterblok/maskal. [ABSTRACT FROM AUTHOR]