Search

Robots in tomato greenhouses need to perceive the plant and plant parts accurately to automate monitoring, harvesting, and de-leafing tasks. Existing perception systems struggle with the high levels of occlusion in plants and often result in poor perception accuracy. One reason for this is because they use fixed cameras or predefined camera movements. Next-best-view (NBV) planning presents an alternate approach, in which the camera viewpoints are reasoned and strategically planned such that the perception accuracy is improved. However, existing NBV-planning algorithms are agnostic to the task-at-hand and give equal importance to all the plant parts. This strategy is inefficient for greenhouse tasks that require targeted perception of specific plant parts, such as the perception of leaf nodes for de-leafing. To improve targeted perception in complex greenhouse environments, NBV planning algorithms need an attention mechanism to focus on the task-relevant plant parts. In this paper, the role of attention in improving targeted perception using an attention-driven NBV planning strategy was investigated. Through simulation experiments using plants with high levels of occlusion and structural complexity, it was shown that focusing attention on task-relevant plant parts can significantly improve the speed and accuracy of 3D reconstruction. Further, with real-world experiments, it was shown that these benefits extend to complex greenhouse conditions with natural variation and occlusion, natural illumination, sensor noise, and uncertainty in camera poses. The results clearly indicate that using attention-driven NBV planning in greenhouses can significantly improve the efficiency of perception and enhance the performance of robotic systems in greenhouse crop production.

Searching and detecting the task-relevant parts of plants is important to automate harvesting and de-leafing of tomato plants using robots. This is challenging due to high levels of occlusion in tomato plants. Active vision is a promising approach in which the robot strategically plans its camera viewpoints to overcome occlusion and improve perception accuracy. However, current active-vision algorithms cannot differentiate between relevant and irrelevant plant parts and spend time on perceiving irrelevant plant parts. This work proposed a semantics-aware active-vision strategy that uses semantic information to identify the relevant plant parts and prioritise them during view planning. The proposed strategy was evaluated on the task of searching and detecting the relevant plant parts using simulation and real-world experiments. In simulation experiments, the semantics-aware strategy proposed could search and detect 81.8% of the relevant plant parts using nine viewpoints. It was significantly faster and detected more plant parts than predefined, random, and volumetric active-vision strategies that do not use semantic information. The strategy proposed was also robust to uncertainty in plant and plant-part positions, plant complexity, and different viewpoint-sampling strategies. In real-world experiments, the semantics-aware strategy could search and detect 82.7% of the relevant plant parts using seven viewpoints, under complex greenhouse conditions with natural variation and occlusion, natural illumination, sensor noise, and uncertainty in camera poses. The results of this work clearly indicate the advantage of using semantics-aware active vision for targeted perception of plant parts and its applicability in the real world. It can significantly improve the efficiency of automated harvesting and de-leafing in tomato crop production.

Within the modern greenhouse horticultural sector energy usage is planned using mathematical models that simulate the greenhouse's future performance. These models contain parameters whose values can be inaccurate which create errors in model predictions. This reduces the effectiveness of energy management and planning done using these models. This study proposes and demonstrates an algorithm to quantify the impact of parameter errors on greenhouse gas and electric power prediction uncertainty. The proposed algorithm introduces a Polynomial Chaos Expansion as a method for the sensitivity analysis in the domain of greenhouse horticulture. Contrary to commonly used sensitivity analyses, this approach introduces the analysis of higher order interactions into the domain of greenhouse horticultural research. It was found that for both electric power and gas production the HPS lamp power rating was the most influential individual parameter. Moreover, this study found that for power demand the uncertainty in parameters relating to the lamp system were far more impactful than those related to the crop or greenhouse structure, with a respective coefficient of variation of 24 %, 5 % and 5 %. This study makes a notable and novel conclusion that for parameters related to the greenhouse structure, larger groups of parameters were responsible for prediction uncertainty through higher order interactions of second to sixth order. These results reinforce the importance of future greenhouse research considering the impact of higher order parameter interactions on prediction uncertainty using the algorithm proposed in this study.

Computer-vision based methods contribute to the availability of high-quality phenotypic datasets. Most computer-vision based methods for plant phenotyping are based on analysis of 2D images. However, previous research showed that for traits related to plant architecture, like internode length, a main limitation of 2D methods was that plants with a curved growing pattern could not be accurately measured. In this work, it was hypothesised that methods based on 3D data can overcome this limitation, while increasing the overall accuracy of the internode length measurements. To test the hypothesis, a method was proposed to estimate internode lengths from 3D point clouds of cucumber plants. First, a deep neural network based on PointNet++ was trained to segment the point clouds into plant parts. The points that were predicted as ‘node’ were then selected and a clustering algorithm was used to group points belonging to the same node. The Euclidean distance between the detected nodes was used as an estimate of the internode length. The results were compared to the results of a previously published method based on 2D images. The results of the 3D method were significantly more accurate than the results of the 2D method. Moreover, in contrast to the 2D method, the internode length estimates of the 3D method were equally accurate for curved plants as well as for straight plants. The results clearly demonstrated that computer-vision based methods to measure plant architecture in general, and more specifically to measure internode length, greatly benefit from the availability of 3D data.

The ability to accurately represent and localise relevant objects is essential for robots to carry out tasks effectively. Traditional approaches, where robots simply capture an image, process that image to take an action, and then forget the information, have proven to struggle in the presence of occlusions. Methods using multi-view perception, which have the potential to address some of these problems, require a world model that guides the collection, integration and extraction of information from multiple viewpoints. Furthermore, constructing a generic representation that can be applied in various environments and tasks is a difficult challenge. In this paper, a novel approach for building generic representations in occluded agro-food environments using multi-view perception and 3D multi-object tracking is introduced. The method is based on a detection algorithm that generates partial point clouds for each detected object, followed by a 3D multi-object tracking algorithm that updates the representation over time. The accuracy of the representation was evaluated in a real-world environment, where successful representation and localisation of tomatoes in tomato plants were achieved, despite high levels of occlusion, with the total count of tomatoes estimated with a maximum error of 5.08% and the tomatoes tracked with an accuracy up to 71.47%. Novel tracking metrics were introduced, demonstrating that valuable insight into the errors in localising and representing the fruits can be provided by their use. This approach presents a novel solution for building representations in occluded agro-food environments, demonstrating potential to enable robots to perform tasks effectively in these challenging environments.

The agro-food industry is turning to robots to address the challenge of labour shortage. However, agro-food environments pose difficulties for robots due to high variation and occlusions. In the presence of these challenges, accurate world models, with information about object location, shape, and properties, are crucial for robots to perform tasks accurately. Building such models is challenging due to the complex and unique nature of agro-food environments, and errors in the model can lead to task execution issues. In this paper, MinkSORT, a novel method for generating tracking features using a 3D sparse convolutional network in a deepSORT-like approach, is proposed to improve the accuracy of world models in agro-food environments. MinkSORT was evaluated using real-world data collected in a tomato greenhouse, where it significantly improved the performance of a baseline model that tracks tomato positions in 3D using a Kalman filter and Mahalanobis distance. MinkSORT improved the HOTA from 42.8% to 44.77%, the association accuracy from 32.55% to 35.55%, and the MOTA from 57.63% to 58.81%. Different contrastive loss functions for training MinkSORT were also evaluated, and it was demonstrated that it leads to improved performance in terms of three separate precision and recall detection outcomes. The proposed method improves world model accuracy, enabling robots to perform tasks such as harvesting and plant maintenance with greater efficiency and accuracy, which is essential for meeting the growing demand for food in a sustainable manner.

Currently the Dutch greenhouse horticultural sector has a high energy demand. The present use of weather forecasts can exacerbate this high energy consumption by contributing to suboptimal prediction and trading of the greenhouse's power demand. This study investigates the role of weather forecast errors on energy prediction power and trading uncertainty in greenhouse horticulture. This was done using an uncertainty analysis and computer model of a tomato producing Venlo style greenhouse in Bleiswijk, The Netherlands. This greenhouse model was used to predict the greenhouse's gas and electrical power demand. The study concluded that errors in the weather forecast of outdoor radiation, temperature and wind speed caused an overestimation of greenhouse energy demand. A sensitivity analysis showed that the radiation forecast error had the greatest impact on predicted greenhouse power demand errors with a mean relative error of 6.1%. Predicted gas demand errors were most dependent on the outside wind speed forecast mean relative error (18.0%) and temperature forecast error (17.2%). A power trading uncertainty analysis was done to investigate the impact of predicted energy demand errors on the cost of buying power on the Dutch imbalance and Amsterdam Power Exchange day-ahead market. This cost analysis found that the volume of initial power trading was greater than corrective trading. Additionally, the higher volatility in short term power prices resulted in higher corrective power costs per unit of power than if the power demand had been initially predicted with more accuracy.

In the published simulation studies on greenhouse climate control that employ optimal control, often non-realistic weather forecasts are employed, e.g. the realisation of the weather or artificially created forecasts are used. This research aims to quantify the effect of weather forecast errors on the performance of the controlled greenhouse system measured in terms of operational return. The operational return is defined as the difference between the cost of resources (resourceuse×cost) and the income through yield (yield×productprice). A stochastic model of the weather forecast error was identified based on historical weather observations and forecasts from a weather forecasting service. An uncertainty analysis using the stochastic model showed that a considerable number of control inputs are sensitive to the forecast errors. A simulation study involving three 7day-intervals throughout the growing season showed, however, that the performance of the controlled greenhouse system is not significantly affected by the forecast error, a performance decrease of 0.03euro.m−2 (2%) was observed with respect to the case in which perfect forecasts were used. The results suggest that an optimal control algorithm which (a) is updated every 15min with the full state information, (b) uses forecasts published every 6h and (c) uses published forecasts with a weather forecast error similar to the weather forecasting service used here, is able to mitigate the effect of the weather forecast error on the performance of the greenhouse system.

Plant scientists require high quality phenotypic datasets. Computer-vision based methods can improve the objectiveness and the accuracy of phenotypic measurements. In this paper, we focus on 3D point clouds for measuring plant architecture of cucumber plants, using spectral data and deep learning (DL). More specifically, the focus of this paper is on the segmentation of the point clouds, such that for each point it is known to which plant part (e.g. leaf or stem) it belongs. It was shown that the availability of spectral data can improve the segmentation, with the mean intersection-over-union rising from 0.90 to 0.95. Furthermore, we analysed the effect of uncertainty in the collection of ground truth data. For this purpose, we hand-labelled 264 point clouds of cucumber plants twice and show that the intra-observer variability between those two annotation sets can be as low as 0.49 for difficult classes, while it was 0.99 for the class with the least uncertainty. Adding the second set of hand-labelled data to the training of the network improved the segmentation performance slightly. Finally, we show the improved performance of a 4-class segmentation over an 8-class segmentation, emphasizing the need for a careful design of plant phenotyping experiments. The results presented in this paper contribute to further development of automated phenotyping methods for complex plant traits.

This paper presents a receding horizon optimal control (RHOC) method with an economic objective function for balancing the cost of resources () with income through yield (). This paper considers the two elements that determine the income through yield. The first element is yield and associated fruit development. A new, computationally viable, approach to model the income through yield is proposed and its prediction accuracy with respect to the original model is evaluated. The new approach employs a model that predicts at each time step, the future income through yield based on the assimilates partitioned to the fruits at the current time step. Simulations suggest that the assumptions made to arrive at the model for the new approach, do not significantly affect the accuracy of the predictions. The second element considered in this paper is the product price and the uncertainty inherent in its forecasts. Historical product price data are used to generate artificial product price forecasts. An uncertainty analysis, in combination with the artificial product price forecasts, showed that the product price forecast error does not considerably affect the optimised control strategy. Season-wide simulations with RHOC suggest that the product price forecast error does not considerably affect the value of the economic objective function.

The growth and the harvestability of a broccoli crop is monitored by the size of the broccoli head. This size estimation is currently done by humans, and this is inconsistent and expensive. The goal of our work was to develop a software algorithm that can estimate the size of field-grown broccoli heads based on RGB-Depth (RGB-D) images. For the algorithm to be successful, the problem of occlusion must be solved, which is the partial visibility of the broccoli head due to overlapping leaves. This partial visibility causes sizing errors. In this research, we studied the use of deep-learning algorithms to deal with occlusions. We specifically applied the Occlusion Region-based Convolutional Neural Network (ORCNN) that segmented both the visible and the amodal region of the broccoli head (which is the visible and the occluded region combined). We hypothesised that ORCNN, with its amodal segmentation, can improve the size estimation of occluded broccoli heads. The ORCNN sizing method was compared with a Mask R–CNN sizing method that only used the visible broccoli region to estimate the size. The sizing performance of both methods was evaluated on a test set of 487 broccoli images with systematic levels of leaf occlusion. With a mean sizing error of 6.4 mm, ORCNN outperformed Mask R–CNN, which had a mean sizing error of 10.7 mm. Furthermore, ORCNN had a significantly lower absolute sizing error on 161 heavily occluded broccoli heads with an occlusion rate between 50% and 90%. Our software and data set are available on https://git.wur.nl/blok012/sizecnn.

Greenhouse models are important tools for the analysis and design of greenhouse systems and for offering decision support to growers. While many models are available, relatively few include the influence of supplementary lighting on the greenhouse climate and crop. This study presents GreenLight, a model for greenhouses with supplemental lighting. GreenLight extends state of the art models by describing the qualitative difference between the common lighting system of high-pressure sodium (HPS) lamps, and the newest technology for horticultural lighting - the light-emitting diodes (LEDs). LEDs differ from HPS lamps in that they operate at lower temperatures, emit mostly convective heat and relatively little radiative heat, and can be more efficient in converting electricity to photosynthetically active radiation (PAR). These differences can have major implications on the greenhouse climate and operation, and on the amount of heat that must be supplied from the greenhouse heating system. Model predictions have been evaluated against data collected in greenhouse compartments equipped with HPS and LED lamps. The model predicted the greenhouse's heating needs with an error of 8–51 W m−2, representing 1–12% of the measured values; the RMSE for indoor temperature was 1.74–2.04 °C; and the RMSE for relative humidity was 5.52–8.5%. The model is freely available as open source MATLAB software at https://github.com/davkat1/GreenLight. It is hoped that it may be further evaluated and used by researchers worldwide to analyse the influence of the most recent lighting technologies on greenhouse climate control.

The possibility of improving the accuracy of climate monitoring in greenhouse cultivation by way of model based filtering was explored. The focus was on estimating the average climate inside a greenhouse compartment. Starting point was employing an extended Kalman filter (EKF), combined with a greenhouse climate differential equation model. In two different greenhouses (A and B), temperature and humidity were monitored with a 5-min sampling resolution with a sensor grid. The available data sets spanned 1 and 0.5 years. With the average over all sensors as reference signal, the root mean squared errors (RMSEs) of the unfiltered signals (coming from single sensors) were 0.43 °C and 0.48 g m −3 for greenhouse A, and 0.80 °C and 0.64 g m −3 for greenhouse B. The filter was compared with a moving average (MA) filter, and an unscented Kalman filter (UKF). Overall, monitoring accuracy was not improved by any of the filters, and in most cases it deteriorated. Performance was strongly linked to the choice of filter, where the EKF outperformed the other filters by a considerable difference. The violations on the assumptions of whiteness and normality of the noise were severe but had a moderate effect on the RMSEs (0.11 °C and 0.10 g m −3 for greenhouse A). A clear link was found between model accuracy and monitoring accuracy. A 10–15 fold decrease of state errors was associated with an RMSE reduction down to 0.1 °C and 0.1 g m −3 , the expected equivalent of increasing the number of climate sensors from 1 to 25.

Crop modelling is an essential part of biosystems engineering; selecting or developing a crop model for a specific application, having its requirements and desires, is difficult if not impossible without the required domain knowledge. This paper presents a fundamentally different model selection approach based on biological functionalities. This is enabled by a common structure, which allows for a combining of components, yielding new models. This increased design space allows the development of models which are better suited to the application than the original models. The use of a common structure, and its potential, are demonstrated by a use-case involving the selection of a tomato crop model for a model-based control application, but the rationales and methodologies can apply to other crops and applications as well. In this paper, 27 valid model combinations have been created from 4 models. In the use-case presented, the models are compared to data originating from a real system. The predictive performance of a model is quantified by the Root-Mean-Squared-Error (RSME) between the predictions and data. One trade-off is model accuracy versus computational speed. With the model set used, a 13% decrease in RSME was obtained by allowing a 7.5% increase in model computation time compared to one of the original models.

One of the most important steps in vision-based weed detection systems is the classification of weeds growing amongst crops. In the EU SmartBot project it was required to effectively control more than 95% of volunteer potatoes and ensure less than 5% of damage of sugar beet. Classification features such as colour, shape and texture have been used individually or in combination for classification studies but they have proved unable to reach the required classification accuracy under natural and varying daylight conditions. A classification algorithm was developed using a Bag-of-Visual-Words (BoVW) model based on Scale-Invariant Feature Transform (SIFT) or Speeded Up Robust Feature (SURF) features with crop row information in the form of the Out-of-Row Regional Index (ORRI). The highest classification accuracy (96.5% with zero false-negatives) was obtained using SIFT and ORRI with Support Vector Machine (SVM) which is considerably better than previously reported research although its 7% false-positives deviated from the requirements. The average classification time of 0.10–0.11 s met the real-time requirements. The SIFT descriptor showed better classification accuracy than the SURF, but classification time did not vary significantly. Adding location information (ORRI) significantly improved overall classification accuracy. SVM showed better classification performance than random forest and neural network. The proposed approach proved its potential under varying natural light conditions, but implementing a practical system, including vegetation segmentation and weed removal may potentially reduce the overall performance and more research is needed.

Assessment of animal status, housing conditions and manually collecting floor eggs are the major daily tasks for poultry farmers. To assist the farmer in these tasks, PoultryBot, an autonomous mobile robot for use in poultry houses has been developed. In earlier research, several components of PoultryBot were discussed in detail. Here, performance of the robot is evaluated under practical conditions. For navigation, different paths were used to assess its navigation performance for various tasks, such as area sweeping and surveying close to walls. PoultryBot proved capable of navigating autonomously more than 3000 m, while avoiding obstacles and dealing with the hens present. The robustness of its navigation performance was tested by confronting PoultryBot with obstacles in different positions with respect to its path and using different settings of the navigation parameters. Both factors clearly influenced the driving behaviour of PoultryBot. For floor egg collection, detection and collection of eggs was assessed at 5 predefined egg positions lateral to the path of the robot. Over 300 eggs were tested; 46% were collected successfully, 37% was not collected successfully, and 16% were missed. The most observed failures occurred when the collection device was just next to the egg. It is thought that this problem can be solved by improving the control algorithm. The results demonstrate the validity of the PoultryBot concept and the possibility of autonomous floor egg collection in commercial poultry houses. Furthermore, they indicate that application of smart autonomous vehicles in dense animal environments is feasible.