Search

Multi-object grasping is a challenging task. It is important for energy and cost-efficient operation of industrial crane manipulators, such as those used to collect tree logs off the forest floor and onto forest machines. In this work, we used synthetic data from physics simulations to explore how data-driven modeling can be used to infer multi-object grasp poses from images. We showed that convolutional neural networks can be trained specifically for synthesizing multi-object grasps. Using RGB-Depth images and instance segmentation masks as input, a U-Net model outputs grasp maps with corresponding grapple orientation and opening width. Given an observation of a pile of logs, the model can be used to synthesize and rate the possible grasp poses and select the most suitable one, with the possibility to respect changing operational constraints such as lift capacity and reach. When tested on previously unseen data, the proposed model found successful grasp poses with an accuracy of 95%.

We investigate how well a physics-based simulator can replicate a real wheel loader performing bucket filling in a pile of soil. The comparison is made using field test time series of the vehicle motion and actuation forces, loaded mass, and total work. The vehicle was modeled as a rigid multibody system with frictional contacts, driveline, and linear actuators. For the soil, we tested discrete element models of different resolutions, with and without multiscale acceleration. The spatio-temporal resolution ranged between 50-400 mm and 2-500 ms, and the computational speed was between 1/10,000 to 5 times faster than real-time. The simulation-to-reality gap was found to be around 10% and exhibited a weak dependence on the level of fidelity, e.g., compatible with real-time simulation. Furthermore, the sensitivity of an optimized force feedback controller under transfer between different simulation domains was investigated. The domain bias was observed to cause a performance reduction of 5% despite the domain gap being about 15%., Comment: 22 pages, 15 figures

This paper presents a method for learning world models for wheel loaders performing automatic loading actions on a pile of soil. Data-driven models were learned to output the resulting pile state, loaded mass, time, and work for a single loading cycle given inputs that include a heightmap of the initial pile shape and action parameters for an automatic bucket-filling controller. Long-horizon planning of sequential loading in a dynamically changing environment is thus enabled as repeated model inference. The models, consisting of deep neural networks, were trained on data from 3D multibody dynamics simulation of over 10,000 random loading actions in gravel piles of different shapes. The accuracy and inference time for predicting the loading performance and the resulting pile state were, on average, 95% in 1.2 ms and 97% in 4.5 ms, respectively. Long-horizon predictions were found feasible over 40 sequential loading actions., Comment: 23 pages, 19 figures

We explore multi-log grasping using reinforcement learning and virtual visual servoing for automated forwarding in a simulated environment. Automation of forest processes is a major challenge, and many techniques regarding robot control pose different challenges due to the unstructured and harsh outdoor environment. Grasping multiple logs involves various problems of dynamics and path planning, where understanding the interaction between the grapple, logs, terrain, and obstacles requires visual information. To address these challenges, we separate image segmentation from crane control and utilise a virtual camera to provide an image stream from reconstructed 3D data. We use Cartesian control to simplify domain transfer to real-world applications. Since log piles are static, visual servoing using a 3D reconstruction of the pile and its surroundings is equivalent to using real camera data until the point of grasping. This relaxes the limits on computational resources and time for the challenge of image segmentation and allows for collecting data in situations where the log piles are not occluded. The disadvantage is the lack of information during grasping. We demonstrate that this problem is manageable and present an agent that is 95% successful in picking one or several logs from challenging piles of 2--5 logs., Comment: 8 pages, 10 figures

We explore sim-to-real transfer of deep reinforcement learning controllers for a heavy vehicle with active suspensions designed for traversing rough terrain. While related research primarily focuses on lightweight robots with electric motors and fast actuation, this study uses a forestry vehicle with a complex hydraulic driveline and slow actuation. We simulate the vehicle using multibody dynamics and apply system identification to find an appropriate set of simulation parameters. We then train policies in simulation using various techniques to mitigate the sim-to-real gap, including domain randomization, action delays, and a reward penalty to encourage smooth control. In reality, the policies trained with action delays and a penalty for erratic actions perform nearly at the same level as in simulation. In experiments on level ground, the motion trajectories closely overlap when turning to either side, as well as in a route tracking scenario. When faced with a ramp that requires active use of the suspensions, the simulated and real motions are in close alignment. This shows that the actuator model together with system identification yields a sufficiently accurate model of the actuators. We observe that policies trained without the additional action penalty exhibit fast switching or bang-bang control. These present smooth motions and high performance in simulation but transfer poorly to reality. We find that policies make marginal use of the local height map for perception, showing no indications of predictive planning. However, the strong transfer capabilities entail that further development concerning perception and performance can be largely confined to simulation., Comment: 15 pages, 18 figures

We present a method that uses high-resolution topography data of rough terrain, and ground vehicle simulation, to predict traversability. Traversability is expressed as three independent measures: the ability to traverse the terrain at a target speed, energy consumption, and acceleration. The measures are continuous and reflect different objectives for planning that go beyond binary classification. A deep neural network is trained to predict the traversability measures from the local heightmap and target speed. To produce training data, we use an articulated vehicle with wheeled bogie suspensions and procedurally generated terrains. We evaluate the model on laser-scanned forest terrains, previously unseen by the model. The model predicts traversability with an accuracy of 90%. Predictions rely on features from the high-dimensional terrain data that surpass local roughness and slope relative to the heading. Correlations show that the three traversability measures are complementary to each other. With an inference speed 3000 times faster than the ground truth simulation and trivially parallelizable, the model is well suited for traversability analysis and optimal path planning over large areas., Comment: 11 pages, 17 figures, 2 tables

Having smart and autonomous earthmoving in mind, we explore high-performance wheel loading in a simulated environment. This paper introduces a wheel loader simulator that combines contacting 3D multibody dynamics with a hybrid continuum-particle terrain model, supporting realistic digging forces and soil displacements at real-time performance. A total of 270,000 simulations are run with different loading actions, pile slopes, and soil to analyze how they affect the loading performance. The results suggest that the preferred digging actions should preserve and exploit a steep pile slope. High digging speed favors high productivity, while energy-efficient loading requires a lower dig speed., Comment: 8 pages, 9 figures, 3 tables

We explore the potential to control terrain vehicles using deep reinforcement in scenarios where human operators and traditional control methods are inadequate. This letter presents a controller that perceives, plans, and successfully controls a 16-tonne forestry vehicle with two frame articulation joints, six wheels, and their actively articulated suspensions to traverse rough terrain. The carefully shaped reward signal promotes safe, environmental, and efficient driving, which leads to the emergence of unprecedented driving skills. We test learned skills in a virtual environment, including terrains reconstructed from high-density laser scans of forest sites. The controller displays the ability to handle obstructing obstacles, slopes up to 27$^\circ$, and a variety of natural terrains, all with limited wheel slip, smooth, and upright traversal with intelligent use of the active suspensions. The results confirm that deep reinforcement learning has the potential to enhance control of vehicles with complex dynamics and high-dimensional observation data compared to human operators or traditional control methods, especially in rough terrain., Comment: 16 pages, 13 figures

Systems for transport and processing of granular media are challenging to analyse, operate and optimise. In the mining and mineral processing industries these systems are chains of processes with complex interplay between the equipment, control, and the processed material. The material properties have natural variations that are usually only known at certain locations. Therefore, we explore a material-oriented approach to digital twins with a particle representation of the granular media. In digital form, the material is treated as pseudo-particles, each representing a large collection of real particles of various sizes, shapes and, mineral properties. Movements and changes in the state of the material are determined by the combined data from control systems, sensors, vehicle telematics, and simulation models at locations where no real sensors can see. The particle-based representation enables material tracking along the chain of processes. Each digital particle can act as a carrier of observational data generated by the equipment as it interacts with the real material. This makes it possible to better learn material properties from process observations, and to predict the effect on downstream processes. We test the technique on a mining simulator and demonstrate analysis that can be performed using data from cross-system material tracking., Comment: 19 pages, 16 figures

Forestry machines are heavy vehicles performing complex manipulation tasks in unstructured production forest environments. Together with the complex dynamics of the on-board hydraulically actuated cranes, the rough forest terrains have posed a particular challenge in forestry automation. In this study, the feasibility of applying reinforcement learning control to forestry crane manipulators is investigated in a simulated environment. Our results show that it is possible to learn successful actuator-space control policies for energy efficient log grasping by invoking a simple curriculum in a deep reinforcement learning setup. Given the pose of the selected logs, our best control policy reaches a grasping success rate of 97%. Including an energy-optimization goal in the reward function, the energy consumption is significantly reduced compared to control policies learned without incentive for energy optimization, while the increase in cycle time is marginal. The energy-optimization effects can be observed in the overall smoother motion and acceleration profiles during crane manipulation., Comment: 8 pages, 6 figures

Reinforcement learning control of an underground loader is investigated in simulated environment, using a multi-agent deep neural network approach. At the start of each loading cycle, one agent selects the dig position from a depth camera image of the pile of fragmented rock. A second agent is responsible for continuous control of the vehicle, with the goal of filling the bucket at the selected loading point, while avoiding collisions, getting stuck, or losing ground traction. It relies on motion and force sensors, as well as on camera and lidar. Using a soft actor-critic algorithm the agents learn policies for efficient bucket filling over many subsequent loading cycles, with clear ability to adapt to the changing environment. The best results, on average 75% of the max capacity, are obtained when including a penalty for energy usage in the reward., Comment: 11 pages, 7 figures

This paper presents static object detection and segmentation method in videos from cluttered scenes. Robust static object detection is still challenging task due to presence of moving objects in many surveillance applications. The level of difficulty is extremely influenced by on how you label the object to be identified as static that do not establish the original background but appeared in the video at different time. In this context, background subtraction technique based on the frame difference concept is applied to the identification of static objects. Firstly, we estimate a frame differencing foreground mask image by computing the difference of each frame with respect to a static reference frame. The Mixture of Gaussian MOG method is applied to detect the moving particles and then outcome foreground mask is subtracted from frame differencing foreground mask. Pre-processing techniques, illumination equalization and de-hazing methods are applied to handle low contrast and to reduce the noise from scattered materials in the air e.g. water droplets and dust particles. Finally, a set of mathematical morphological operation and largest connected-component analysis is applied to segment the object and suppress the noise. The proposed method was built for rock breaker station application and effectively validated with real, synthetic and two public data sets. The results demonstrate the proposed approach can robustly detect, segmented the static objects without any prior information of tracking., Comment: 9 pages, 14 figures

This paper presents a multi stage 3D shape reconstruction system of multiple object scenes by considering the silhouette inconsistencies in shape-from silhouette SFS method. These inconsistencies are common in multiple view images due to object occlusions in different views, segmentation and shadows or reflection due to objects or light directions. These factors raise huge challenges when attempting to construct the 3D shape by using existing approaches which reconstruct only that part of the volume which projects consistently in all the silhouettes, leaving the rest unreconstructed. As a result, final shape are not robust due to multi view objects occlusion and shadows. In this regard, we consider the primary factors affecting reconstruction by analyzing the multiple images and perform pre-processing steps to optimize the silhouettes. Finally, the 3D shape is reconstructed by using the volumetric approach SFS. Theory and experimental results show that, the performance of the modified algorithm was efficiently improved, which can improve the accuracy of the reconstructed shape and being robust to errors in the silhouettes, volume and computational inexpensive., Comment: 9 pages, 11 figures

A multiscale model for real-time simulation of terrain dynamics is explored. To represent the dynamics on different scales the model combines the description of soil as a continuous solid, as distinct particles and as rigid multibodies. The models are dynamically coupled to each other and to the earthmoving equipment. Agitated soil is represented by a hybrid of contacting particles and continuum solid, with the moving equipment and resting soil as geometric boundaries. Each zone of active soil is aggregated into distinct bodies, with the proper mass, momentum and frictional-cohesive properties, which constrain the equipment's multibody dynamics. The particle model parameters are pre-calibrated to the bulk mechanical parameters for a wide range of different soils. The result is a computationally efficient model for earthmoving operations that resolve the motion of the soil, using a fast iterative solver, and provide realistic forces and dynamic for the equipment, using a direct solver for high numerical precision. Numerical simulations of excavation and bulldozing operations are performed to validate the model and measure the computational performance. Reference data is produced using coupled discrete element and multibody dynamics simulations at relatively high resolution. The digging resistance and soil displacements with the real-time multiscale model agree with the reference model up to 10-25%, and run more than three orders of magnitude faster., Comment: 35 pages, 24 figures

We investigate the use of reduced-order modelling to run discrete element simulations at higher speeds. Taking a data-driven approach, we run many offline simulations in advance and train a model to predict the velocity field from the mass distribution and system control signals. Rapid model inference of particle velocities replaces the intense process of computing contact forces and velocity updates. In coupled DEM and multibody system simulation the predictor model can be trained to output the interfacial reaction forces as well. An adaptive model order reduction technique is investigated, decomposing the media in domains of solid, liquid, and gaseous state. The model reduction is applied to solid and liquid domains where the particle motion is strongly correlated with the mean flow, while resolved DEM is used for gaseous domains. Using a ridge regression predictor, the performance is tested on simulations of a pile discharge and bulldozing. The measured accuracy is about 90% and 65%, respectively, and the speed-up range between 10 and 60., Comment: 17 pages, 15 figures

A method for simulation of elastoplastic solids in multibody systems with nonsmooth and multidomain dynamics is developed. The solid is discretised into pseudo-particles using the meshfree moving least squares method. The particles carry strain and stress tensor variables that are mapped to deformation constraints and constraint forces. The discretised solid model thus fit a unified framework for nonsmooth multidomain dynamics for realtime simulations including strong coupling of rigid multibodies with complex kinematic constraints such as articulation joints, unilateral contacts with dry friction, drivelines and hydraulics. The nonsmooth formulation allow for impulses, due to impacts for instance, to propagate instantly between the rigid multibody and the solid. Plasticity is introduced through an associative perfectly plastic modified Drucker-Prager model. The elastic and plastic dynamics is verified for simple test systems and the capability of simulating tracked terrain vehicles driving on a deformable terrain is demonstrated., Comment: 26 pages, 12 figures

A method for adaptive model order reduction for nonsmooth discrete element simulation is developed and analysed in numerical experiments. Regions of the granular media that collectively move as rigid bodies are substituted with rigid bodies of the corresponding shape and mass distribution. The method also support particles merging with articulated multibody systems. A model approximation error is defined and used to derive conditions for when and where to apply reduction and refinement back into particles and smaller rigid bodies. Three methods for refinement are proposed and tested: prediction from contact events, trial solutions computed in the background and using split sensors. The computational performance can be increased by 5 - 50 times for model reduction level between 70 - 95%., Comment: 28 pages, 14 figures

The effect on the convergence of warm starting the projected Gauss-Seidel solver for nonsmooth discrete element simulation of granular matter are investigated. It is found that the computational performance can be increased by a factor 2 to 5., Comment: 10 pages, 18 figures

We present a coarse grained rigid multibody model of a subunit assembled helix-like polymer, e.g., adhesion pili expressed by bacteria, that is capable of describing the polymers force-extension response. With building blocks representing individual subunits the model appropriately describes the complex behavior of pili expressed by the gram-negative uropathogenic Escherichia coli bacteria under the action of an external force. Numerical simulations show that the dynamics of the model, which include both the effects of unwinding and rewinding, are in good quantitative agreement with the characteristic force-extension response as observed experimentally for type 1 and P pili. By tuning the model, it is also possible to reproduce the force-extension response in the presence of anti-shaft antibodies, which dramatically changes the mechanical properties. Thus, the model and the results in this work give enhanced understanding of how a pilus unwinds under action of external forces and provide new perspective of the complex bacterial adhesion processes., Comment: 25 pages, 5 figures

In magnetized plasmas gravitational and electromagnetic waves may interact coherently and exchange energy between themselves and with plasma flows. We derive the wave interaction equations for these processes in the case of waves propagating perpendicular or parallel to the plasma background magnetic field. In the latter case, the electromagnetic waves are taken to be circularly polarized waves of arbitrary amplitude. We allow for a background drift flow of the plasma components which increases the number of possible evolution scenarios. The interaction equations are solved analytically and the characteristic time scales for conversion between gravitational and electromagnetic waves are found. In particular, it is shown that in the presence of a drift flow there are explosive instabilities resulting in the generation of gravitational and electromagnetic waves. Conversely, we show that energetic waves can interact to accelerate particles and thereby \emph{produce} a drift flow. The relevance of these results for astrophysical and cosmological plasmas is discussed., Comment: 12 pages, 1 figure, typos corrected and numerical example added

The exact 1+3 covariant dynamical fluid equations for a multi-component plasma, together with Maxwell's equations are presented in such a way as to make them suitable for a gauge-invariant analysis of linear density and velocity perturbations of the Friedmann-Robertson-Walker model. In the case where the matter is described by a two component plasma where thermal effects are neglected, a mode representing high-frequency plasma oscillations is found in addition to the standard growing and decaying gravitational instability picture. Further applications of these equations are also discussed., Comment: 14 pages (example added), to appear in Class. Quantum Grav

Generation of wake fields by a short electromagnetic pulse in a plasma with an inhomogeneous background magnetic field and density profile is considered, and a wave equation is derived. Transmission and reflection coefficients are calculated in a medium with sharp discontinuities. Particular attention is focused on examples where the longitudinal part of the electromagnetic field is amplified for the transmitted wave. Furthermore, it is noted that the wake field can propagate out of the plasma and thereby provide information about the electron density profile. A method for reconstructing the background density profile from a measured wake field spectrum is proposed and a numerical example is given., Comment: 12 pages in LaTeX style, 11 eps figures

We study the propagation of gravitational waves in a collisionless plasma with an external magnetic field parallel to the direction of propagation. Due to resonant interaction with the plasma particles the gravitational wave experiences cyclotron damping or growth, the latter case being possible if the distribution function for any of the particle species deviates from thermodynamical equilibrium. Furthermore, we examine how the damping and dispersion depends on temperature and on the ratio between the cyclotron- and gravitational wave frequency. The presence of the magnetic field leads to different dispersion relations for different polarizations, which in turn imply Faraday rotation of gravitational waves., Comment: 15 pages, 3 figures. Accepted for publication in Phys. Rev. D

In the presence of an external magnetic field, wake fields generated by a short laser pulse can propagate out of the plasma, and thereby provide information about the electron density profile. A method for reconstructing the density profile from a measured wake field spectrum is proposed and a numerical example is given. Finally, we compare our proposal with existing plasma density diagnostic techniques., Comment: 5 pages, 2 figures