Search

We present a framework for identifying, communicating, and addressing risk in shared-autonomy mobile manipulator applications. This framework is centered on the capacity of the mobile manipulator to sense its environment, interpret complex and cluttered scenes, and estimate the probability of actions and configuration states that may result in task failures, such as collision (i.e., identifying “risk”). If the threshold for acceptable risk is exceeded, a remote operator is notified and presented with timely, actionable information in which the person can quickly assess the situation and provide guidance for the robot. This framework is demonstrated with a use case in which a mobile manipulator performs machine tending and material handling tasks.

Conventional material extrusion additive manufacturing (AM) processes require the user to make a trade-off between surface quality and build time of the part. A large bead filament deposition can speed up the build process; however, it leads to surfaces with high roughness due to the stair-stepping effect. The surface quality can be improved by using a small bead filament deposition, which increases the build time of the part. We present a new approach incorporating hybrid multi-resolution layers in material extrusion additive manufacturing to provide excellent surface quality without increasing the build time. Our slicing algorithm generates planar layers with large filaments to fill the interior regions in less time. The generated exterior layers are conformal and use small filaments to reduce the stair-stepping effect and improve surface quality. We also present a path planning algorithm to build parts with a single manipulator using a multi-nozzle extrusion tool. The path planning algorithm generates a smooth material deposition path by avoiding collision between the tool and the already built layers. It reduces the collision checks and performs collision detection in a computationally efficient manner. We build five parts to validate our approach and illustrate the benefits of multi-resolution AM.

3D reconstruction technology is used in a wide variety of applications. Currently, automatically creating accurate pointclouds for large parts requires expensive hardware. We are interested in using low-cost depth cameras mounted on commonly available industrial robots to create accurate pointclouds for large parts automatically. Manufacturing applications require fast cycle times. Therefore, we are interested in speeding up the 3D reconstruction process. We present algorithmic advances in 3D reconstruction that achieve a sub-millimeter accuracy using a low-cost depth camera. Our system can be used to determine a pointcloud model of large and complex parts. Advances in camera calibration, cycle time reduction for pointcloud capturing, and uncertainty estimation are made in this work. We continuously capture point-clouds at an optimal camera location with respect to part distance during robot motion execution. The redundancy in pointclouds achieved by the moving camera significantly reduces errors in measurements without increasing cycle time. Our system produces sub-millimeter accuracy.

Automatically detecting surface defects from images is an essential capability in manufacturing applications. Traditional image processing techniques are useful in solving a specific class of problems. However, these techniques do not handle noise, variations in lighting conditions, and backgrounds with complex textures. In recent times, deep learning has been widely explored for use in automation of defect detection. This survey article presents three different ways of classifying various efforts in literature for surface defect detection using deep learning techniques. These three ways are based on defect detection context, learning techniques, and defect localization and classification method respectively. This article also identifies future research directions based on the trends in the deep learning area.

Tool-Path planning is the foundation for automating many manufacturing processes. Robotic manipulators are increasingly being considered to automate tasks that require complex tool motions. Robotic manipulators provide extra degrees of freedom and are more flexible than traditional automation technologies. However, a tool-path needs to be planned and given to the manipulator trajectory generator as input. The traditional tool-path planning considers the tool to have one contact point, known as the Tool Center Point (TCP). This underutilizes the available flexibility of the manipulator. To make use of the manipulator’s flexibility, multiple contact points or multiple TCPs can be considered. These tool contact considerations make the tool path planning problem complex and computationally challenging. In this paper, we present a novel tool path planning algorithm. Our algorithm incorporates the multiple tool contact points consideration during tool-path planning in an efficient manner to generate a high-quality tool-path in a reasonable amount of run time. We evaluated our algorithm on four different test cases. The test cases include the parts with complex geometry and tools which can have surface area for establishing contacts with the part. In these test cases, a traditional single contact tool-path will not work due to geometric constraints, and multiple tool contacts make the problem challenging. Our algorithm was able to generate tool-paths for these four test cases successfully.

Multi-arm mobile manipulators can be represented as a combination of multiple robotic agents from the perspective of task-assignment and motion planning. Depending upon the task, agents might collaborate or work independently. Integrating motion planning with task-agent assignment is a computationally slow process as infeasible assignments can only be detected through expensive motion planning queries. We present three speed-up techniques for addressing this problem-(1) spatial constraint checking using conservative surrogates for motion planners, (2) instantiating symbolic conditions for pruning infeasible assignments, and (3) efficiently caching and reusing previously generated motion plans. We show that the developed method is useful for real-world operations that require complex interaction and coordination among high-DOF robotic agents.

We present a context-dependent bi-directional tree-search framework for point-to-point path planning for manipulators. Conceptually, our framework is composed of six modules: tree selection, focus selection, node selection, target selection, extend selection and connection type selection. Each module consists of a set of interchangeable strategies. By exploiting synergistic interaction between these strategies and selecting appropriate strategies based the contextual cues from the search state, we show an instance of our framework that computes high-quality solutions in a variety of complex scenarios with a low failure rate. We also show that some popular path planning methods in the literature can be easily represented in our framework. We compare our approach with these popular methods in a diverse set of test scenarios. We report a 15-fold reduction in failure rate coupled with at least a 26% drop in solution suboptimality when compared to the best of the alternative methods.

We present a two-layered architecture for task-agent assignment and motion planning of Bi-manual Mobile manipulators for executing complex tasks. We use search trees in temporal windows to determine feasible task assignments of agents using task and spatial constraint-based heuristics. We also introduce a caching scheme for moving between different trees so as to avoid re-planning of those portions of the robot motion that were successful. This greatly reduces the number of calls to the motion planner as compared to direct motion planning after task-agent assignment. We have shown our approach works on a set of complex tasks with significantly lower computation times.

Low production volume applications cannot use custom fixtures to reduce uncertainties in part locations. Traditional tool-path planners for area coverage applications may not perform satisfactorily when there is significant uncertainty in part locations because these planners do not account for the tool-part contact and impedance controller parameters during path execution. This may lead to tool collision with the part, incorrect force at the Tool Center Point (TCP), and loss of contact while performing area coverage operations. Moreover, the poorly designed tool-path and incorrectly estimated impedance control parameters also lead to poor task performance. To avoid such failures, we concurrently design a collision-free tool-paths, generate executable robot trajectory, and select the impedance compensation parameters so that it can take care of part uncertainties and provide the desired tool-part contact. We apply the designed algorithm for area coverage manufacturing applications. Finally, we evaluate the algorithm performance by conducting physical experiments.

High-performance composites are widely used in industry because of specific mechanical properties and lightweighting opportunities. Current automation solutions to manufacturing components from prepreg (pre-impregnated precursor material) sheets are limited. Our previous work has demonstrated the technical feasibility of a robotic cell to automate the sheet layup process. Many decisions are required for the cell to function correctly, and the time necessary to make these decisions must be reduced to utilize the cell effectively. Robot placement with respect to the mold is a significant and complex decision problem. Ensuring that robots can collaborate effectively requires addressing multiple constraints related to the robot workspace, singularity, and velocities. Solving this problem requires developing computationally efficient algorithms to find feasible robot placements in the cell. We describe an approach based on successive solution refinement strategy to identify a cell design that satisfies all constraints related to robot placement.

Successfully completing a complex manufacturing task requires finding a feasible placement of the workpiece in the robot workspace. The workpiece placement should be such that the task surfaces on the workpiece are reachable by the robot, the robot can apply the required forces, and the end-effector/tool can move with the desired velocity. This paper formulates the problem of identifying a feasible placement as a non-linear optimization problem over the constraint violation functions. This is a computationally challenging problem. We show that this problem can be solved by successively searching for the solution by incrementally applying different constraints. We demonstrate the feasibility of our approach using several complex workpieces.

Extrusion-based additive manufacturing (AM), also known as fused deposition modeling (FDM) extrudes filaments through a heated nozzle and builds a part layer-by-layer. Using a smaller diameter nozzle can achieve better surface finish. However, there is a trade-off between surface finish and build times as using a small diameter nozzle leads to smaller layer thickness and long build times. Traditional FDM printers create a part with planar layers, and this restricts control over fiber orientations. This paper presents a robotic cell for multi-resolution AM. The cell consists of two 6 degrees of freedom (DOF) robot manipulators capable of printing non-planar and/or planar layers. We describe algorithms for decomposing parts into multi-resolution layers and generating collision-free trajectories for the robot manipulators. We validate our approach by printing five parts with multi-resolution.

We pose the problem of path-constrained trajectory generation for the synchronous motion of multi-robot systems as a non-linear optimization problem. Our method determines appropriate parametric representation for the configuration variables, generates an approximate solution as a starting point for the optimization method, and uses successive refinement techniques to solve the problem in a computationally efficient manner. We have demonstrated the effectiveness of the proposed method on challenging simulation and physical experiments with high degrees of freedom robotic systems.

Hand layup is a commonly used process for making composite structures from several plies of carbon-fiber prepreg. The process involves multiple human operators manipulating and conforming layers of prepreg to a mold. The manual layup process is ergonomically challenging, tedious, and limits throughput. Moreover, different operators may perform the process differently, and hence introduce inconsistency. We have developed a multi-robot cell to automate the layup process. A human expert provides a sequence to conform to the ply and types of end-effectors to be used as input to the system. The system automatically generates trajectories for the robots that can achieve the specified layup. Using the cell requires the automated generation of different types of plans. This paper addresses two main planning problems: (a) generating plans to grasp and manipulate the ply and (b) generating feasible robot trajectories. We use a hybrid-physics based simulator coupled with a state space search to find grasp plans. The system employs a strategy that applies constraints successively in a non-linear optimization formulation to identify suitable placements of the robots around the mold so that feasible trajectories can be generated. Our system can generate plans in a computationally efficient manner, and it can handle a wide variety of complex parts. We demonstrate the automated layup by conducting physical experiments on an industry-inspired mold using the generated plans.

Many practical operations like search and rescue operations are time-sensitive missions. In this paper, we present a wave-aware deliberative planner that avoids collisions with traffic vessels and also opportunistically traverses wavefields generated by these vessels while minimizing the execution time and the risk of failure. The planner performs a search over a 4D pose-time lattice to generate a collision-free, minimum-risk sequence of motion goals. It addresses motion uncertainty, failure risk and perception uncertainty. We also present heuristics and adaptive motion goals to speed-up computation. We present simulation results showing that our wave-aware planner produces plans that execute faster compared to the typical reactive planning scheme. We also present results from physical experiments showing the feasibility of the proposed approach.

This paper presents a new trajectory planning algorithm based on context-dependent search strategy switching for manipulators working in confined workspaces. This problem has an inherently complex and large search space. Finding good solutions to this problem requires developing an efficient search strategy using appropriately-designed motion primitives and heuristic functions to guide the search. The algorithm presented in this paper monitors the search progress and uses different search strategies in different parts of the search space. This allows it to solve manipulator trajectory planning problems for highly-confined workspaces. It is also able to seek input from the human user when search heuristics are unable to provide guidance to the search process. We present results on challenging example parts and compare them with the previously reported approaches.

Warehouses and factories are beginning to deploy mobile manipulators for transporting parts between machines and work stations. Minimizing the number of these robots on the shop floor requires that each robot should complete the assigned task in a time-optimal manner and therefore maximize the robot's capacity. In this paper we present a search-based algorithm for generating time-optimal trajectories for picking and transporting parts using a mobile manipulator. The approach is based on a hierarchical search that uses discrete search for planning of the mobile base. The time cost due to the motion of the manipulator for picking up the part is evaluated and integrated with the search state space at the appropriate times. The trajectories generated result in picking up of the part with the manipulator while the mobile base is in motion. The performance of the algorithm is studied for different factory sizes and obstacle densities. This approach enables the mobile manipulator to perform pick and transport operation in a smaller time duration compared to the operation where it stops to pick up the object.

We propose a hybrid human-robot cell for reducing labor in multi-layer prepreg composite sheet layup process. The human expert performs initial experiments to assess the process sequence, and the direction of the layup. The expert knowledge is then transferred to the system in the form of high level commands which are converted into low-level collision free trajectories for the robots in the cell. This paper demonstrates the feasibility of the composite sheet layup using the robotic manipulators along with the design of end-effector tools that are used during the layup process. We give a comparison between manual and hybrid cell for execution times in different steps involved. Finally, we give an overview of process consistency benefits of the hybrid cell.

Using fixtures for assembly operations is a common practice in manufacturing processes with high production volume. For automated assembly cells using robotic arms, trajectories are programmed manually and robots follow the same path repeatedly. It is not economically feasible to build fixed fixtures for small volume productions as they require high accuracy and are part specific. Moreover, hand coding robot trajectories is a time consuming task. The uncertainties in part localization and inaccuracy in robot motions make it challenging to automate the task of assembling two parts with tight tolerances. Researchers in past have developed methods for automating the assembly task using contact-based search schemes and impedance control-based trajectory execution. Both of these approaches may lead to undesired collision with critical features on the parts. Our method guarantees safety for parts with delicate features during the assembly process. Our approach enables us to select optimum impedance control parameters and utilizes a learning-based search strategy to complete assembly tasks under uncertainties in bounded time. Our approach was tested on an assembly of two rectangular workpieces using KUKA IIWA 7 manipulator. The method we propose was able to successfully select the optimal control parameters. The learning-based search strategy successfully estimated the uncertainty in pose of parts and converged in few iterations.

Surface finishing is an important manufacturing process. Many parts with complex geometries require finishing of internal regions before they can be used. In small and medium volume productions most of the finishing tasks are non-repetitive in nature, and have to be performed manually. These finishing operations for parts with complex geometries can be quite labor intensive, and may pose risk to humans. We have developed a collaborative finishing system where human operators work on high level decision making, and the robot assistants carry out the labor intensive low level finishing tasks. The human operator guides the robotic system by transferring operator knowledge through a user interface. Our system generates instructions for the robots based on the user inputs and task requirements. We have also developed a planning algorithm that automatically computes the paths for the robots by using the CAD model of the part. This significantly reduces the robot programming time and improves the efficiency of the finishing system. If needed, the system seeks help from the human operator by generating notifications.

This paper introduces an approach for decomposing exploration tasks among multiple unmanned surface vehicles (USVs) in congested regions. In order to ensure effective distribution of the workload, the algorithm has to consider the effects of the environmental constraints on the USVs. The performance of a USV is influenced by the surface currents, risk of collision with the civilian traffic, and varying depths due to tides and weather. The team of USVs needs to explore a certain region of the harbor and we need to develop an algorithm to decompose the region of interest into multiple subregions. The algorithm overlays a two-dimensional grid upon a given map to convert it to an occupancy grid, and then proceeds to partition the region of interest among the multiple USVs assigned to explore the region. During partitioning, the rate at which each USV is able to travel varies with the applicable speed limits at the location. The objective is to minimize the time taken for the last USV to finish exploring the assigned area. We use the particle swarm optimization (PSO) method to compute the optimal region partitions. The method is verified by running simulations in different test environments. We also analyze the performance of the developed method in environments where speed restrictions are not known in advance.

The growing variety and complexity of marine research and application oriented tasks requires unmanned surface vehicles (USVs) to operate fully autonomously over long time horizons even in environments with significant civilian traffic. In order to address this challenge, we have developed a lattice-based 5D trajectory planner for USVs. The planner estimates collision risk and reasons about the availability of contingency maneuvers to counteract unpredictable behaviors of civilian vessels. The planner also incorporates avoidance behaviors of the vessels into the search for a dynamically feasible trajectory to minimize collision risk. In order to be computationally efficient, it dynamically scales the control action primitives of a trajectory based on the distribution and concentration of civilian vessels while preserving the dynamical feasibility of the primitives. We present a novel congestion metric to compare the complexity of different scenarios when evaluating the performance of the planner. Our results demonstrate that the basic version of the risk and contingency-aware planner (RCAP) significantly decreases the number of collisions compared to a baseline, velocity obstacles based planner, especially in complex scenarios with a high number of civilian vessels. The adaptive version of the planner (A-RCAP) improves the computational performance of RCAP by 500 %. This leads to a high replanning rate, which allows shorter traversal distances and smaller arrival times, while ensuring comparable incidence of collisions.

The performance of four automatically-generated path planning behaviors is evaluated through field-testing. Experiments were conducted using a model-referenced trajectory planner, which was implemented on unmanned surface vehicles (USVs) of different size, thrust, and maneuverability characteristics. The planner combines a local search based on the Velocity Obstacles (VO) concept with a global, lattice-based search for a dynamically feasible trajectory (determined using models of the nonlinear dynamics, nonholonomic constraints, and low-level control of each system). Multiple low-level USV heading and speed controllers, varying in complexity from proportional control to nonlinear backstepping control, were tested with the planner. High planning performance is achieved by searching within a resolution-adaptive space of the USV׳s candidate motion goals. Sampling is constrained to ensure that the International Regulations for the Prevention of Collisions at Sea (COLREGs) are followed. Prior to on-water implementation, the system was fine-tuned in simulation. Four automatically-generated USV behaviors were demonstrated in on-water tests within a complex, static obstacle field: 1) Approach (USV approaches a fixed target); 2) Follow (USV approaches a moving target); 3) Single COLREGs-Compliant Crossing (USV approaches a fixed target while avoiding another vessel); and 4) Multiple COLREGs-Compliant Crossings (USV approaches a fixed target while avoiding an interference vessel multiple times).

In this paper, we introduce an approach for decomposing exploration tasks among multiple Unmanned Surface Vehicles (USVs) in port regions. In order to ensure effective distribution of the workload, the algorithm has to consider the effects of the environment on the physical constraints of the USVs. The performance of the USV is influenced by the surface currents, risk of collision with the civilian traffic, and varying depths as a result of tides, and weather. In our approach, we want the team of USVs to explore certain region of the harbor. The algorithm has to decompose the region of interest into multiple sub-regions by considering the maximum operating velocity of each USV in the given environmental conditions. The algorithm overlays a 2D grid upon a given map to convert it to an occupancy grid, and then proceeds to partition the region of interest among the multiple USVs assigned to explore the region. During partitioning, each USV covers the maximum area that is possible by operating at maximum velocity at each time-step. The objective is to minimize the time taken for the last USV to finish claiming its area exploration. We use the particle swarm optimization (PSO) method to compute the optimal region partitions. The method is verified by running simulations in different test environments. We also analyze the performance of the developed method in environments with unknown velocity profiles.

The capability of following a moving target in an environment with obstacles is required as a basic and necessary function for realizing an autonomous unmanned surface vehicle (USV). Many target following scenarios involve a follower and target vehicles that may have different maneuvering capabilities. Moreover, the follower vehicle may not have prior information about the intended motion of the target boat. This paper presents a trajectory planning and tracking approach for following a differentially constrained target vehicle operating in an obstacle field. The developed approach includes a novel algorithm for computing a desired pose and surge speed in the vicinity of the target boat, jointly defined as a motion goal, and tightly integrates it with trajectory planning and tracking components of the entire system. The trajectory planner generates a dynamically feasible, collision-free trajectory to allow the USV to safely reach the computed motion goal. Trajectory planning needs to be sufficiently fast and yet produce dynamically feasible and short trajectories due to the moving target. This required speeding up the planning by searching for trajectories through a hybrid, pose-position state space using a multi-resolution control action set. The search in the velocity space is decoupled from the search for a trajectory in the pose space. Therefore, the underlying trajectory tracking controller computes desired surge speed for each segment of the trajectory and ensures that the USV maintains it. We have carried out simulation as well as experimental studies to demonstrate the effectiveness of the developed approach.

We present a model-predictive trajectory planning algorithm for following a target boat by an autonomous unmanned surface vehicle (USV) in an environment with static obstacle regions and civilian boats. The planner developed in this work is capable of making a balanced trade-off among the following, possibly conflicting criteria: the risk of losing the target boat, trajectory length, risk of collision with obstacles, violation of the Coast Guard Collision Regulations (COLREGs), also known as “rules of the road”, and execution of avoidance maneuvers against vessels that do not follow the rules. The planner addresses these criteria by combining a search for a dynamically feasible trajectory to a suitable pose behind the target boat in 4D state space, forming a time-extended lattice, and reactive planning that tracks this trajectory using control actions that respect the USV dynamics and are compliant with COLREGs. The reactive part of the planner represents a generalization of the velocity obstacles paradigm by computing obstacles in the control space using a system-identified, dynamic model of the USV as well as worst-case and probabilistic predictive motion models of other vessels. We present simulation and experimental results using an autonomous unmanned surface vehicle platform and a human-driven vessel to demonstrate that the planner is capable of fulfilling the above mentioned criteria.

This article presents an experimental assessment of an Unmanned Surface Vehicle (USV) executing an approach behavior to several stationary targets in an obstacle field. A lattice-based trajectory planner is implemented with a priori knowledge of the vehicle characteristics. In parallel, a low-level controller is developed for the vehicle using a proportional control law. These systems are integrated on the USV control system using the Lightweight Communications and Marshalling (LCM) message passing system. Filtered vehicle-state information from onboard sensors is passed to the planner, which returns a least-cost, dynamically feasible trajectory for achieving the ascertained goal. The system was tested in a 750 m by 150 m area of the US Intracoastal Waterway in South Florida in the presence of wind and wave disturbances to characterize its effectiveness in a real-world scenario. The vehicle was able to replicate behavior predicted in simulations when navigating around obstacles. The approach distance to each target was favorably lower than the user-defined limit. Owing to the fact that the USV uses differential thrust for steering, the vehicle tracked the planned trajectories better at lower speeds.Copyright © 2013 by ASME

Safe and efficient following of a time varying motion goal by an autonomous unmanned surface vehicle (USV) in a sea environment with obstacles is a challenge. The vehicle’s tracking capability is inherently influenced by its dynamics, the motion characteristics of the motion goal, as well as by the configuration of obstacles in the marine environment. We have developed an approach that utilizes a lattice-based trajectory planning to generate a dynamically feasible, resolution optimal, collision-free trajectory to allow the vehicle to reliably reach the motion goal. We utilized a trajectory following controller to achieve high tracking efficiency while still preserving motion safety. The entire approach is based on the developed USV system architecture that encapsulates the necessary trajectory planning components. We demonstrated the effectiveness of the developed planner in a simulated environment with static obstacles. In addition, we have developed a physical evaluation setup.