Search

We still struggle to deliver autonomous robots that perform manipulation tasks as simple for a human as picking up items. A portion of the difficulty of this task lays on the fact that such operation requires a robot that can deal with uncertainty in an unstructured environment. We propose in this thesis the use of visual and tactile perception for providing solutions that can improve the robustness of a robotic manipulator in such environment. In this thesis, we approach robotic grasping using a single 3D point cloud with a partial view of the objects present in the scene. Moreover, the objects are unknown: they have not been previously recognised and we do not have a 3D model to compute candidate grasping points. In experimentation, we prove that our solution is fast and robust, taking in average 17 ms to find a grasp which is stable 85% of the time. Tactile sensors provide a rich source of information regarding the contact experienced by a robotic hand during the manipulation of an object. In this thesis, we exploit with deep learning this type of data for approaching the prediction of the stability of a grasp and the detection of the direction of slip of a contacted object. We prove that our solutions could correctly predict stability 76% of the time with a single tactile reading. We also demonstrate that learning temporal and spatial patterns leads to detections of the direction of slip which are correct up to 82% of the time and are only delayed 50 ms after the actual slip event begins. Despite the good results achieved on the previous two tactile tasks, this data modality has a serious flaw: it can only be registered during contact. In contrast, humans can estimate the feeling of grasping an object just by looking at it. Inspired by this, we present in this thesis our contributions for learning to generate tactile responses from vision. We propose a supervised solution based on training a deep neural network that models the behaviour of a tactile sensor, give

Tactile perception is a rich source of information for robotic grasping: it allows a robot to identify a grasped object and assess the stability of a grasp, among other things. However, the tactile sensor must come into contact with the target object in order to produce readings. As a result, tactile data can only be attained if a real contact is made. We propose to overcome this restriction by employing a method that models the behaviour of a tactile sensor using 3D vision and grasp information as a stimulus. Our system regresses the quantified tactile response that would be experienced if this grasp were performed on the object. We experiment with 16 items and 4 tactile data modalities to show that our proposal learns this task with low error.

Through the use of tactile perception, a manipulator can estimate the stability of its grip, among others. However, tactile sensors are only activated upon contact. In contrast, humans can estimate the feeling of touching an object from its visual appearance. Providing robots with this ability to generate tactile perception from vision is desirable to achieve autonomy. To accomplish this, we propose using a Generative Adversarial Network. Our system learns to generate tactile responses using as stimulus a visual representation of the object and target grasping data. Since collecting labeled samples of robotic tactile responses consumes hardware resources and time, we apply semi-supervised techniques. For this work, we collected 4000 samples with 4 deformable items and experiment with 4 tactile modalities.

Este trabajo presenta un método para clasificar objetos agarrados con una mano robótica multidedo combinando en un descriptor híbrido datos propioceptivos y táctiles. Los datos propioceptivos se obtienen a partir de las posiciones articulares de la mano y los táctiles se extraen del contacto registrado por células de presión instaladas en las falanges. La aproximación propuesta permite identificar el objeto aprendiendo de forma implícita su geometría y rigidez usando los datos que facilitan los sensores. En este trabajo demostramos que el uso de datos bimodales con técnicas de aprendizaje supervisado mejora la tasa de reconocimiento. En la experimentación, se han llevado a cabo más de 3000 agarres de hasta 7 objetos domésticos distintos, obteniendo clasificaciones correctas del 95%con métrica F1, realizando una única palpación del objeto. Además, la generalización del método se ha verificado entrenando nuestro sistema con unos objetos y posteriormente, clasificando otros nuevos similares., This work presents a method to classify grasped objects with a multi-fingered robotic hand combining proprioceptive and tactile data in a hybrid descriptor. The proprioceptive data are obtained from the joint positions of the hand and the tactile data are obtained from the contact registered by pressure cells installed on the phalanges. The proposed approach allows us to identify the grasped object by learning the contact geometry and stiffness from the readings by sensors. In this work, we show that using bimodal data of different nature along with supervised learning techniques improves the recognition rate. In experimentation, more than 3000 grasps of up to 7 different domestic objects have been carried out, obtaining an average F1 score around 95 %, performing just a single grasp. In addition, the generalization of the method has been verified by training our system with certain objects and classifying new, similar ones without any prior knowledge.

The task of robotic grasping brings together several challenges. Among them, we focus on the calculus of where gripper plates should be placed over an object’s surface in order to grasp it. To do this, we have developed a method based on visual information. The main goal is to geometrically analyse a single 3D point cloud, where the object is present, to find the best pair of contact points so a gripper can perform a stable grasp of the object. Our proposal is to find these points near a perpendicular cutting plane to the object’s main axis through its centroid. We have found that this method shows promising experimental results fast and accurate enough to be used on real service robots.

En tareas de manipulación robótica es de especial interés detectar si un agarre es estable o por el contrario, el objeto agarrado se desliza entre los dedos debido a un contacto inadecuado. Con frecuencia, la inestabilidad en el agarre puede ser como consecuencia de una mala pose de la mano o pinza robótica durante su ejecución y o una presión de contacto insuficiente a la hora de ejercer la tarea. El empleo de información táctil y la representación de ésta es vital para llevar a cabo la predicción de estabilidad en el agarre. En este trabajo, se presentan y comparan distintas metodologías para representar la información táctil, así como los métodos de aprendizaje más adecuados en función de la representación táctil escogida.

Tactile sensors provide useful contact data during the interaction with an object which can be used to accurately learn to determine the stability of a grasp. Most of the works in the literature represented tactile readings as plain feature vectors or matrix-like tactile images, using them to train machine learning models. In this work, we explore an alternative way of exploiting tactile information to predict grasp stability by leveraging graph-like representations of tactile data, which preserve the actual spatial arrangement of the sensor's taxels and their locality. In experimentation, we trained a Graph Neural Network to binary classify grasps as stable or slippery ones. To train such network and prove its predictive capabilities for the problem at hand, we captured a novel dataset of ~ 5000 three-fingered grasps across 41 objects for training and 1000 grasps with 10 unknown objects for testing. Our experiments prove that this novel approach can be effectively used to predict grasp stability.

Latest trends in robotic grasping combine vision and touch for improving the performance of systems at tasks like stability prediction. However, tactile data are only available during the grasp, limiting the set of scenarios in which multimodal solutions can be applied. Could we obtain it prior to grasping? We explore the use of visual perception as a stimulus for generating tactile data so the robotic system can ”feel” the response of the tactile perception just by looking at the object.

One of the challenges in robotic grasping tasks is the problem of detecting whether a grip is stable or not. The lack of stability during a manipulation operation usually causes the slippage of the grasped object due to poor contact forces. Frequently, an unstable grip can be caused by an inadequate pose of the robotic hand or by insufficient contact pressure, or both. The use of tactile data is essential to check such conditions and, therefore, predict the stability of a grasp. In this work, we present and compare different methodologies based on deep learning in order to represent and process tactile data for both stability and slip prediction.

Industrial and service robots deal with the complex task of grasping objects that have different shapes and which are seen from diverse points of view. In order to autonomously perform grasps, the robot must calculate where to place its robotic hand to ensure that the grasp is stable. We propose a method to find the best pair of grasping points given a three-dimensional point cloud with the partial view of an unknown object. We use a set of straightforward geometric rules to explore the cloud and propose grasping points on the surface of the object. We then adapt the pair of contacts to a multi-fingered hand used in experimentation. We prove that, after performing 500 grasps of different objects, our approach is fast, taking an average of 17.5 ms to propose contacts, while attaining a grasp success rate of 85.5%. Moreover, the method is sufficiently flexible and stable to work with objects in changing environments, such as those confronted by industrial or service robots.

Tactile sensors provide useful contact data during the interaction with an object which can be used to accurately learn to determine the stability of a grasp. Most of the works in the literature represented tactile readings as plain feature vectors or matrix-like tactile images, using them to train machine learning models. In this work, we explore an alternative way of exploiting tactile information to predict grasp stability by leveraging graph-like representations of tactile data, which preserve the actual spatial arrangement of the sensor's taxels and their locality. In experimentation, we trained a Graph Neural Network to binary classify grasps as stable or slippery ones. To train such network and prove its predictive capabilities for the problem at hand, we captured a novel dataset of approximately 5000 three-fingered grasps across 41 objects for training and 1000 grasps with 10 unknown objects for testing. Our experiments prove that this novel approach can be effectively used to predict grasp stability.

In the past, methods for hand sign recognition have been successfully tested in Human Robot Interaction (HRI) using traditional methodologies based on static image features and machine learning. However, the recognition of gestures in video sequences is a problem still open, because current detection methods achieve low scores when the background is undefined or in unstructured scenarios. Deep learning techniques are being applied to approach a solution for this problem in recent years. In this paper, we present a study in which we analyse the performance of a 3DCNN architecture for hand gesture recognition in an unstructured scenario. The system yields a score of 73% in both accuracy and F1. The aim of the work is the implementation of a system for commanding robots with gestures recorded by video in real scenarios.

Robotic manipulators have to constantly deal with the complex task of detecting whether a grasp is stable or, in contrast, whether the grasped object is slipping. Recognising the type of slippage—translational, rotational—and its direction is more challenging than detecting only stability, but is simultaneously of greater use as regards correcting the aforementioned grasping issues. In this work, we propose a learning methodology for detecting the direction of a slip (seven categories) using spatio-temporal tactile features learnt from one tactile sensor. Tactile readings are, therefore, pre-processed and fed to a ConvLSTM that learns to detect these directions with just 50 ms of data. We have extensively evaluated the performance of the system and have achieved relatively high results at the detection of the direction of slip on unseen objects with familiar properties (82.56% accuracy).

Tactile sensors supply useful information during the interaction with an object that can be used for assessing the stability of a grasp. Most of the previous works on this topic processed tactile readings as signals by calculating hand-picked features. Some of them have processed these readings as images calculating characteristics on matrix-like sensors. In this work, we explore how non-matrix sensors (sensors with taxels not arranged exactly in a matrix) can be processed as tactile images as well. In addition, we prove that they can be used for predicting grasp stability by training a Convolutional Neural Network (CNN) with them. We captured over 2500 real three-fingered grasps on 41 everyday objects to train a CNN that exploited the local connectivity inherent on the non-matrix tactile sensors, achieving 94.2% F1-score on predicting stability.

This paper presents a system that combines computer vision and surface electromyography techniques to perform grasping tasks with a robotic hand. In order to achieve a reliable grasping action, the vision-driven system is used to compute pre-grasping poses of the robotic system based on the analysis of tridimensional object features. Then, the human operator can correct the pre-grasping pose of the robot using surface electromyographic signals from the forearm during wrist flexion and extension. Weak wrist flexions and extensions allow a fine adjustment of the robotic system to grasp the object and finally, when the operator considers that the grasping position is optimal, a strong flexion is performed to initiate the grasping of the object. The system has been tested with several subjects to check its performance showing a grasping accuracy of around 95% of the attempted grasps which increases in more than a 13% the grasping accuracy of previous experiments in which electromyographic control was not implemented.

Las tareas de manipulación de objetos, en ocasiones, requieren hacer uso de dos o más robots cooperando. En la industria 4.0 cada vez es más demandada la robótica asistencial por ejemplo para llevar a cabo tareas como levantar, arrastrar o empujar un bulto/paquete de elevado peso y/o de ciertas dimensiones. Por lo que cada vez es más frecuente encontrar robots con forma humana destinados a ayudar al operario humano en actividades en las que se dan este tipo de movimientos. En este artículo se presenta una plataforma robótica asistida por visión para llevar a cabo tareas de agarre y manipulación bimanual de objetos. La plataforma robótica consta de un torso metálico con articulación rotacional en la cadera y de dos manipuladores industriales, de 7 grados de libertad, que actúan como brazos y que a su vez montan en sus extremos sendas manos robóticas multidedo. Cada una de las extremidades superiores está dotada de percepción visual, ya que hacen uso de 2 sensores RGBD ubicados en configuración eye-to-hand. La plataforma ha sido empleada y testeada con éxito para llevar a cabo agarres bimanuales de objetos en aras de poder desarrollar tareas de manipulación cooperativa de manera coordinada entre ambas extremidades., Manipulation tasks of objects, sometimes, requires the use of two or more cooperating robots. In the industry 4.0, assistance robotic is being more and more demanded, for example, to carry out tasks such as lifting, dragging or pushing of both heavy and big packages. Consequently, it is possible to find robots with human appearance addressed on helping human operators in activities in which these types of movements occur. In this article, a vision-assisted robotic platform is presented to carry out both grasping tasks and bimanual manipulation of objects. The robotic platform consists of a metallic torso with rotational joint at the hip and two industrial manipulators, with 7 degrees of freedom, which act as arms. Each arm mounts a multifinger robotic hand at the end. Each of the upper extremities use visual perception from 3 RGBD sensors located in an eye-to-hand configuration. The platform has been successfully used and tested to carry out bimanual object grasping in order to develop cooperative manipulation tasks in a coordinated way between both robotic extremities.

Este trabajo se presenta una aproximación híbrida propioceptiva-táctil para reconocer objetos agarrados. La información propioceptiva de una mano robótica es usada para estimar la geometría de contacto y así, distinguir la forma de cada uno de los objetos que están siendo agarrados. La geometría de contacto viene determinada por los datos articulares de la mano robótica cuando ésta lleva a cabo un agarre en configuración de cierre sobre la superficie del objeto. Además, la información táctil permite determinar propiedades de rigidez y flexibilidad del objeto agarrado, mejorando el proceso de reconocimiento cuando la geometría de contacto y por lo tanto, la forma de los objetos es similar. El método propuesto emplea técnicas de clasificación de aprendizaje supervisado para combinar los datos de ambos tipos de sensores e identificar el tipo de objeto con un porcentaje de acierto medio del 95,5% con métrica ‘accuracy’ y 95.3% con F1(F-score) aun en presencia de incertidumbre de medida y ambigüedad. Estas ratios de acierto se han alcanzado experimentando con 7 objetos domésticos y llevando a cabo más de 3000 agarres., This work presents a hybrid proprioceptive-tactile approach to recognize grasped objects. Proprioceptive data of a robotic hand are used to estimate contact geometry and thus, to distinguish the shape of each of the objects that are being grasped. The contact geometry is determined by the joint data of the robotic hand when it carries out a grip in closure grasps configuration on the object surface. In addition, the tactile data allow to robotic hand to determine rigidity and flexibility properties of the grasped object, improving the recognition process when the contact geometry and therefore, the shapes of different objects are similar. The proposed method employs supervised learning classification techniques to combine the data from both types of sensors and identify the type of object with an average success rate of 95,5% (with accuracy) and 95.3% (with F1 or F-score)even in the presence of measurement with uncertainty and ambiguity of pose. These success ratios have been achieved by experimenting with 7 different objects and performing more than 3000 grasps.

[Resumen] Las tareas de manipulación de objetos, en ocasiones, requieren hacer uso de dos o más robots cooperando. En la industria 4.0 cada vez es más demandada la robótica asistencial por ejemplo para llevar a cabo tareas como levantar, arrastrar o empujar un bulto/paquete de elevado peso y/o de ciertas dimensiones. Por lo que cada vez es más frecuente encontrar robots con forma humana destinados a ayudar al operario humano en actividades en las que se dan este tipo de movimientos. En este artículo se presenta una plataforma robótica asistida por visión para llevar a cabo tare- as de agarre y manipulación bimanual de objetos. La plataforma robótica consta de un torso metálico con articulación rotacional en la cadera y de dos manipuladores industriales, de 7 grados de libertad, que actúan como brazos y que a su vez montan en sus extremos sendas manos robóticas multidedo. Cada una de las extremidades superiores está dotada de percepción visual, ya que hacen uso de 2 sensores RGBD ubicados en configuración eye-to-hand. La plataforma ha sido empleada y testeada con éxito para llevar a cabo agarres bimanuales de objetos en aras de poder desarrollar tareas de manipulación cooperativa de manera coordinada entre ambas extremidades., [Abstract] Manipulation tasks of objects, sometimes, requires the use of two or more cooperating robots. In the industry 4.0, assistance robotic is being more and more demanded, for example, to carry out tasks such as lifting, dragging or pushing of both heavy and big packages. Consequently, it is possible to find robots with human appearance addressed on helping human operators in activities in which these types of movements occur. In this article, a vision-assisted robotic platform is presented to carry out both grasping tasks and bimanual manipulation of objects. The robotic platform consists of a metallic torso with rotational joint at the hip and two industrial manipulators, with 7 degrees of freedom, which act as arms. Each arm mounts a multi-finger robotic hand at the end. Each of the upper extremities use visual perception from 3 RGBD sensors located in an eye-to-hand configuration. The platform has been successfully used and tested to carry out bi- manual object grasping in order to develop cooperative manipulation tasks in a coordinated way between both robotic extremities

In this paper, we focus on the task of computing a pair of points for grasping unknown objects, given a single point cloud scene with a partial view of them. The main goal is to estimate the best pair of 3D-located points so that a gripper can perform a stable grasp over the objects in the scene with no prior knowledge of their shape. We propose a geometrical approach to find those contact points by placing them near a perpendicular cutting plane to the object’s main axis and through its centroid. During the experimentation we have found that this solution is fast enough and gives sufficiently stable grasps for being used on a real service robot.

Los datos abiertos se están convirtiendo en una fuente de datos muy útil en diversos ámbitos como el de la movilidad, el turístico o el ámbito periodístico. Sin embargo, se ha trabajado muy poco en uno de los ámbitos más relevantes par a la humanidad: el ámbito de las ciencias de la salud. Esto es debido, sobre todo, a las reticencias que plantea la apertura de datos clínicos y su aparente contradicción con la privacidad de las personas. En este trabajo se plantea como objetivo la creación de un sistema recomendador en salud utilizando datos abiertos con el fin de mostrar la gran oportunidad que representa la reutilización de estos datos abiertos. Para ello se estudiarán los datos abiertos, actualmente disponibles, en salud a nivel nacional e internacional, se analizarán las bases de datos encontradas y, se realizará la extracción de descriptores y caracterización de la información.