Search

The most common approaches for classification rely on the inference of a specific class. However, every category could be naturally organized within a taxonomic tree, from the most general concept to the specific element, and that is how human knowledge works. This representation avoids the necessity of learning roughly the same features for a range of very similar categories, and it is easier to understand and work with and provides a classification for each abstraction level. In this paper, we carry out an exhaustive study of different methods to perform multilevel classification applied to the task of classifying wild animals and plant species. Different convolutional backbones, data setups, and ensembling techniques are explored to find the model which provides the best performance. As our experimentation remarks, in order to achieve the best performance on the datasets that are arranged in a tree-like structure, the classifier must feature an EfficientNetB5 backbone with an input size of 300 × 300 px, followed by a multilevel classifier. In addition, a Multiscale Crop data augmentation process must be carried out. Finally, the accuracy of this setup is a 62% top-1 accuracy and 88% top-5 accuracy. The architecture could benefit for an accuracy boost if it is involved in an ensemble of cascade classifiers, but the computational demand is unbearable for any real application.

Cognitive architectures allow robots to perform their operations by drawing on a process that aims to simulate human reasoning. This paper presents an integrated semantic artificial memory system in cognitive architecture based on symbolic reasoning and a connective representation of the knowledge. This memory system attempts to simulate how humans learn to distinguish instances of particular objects within their class using a convolutional network to detect the relevant elements of an image. We use a vector with the extracted features to learn to discriminate an instance of another element from the same class. A novel feature of our approach is its autonomous learning process during the operation of the robot, integrating a deep learning embedding with a statistical classifier. The usefulness and robustness of this method are demonstrated by applying it to a social robot that learns to differentiate people. Finally, experiments are carried out to validate our approach, comparing the detection results with several alternative methods. This work has been funded by the Spanish Government TIN2016-76515-R grant for the COMBAHO project, supported with Feder funds. This work has also been supported by a Spanish grant for PhD studies ACIF/2017/243 and FPU16/00887.

Convolutional Neural Networks (CNNs) have become the default paradigm for addressing classification problems, especially, but not only, in image recognition. This is mainly due to the high success rate they provide. Although there are currently approaches that apply deep learning to the 3D shape recognition problem, they are either too slow for online use or too error-prone. To fill this gap, we propose 3DSliceLeNet, a deep learning architecture for point cloud classification. Our proposal converts the input point clouds into a two-dimensional representation by performing a slicing process and projecting the points to the principal planes, thus generating images that are used by the convolutional architecture. 3DSliceLeNet successfully achieves both high accuracy and low computational cost. A dense set of experiments has been conducted to validate our system under the ModelNet challenge, a large-scale 3D Computer Aided Design (CAD) model dataset. Our proposal achieves a success rate of 94.37% and an Area Under Curve (AUC) of 0.978 on the ModelNet-10 classification task. Grant PID2019-104818RB-I00 funded by MCIN/AEI/10.13039/501100011033 and by “ERDF A way of making Europe”.

Currently, state-of-the-art methods for 3D object recognition rely in a deep learning-pipeline. Nonetheless, these methods require a large amount of data that is not easy to obtain. In addition to that, the majority of them exploit features of the datasets, like the fact of being CAD models to create rendered representation which will not work in real life because the 3D sensors provide point clouds. We propose a novel global descriptor for point clouds which takes advantage of the fractal dimension of the objects. Our approach introduces many benefits, such as being agnostic to the density of points of the sample, number of points in the input cloud, sensor of choice, and noise up to a level, and it works on real life point cloud data provided by commercial sensors. We tested our descriptor for 3D object recognition using ModelNet, which is a well-known dataset for that task. Our approach achieves 92.84% accuracy on the ModelNet10, and 88.74% accuracy on the ModelNet40. This work was supported in part by the Spanish Government, with Feder funds, under Grant PID2019-104818RB-I00, and in part by the Spanish Grants for Ph.D. studies under Grant ACIF/2017/243 and Grant FPU16/00887.

In this paper, we present a novel deep learning-based architecture, which is under the scope of expert and intelligent systems, to perform accurate real-time tridimensional hand pose estimation using a single RGB frame as an input, so there is no need to use multiple cameras or points of view, or RGB-D devices. The proposed pipeline is composed of two convolutional neural network architectures. The first one is in charge of detecting the hand in the image. The second one is able to accurately infer the tridimensional position of the joints retrieving, thus, the full hand pose. To do this, we captured our own large-scale dataset composed of images of hands and the corresponding 3D joints annotations. The proposal achieved a 3D hand pose mean error of below 5 mm on both the proposed dataset and Stereo Hand Pose Tracking Benchmark, which is a public dataset. Our method also outperforms the state-of-the-art methods. We also demonstrate in this paper the application of the proposal to perform a robotic hand teleoperation with high success. This work has been supported by the Spanish Government TIN2016-76515R Grant, supported with Feder funds. This work has also been supported by a Spanish grant for PhD studies ACIF/2017/243

In this work, we introduce HaReS, a hand rehabilitation system. Our proposal integrates a series of exercises, jointly developed with a foundation for those with motor and cognitive injuries, that are aimed at improving the skills of patients and the adherence to the rehabilitation plan. Our system takes advantage of a low-cost hand-tracking device to provide a quantitative analysis of the performance of the patient. It also integrates a low-cost surface electromyography (sEMG) sensor in order to provide insight about which muscles are being activated while completing the exercises. It is also modular and can be deployed on a social robot. We tested our proposal in two different facilities for rehabilitation with high success. The therapists and patients felt more motivation while using HaReS, which improved the adherence to the rehabilitation plan. In addition, the therapists were able to provide services to more patients than when they used their traditional methodology. This work was funded by a Spanish Government PID2019-104818RB-I00 grant, supported by Feder funds. It was also supported by Spanish grants for PhD studies ACIF/2017/243 and FPU16/00887.

Over one billion people in the world live with some form of disability. This is incessantly increasing due to aging population and chronic diseases. Among the emerging social needs, rehabilitation services are the most required. However, they are scarce and expensive what considerably limits access to them. In this paper, we propose EVA, an augmented reality platform to engage and supervise rehabilitation sessions at home using low-cost sensors. It also stores the user’s statistics and allows therapists to tailor the exercise programs according to their performance. This system has been evaluated in both qualitative and quantitative ways obtaining very promising results. This work has been supported by the Spanish Government TIN2016-76515R Grant, supported with Feder funds. Edmanuel Cruz is funded by a Panamenian grant for Ph.D. studies IFARHU and SENACYT 270-2016-207. This work has also been supported by a Spanish grant for PhD studies ACIF/2017/243 and FPU16/00887. Thanks also to Nvidia for the generous donation of a Titan Xp and a Quadro P6000.

In the last years, the care of dependent people, either by disease, accident, disability, or age, is one of the current priority research topics in developed countries. Moreover, such care is intended to be at patients home, in order to minimize the cost of therapies. Patients rehabilitation will be fulfilled when their integration in society is achieved, either in the family or in a work environment. To address this challenge, we propose the development and evaluation of an assistant for people with acquired brain injury or dependents. This assistant is twofold: in the patient’s home is based on the design and use of an intelligent environment with abilities to monitor and active learning, combined with an autonomous social robot for interactive assistance and stimulation. On the other hand, it is complemented with an outdoor assistant, to help patients under disorientation or complex situations. This involves the integration of several existing technologies and provides solutions to a variety of technological challenges. Deep leaning-based techniques are proposed as core technology to solve these problems. This work has been funded by the Spanish Government TIN2016-76515-R grant for the COMBAHO project, supported with Feder funds. It has also been supported by the University of Alicante project GRE16-19 and the Valencian Government GV/2018/022. This work has also been supported by Spanish grants for PhD studies FPU15/04516 and ACIF/2017/243. Experiments were made possible by a generous hardware donation from NVIDIA.

Deep learning-based methods have proven to be the best performers when it comes to object recognition cues both in images and tridimensional data. Nonetheless, when it comes to 3D object recognition, the authors tend to convert the 3D data to images and then perform their classification. However, despite its accuracy, this approach has some issues. In this work, we present a deep learning pipeline for object recognition that takes a point cloud as input and provides the classification probabilities as output. Our proposal is trained on synthetic CAD objects and is able to perform accurately when fed with real data provided by commercial sensors. Unlike most approaches, our method is specifically trained to work on partial views of the objects rather than on a full representation, which is not the representation of the objects as captured by commercial sensors. We trained our proposal with the ModelNet10 dataset and achieved a 78.39% accuracy. We also tested it by adding noise to the dataset and against a number of datasets and real data with high success. This work has been funded by the Spanish Government TIN2016-76515-R grant for the COMBAHO project, supported with Feder funds. It has also been supported by Spanish grants for PhD studies ACIF/2017/243 and FPU16/00887.

In the past years, several works on urban object detection from the point of view of a person have been made. These works are intended to provide an enhanced understanding of the environ- ment for blind and visually challenged people. The mentioned approaches mostly rely in deep learning and machine learning methods. Nonetheless, these approaches only work with direct and bright light, namely, they will only perform correctly on daylight conditions. This is because deep learning algorithms require large amounts of data and the currently available datasets do not address this matter. In this work, we propose UrOAC, a dataset of urban objects captured in a range of different lightning conditions, from bright daylight to low and poor night-time lighting conditions. In the latter, the objects are only lit by low ambient light, street lamps and headlights of passing-by vehicles. The dataset depicts the following objects: pedestrian crosswalks, green traffic lights and red traffic lights. The annotations include the category and the bounding-box of each object. This dataset could be used for improve the performance at night-time and under low-light conditions of any vision-based method that involves urban objects. For instance, guidance and object detection devices for the visually challenged or self-driving and intelligent vehicles. This work has been supported by the Spanish Government PID2019-104818RB-I00 Grant, co-funded by EU Structural Funds.

The capacity of a robot to automatically adapt to new environments is crucial, especially in social robotics. Often, when these robots are deployed in home or office environments, they tend to fail because they lack the ability to adapt to new and continuously changing scenarios. In order to accomplish this task, robots must obtain new information from the environment, and then add it to their already learned knowledge. Deep learning techniques are often used to tackle this problem successfully. However, these approaches, complete retraining of the models, which is highly time-consuming. In this work, several strategies are tested to find the best way to include new knowledge in an already learned model in a deep learning pipeline, putting the spotlight on the time spent for this training. We tackle the localization problem in the long term with a deep learning approach and testing several retraining strategies. The results of the experiments are discussed and, finally, the best approach is deployed on a Pepper robot. The capacity of a robot to automatically adapt to new environments is crucial, especially in social robotics. Often, when these robots are deployed in home or office environments, they tend to fail because they lack the ability to adapt to new and continuously changing scenarios. In order to accomplish this task, robots must obtain new information from the environment, and then add it to their already learned knowledge. Deep learning techniques are often used to tackle this problem successfully. However, these approaches, complete retraining of the models, which is highly time-consuming. In this work, several strategies are tested to find the best way to include new knowledge in an already learned model in a deep learning pipeline, putting the spotlight on the time spent for this training. We tackle the localization problem in the long term with a deep learning approach and testing several retraining strategies. The results of the experiments are discussed and, finally, the best approach is deployed on a Pepper robot.

As estimated by the World Health Organization, there are millions of people who lives with some form of vision impairment. As a consequence, some of them present mobility problems in outdoor environments. With the aim of helping them, we propose in this work a system which is capable of delivering the position of potential obstacles in outdoor scenarios. Our approach is based on non-intrusive wearable devices and focuses also on being low-cost. First, a depth map of the scene is estimated from a color image, which provides 3D information of the environment. Then, an urban object detector is in charge of detecting the semantics of the objects in the scene. Finally, the three-dimensional and semantic data is summarized in a simpler representation of the potential obstacles the users have in front of them. This information is transmitted to the user through spoken or haptic feedback. Our system is able to run at about 3.8 fps and achieved a 87.99% mean accuracy in obstacle presence detection. Finally, we deployed our system in a pilot test which involved an actual person with vision impairment, who validated the effectiveness of our proposal for improving its navigation capabilities in outdoors. This work has been supported by the Spanish Government TIN2016-76515R Grant, supported with Feder funds, the University of Alicante project GRE16-19, and by the Valencian Government project GV/2018/022. Edmanuel Cruz is funded by a Panamenian grant for PhD studies IFARHU & SENACYT 270-2016-207. This work has also been supported by a Spanish grant for PhD studies ACIF/2017/243. Thanks also to Nvidia for the generous donation of a Titan Xp and a Quadro P6000.

In several large retail stores, such as malls, sport or food stores, the customer often feels lost due to the difficulty in finding a product. Although these large stores usually have visual signs to guide customers toward specific products, sometimes these signs are also hard to find and are not updated. In this paper, we propose a system that jointly combines deep learning and augmented reality techniques to provide the customer with useful information. First, the proposed system learns the visual appearance of different areas in the store using a deep learning architecture. Then, customers can use their mobile devices to take a picture of the area where they are located within the store. Uploading this image to the system trained for image classification, we are able to identify the area where the customer is located. Then, using this information and novel augmented reality techniques, we provide information about the area where the customer is located: route to another area where a product is available, 3D product visualization, user location, analytics, etc. The system developed is able to successfully locate a user in an example store with 98% accuracy. The combination of deep learning systems together with augmented reality techniques shows promising results toward improving user experience in retail/commerce applications: branding, advance visualization, personalization, enhanced customer experience, etc. In several large retail stores, such as malls, sport or food stores, the customer often feels lost due to the difficulty in finding a product. Although these large stores usually have visual signs to guide customers toward specific products, sometimes these signs are also hard to find and are not updated. In this paper, we propose a system that jointly combines deep learning and augmented reality techniques to provide the customer with useful information. First, the proposed system learns the visual appearance of different areas in the store using a deep learning architecture. Then, customers can use their mobile devices to take a picture of the area where they are located within the store. Uploading this image to the system trained for image classification, we are able to identify the area where the customer is located. Then, using this information and novel augmented reality techniques, we provide information about the area where the customer is located: route to another area where a product is available, 3D product visualization, user location, analytics, etc. The system developed is able to successfully locate a user in an example store with 98% accuracy. The combination of deep learning systems together with augmented reality techniques shows promising results toward improving user experience in retail/commerce applications: branding, advance visualization, personalization, enhanced customer experience, etc.

In this paper, we propose a new dataset for outdoor depth estimation from single and stereo RGB images. The dataset was acquired from the point of view of a pedestrian. Currently, the most novel approaches take advantage of deep learning-based techniques, which have proven to outperform traditional state-of-the-art computer vision methods. Nonetheless, these methods require large amounts of reliable ground-truth data. Despite there already existing several datasets that could be used for depth estimation, almost none of them are outdoor-oriented from an egocentric point of view. Our dataset introduces a large number of high-definition pairs of color frames and corresponding depth maps from a human perspective. In addition, the proposed dataset also features human interaction and great variability of data, as shown in this work., Design Type(s)modeling and simulation objective • image creation and editing objective • database creation objectiveMeasurement Type(s)imageTechnology Type(s)digital cameraFactor Type(s)geographic locationSample Characteristic(s)Alicante • anthropogenic habitat Machine-accessible metadata file describing the reported data (ISA-Tab format)

Schaeffer's sign language consists of a reduced set of gestures designed to help children with autism or cognitive learning disabilities to develop adequate communication skills. Our automatic recognition system for Schaeffer's gesture language uses the information provided by an RGB-D camera to capture body motion and recognize gestures using dynamic time warping combined with k-nearest neighbors methods. The learning process is reinforced by the interaction with the proposed system that accelerates learning itself thus helping both children and educators. To demonstrate the validity of the system, a set of qualitative experiments with children were carried out. As a result, a system which is able to recognize a subset of 11 gestures of Schaeffer's sign language online was achieved. This work has been supported by the Spanish Government DPI2013-40534-R Grant, supported with Feder funds.

As it is well known, some versions of the Pepper robot provide poor depth perception due to the lenses it has in front of the tridimensional sensor. In this paper, we present a method to improving that faulty 3D perception. Our proposal is based on a combination of the actual depth readings of Pepper and a deep learning-based monocular depth estimation. As shown, the combination of both of them provides a better 3D representation of the scene. In previous works we made an initial approximation of this fusion technique, but it had some drawbacks. In this paper we analyze the pros and cons of the Pepper readings, the monocular depth estimation method and our previous fusion method. Finally, we demonstrate that the proposed fusion method outperforms them all. This work was supported in part by the Spanish Government, through Feder funds under Grant TIN2016-76515R, and in part by the Spanish Grant for Ph.D. studies under Grant ACIF/2017/243 and Grant FPU16/00887. The work of E. Cruz was supported by the Panamenian Grant for Ph.D. studies IFARHU & SENACYT under Grant 270-2016-207.

Ambient assisted living (AAL) environments are currently a key focus of interest as an option to assist and monitor disabled and elderly people. These systems can improve their quality of life and personal autonomy by detecting events such as entering potentially dangerous areas, potential fall events, or extended stays in the same place. Nonetheless, there are areas that remain outside the scope of AAL systems due to the placement of cameras. There also exist sources of danger in the scope of the camera that the AAL system cannot detect. These sources of danger are relatively small in size, occluded, or nonstatic. To solve this problem, we propose the inclusion of a robot which maps such uncovered areas looking for new potentially dangerous areas that go unnoticed by the AAL. The robot then sends this information to the AAL system in order to improve its performance. Experimentation in real-life scenarios successfully validates our approach. This work was supported by the Spanish Government TIN2016-76515R Grant, supported with FEDER funds. This work was also supported by a Spanish grant for PhD studies (ACIF/2017/243 and FPU16/00887).

Accurate visual hand pose estimation at joint level has several applications for human-robot interaction, natural user interfaces and virtual/augmented reality applications. However, it is still an open problem being addressed by the computer vision community. Recent novel deep learning techniques may help circumvent the limitations of standard approaches. However, they require large amounts of accurate annotated data. Hand pose datasets that have been released so far present issues such as limited number of samples, inaccurate data or high-level annotations. Moreover, most of them are focused on depth-based approaches, providing only depth information (missing RGB data). In this work, we present a novel multiview hand pose dataset in which we provide hand color images and different kind of annotations for each sample, i.e. the bounding box and the 2D and 3D location on the joints in the hand. Furthermore, we introduce a simple yet accurate deep learning architecture for real-time robust 2D hand pose estimation. Then, we conduct experiments that show how the use of the proposed dataset in the training stage produces accurate results for 2D hand pose estimation using a single color camera. This work has been funded by the Spanish Government TIN2016-76515-R grant for the COMBAHO project, supported with Feder funds. It has also been supported by the Generalitat Valenciana project GV/2018/22 and by a Spanish grant for PhD studies ACIF/2017/243.

In this work, we carry out a study of the effect of adverse conditions, which characterize real-world scenes, on the accuracy of a Convolutional Neural Network applied to 3D object class recognition. Firstly, we discuss possible ways of representing 3D data to feed the network. In addition, we propose a set of representations to be tested. Those representations consist of a grid-like structure (fixed and adaptive) and a measure for the occupancy of each cell of the grid (binary and normalized point density). After that, we propose and implement a Convolutional Neural Network for 3D object recognition using Caffe. At last, we carry out an in-depth study of the performance of the network over a 3D CAD model dataset, the Princeton ModelNet project, synthetically simulating occlusions and noise models featured by common RGB-D sensors. The results show that the volumetric representations for 3D data play a key role on the recognition process and Convolutional Neural Network can be considerably robust to noise and occlusions if a proper representation is chosen. This work has been supported by the Spanish Government DPI2013-40534-R grant for the SIRMAVED project, also supported with FEDER funds. This work has also been funded by the grant “Ayudas para Estudios de Máster e Iniciación a la Investigación” from the University of Alicante.

Finding an appropriate environment representation is a crucial problem in robotics. 3D data has been recently used thanks to the advent of low cost RGB-D cameras. We propose a new way to represent a 3D map based on the information provided by an expert. Namely, the expert is the output of a Convolutional Neural Network trained with deep learning techniques. Relying on such information, we propose the generation of 3D maps using individual semantic labels, which are associated with environment objects or semantic labels. So, for each label we are provided with a partial 3D map whose data belong to the 3D perceptions, namely point clouds, which have an associated probability above a given threshold. The final map is obtained by registering and merging all these partial maps. The use of semantic labels provide us a with way to build the map while recognizing objects. This work has been supported by the Spanish Government TIN2016-76515-R Grant, supported with Feder funds, and by grant of Vicerrectorado de Investigación y Transferencia de Conocimiento para el fomento de la I+D+i en la Universidad de Alicante 2016.

In this work, we have developed a multisensor system for rehabilitation and interaction with persons with motor and cognitive disabilities. The system enables them to perform different therapies using multiple modes of interaction (head and body pose, hand gestures, voice, touch and gaze) depending on the type and degree of disability. Through a training process, the system can be customized enabling the definition of patients’ own gestures for each sensor. The system is integrated with a range of applications for rehabilitation. Examples of these applications are puzzle solving, mazes and text writing using predictive text tools. The system also provides a flexible and modular framework for the development of new applications oriented towards novel rehabilitation therapies. The proposed system has been integrated in a mobile robotic platform and uses low-cost sensors allowing to perform non-intrusive rehabilitation therapies at home. Videos showing the proposed system and users interacting in different ways (multimodal) are available on our project website www.rovit.ua.es/patente/. This work has been funded by the Spanish Government TIN2016-76515-R grant for the COMBAHO project, supported with Feder funds. This work has also been supported by a Spanish national grant for PhD studies FPU15/04516.

Every year, a significant number of people lose a body part in an accident, through sickness or in high-risk manual jobs. Several studies and research works have tried to reduce the constraints and risks in their lives through the use of technology. This work proposes a learning-based approach that performs gesture recognition using a surface electromyography-based device, the Myo Armband released by Thalmic Labs, which is a commercial device and has eight non-intrusive low-cost sensors. With 35 able-bodied subjects, and using the Myo Armband device, which is able to record data at about 200 MHz, we collected a dataset that includes six dissimilar hand gestures. We used a gated recurrent unit network to train a system that, as input, takes raw signals extracted from the surface electromyography sensors. The proposed approach obtained a 99.90% training accuracy and 99.75% validation accuracy. We also evaluated the proposed system on a test set (new subjects) obtaining an accuracy of 77.85%. In addition, we showed the test prediction results for each gesture separately and analyzed which gestures for the Myo armband with our suggested network can be difficult to distinguish accurately. Moreover, we studied for first time the gated recurrent unit network capability in gesture recognition approaches. Finally, we integrated our method in a system that is able to classify live hand gestures. This work was supported by the Spanish Government TIN2016-76515R grant, supported with Feder funds. It has also been funded by the University of Alicante project GRE16-19, by the Valencian Government project GV/2018/022, and by a Spanish grant for PhD studies ACIF/2017/243.