Search

This paper presents a study on enhancing human–robot interaction (HRI) through multimodal emotional recognition within social robotics. Using the humanoid robot Pepper as a testbed, we integrate visual, auditory, and textual analysis to improve emotion recognition accuracy and contextual understanding. The proposed framework combines pretrained neural networks with fine-tuning techniques tailored to specific users, demonstrating that high accuracy in emotion recognition can be achieved by adapting the models to the individual emotional expressions of each user. This approach addresses the inherent variability in emotional expression across individuals, making it feasible to deploy personalized emotion recognition systems. Our experiments validate the effectiveness of this methodology, achieving high precision in multimodal emotion recognition through fine-tuning, while maintaining adaptability in real-world scenarios. These enhancements significantly improve Pepper’s interactive and empathetic capabilities, allowing it to engage more naturally with users in assistive, educational, and healthcare settings. This study not only advances the field of HRI but also provides a reproducible framework for integrating multimodal emotion recognition into commercial humanoid robots, bridging the gap between research prototypes and practical applications.

A variety of methods for 3D object recognition and registration based on a deep learning pipeline have recently emerged. Nonetheless, these methods require large amounts of data that are not easy to obtain, sometimes rendering them virtually useless in real-life scenarios due to a lack of generalization capabilities. To counter this, we propose a novel local descriptor that takes advantage of the fractal dimension. For each 3D point, we create a descriptor by computing the fractal dimension of the neighbors at different radii. Our redmethod has many benefits, such as being agnostic to the sensor of choice and noise, up to a level, and having few parameters to tinker with. Furthermore, it requires no training and does not rely on semantic information. We test our descriptor using well-known datasets and it largely outperforms Fast Point Feature Histogram, which is the state-of-the-art descriptor for 3D data. We also apply our descriptor to a registration pipeline and achieve accurate three-dimensional representations of the scenes, which are captured with a commercial sensor.

Abstract Human Muscular Manipulability is a metric that measures the comfort of an specific pose and it can be used for a variety of applications related to healthcare. For this reason, we introduce KIMHu: a Kinematic, Imaging and electroMyography dataset for Human muscular manipulability index prediction. The dataset is comprised of images, depth maps, skeleton tracking data, electromyography recordings and 3 different Human Muscular Manipulability indexes of 20 participants performing different physical exercises with their arm. The methodology followed to acquire and process the data is also presented for future replication. A specific analysis framework for Human Muscular Manipulability is proposed in order to provide benchmarking tools based on this dataset.

Convolutional Neural Networks (CNNs) have become the default paradigm for addressing classification problems, especially, but not only, in image recognition. This is mainly due to their high success rate. Although a number of approaches currently apply deep learning to the 3D shape recognition problem, they are either too slow for online use or too error-prone. To fill this gap, we propose 3DSliceLeNet, a deep learning architecture for point cloud classification. Our proposal converts the input point clouds into a two-dimensional representation by performing a slicing process and projecting the points to the principal planes, thus generating images that are used by the convolutional architecture. 3DSliceLeNet successfully achieves both high accuracy and low computational cost. A dense set of experiments has been conducted to validate our system under the ModelNet challenge, a large-scale 3D Computer Aided Design (CAD) model dataset. Our proposal achieves a success rate of 94.37% and an Area under Curve (AUC) of 0.978 on the ModelNet-10 classification task.

In the past years, several works on urban object detection from the point of view of a person have been made. These works are intended to provide an enhanced understanding of the environment for blind and visually challenged people. The mentioned approaches mostly rely in deep learning and machine learning methods. Nonetheless, these approaches only work with direct and bright light, namely, they will only perform correctly on daylight conditions. This is because deep learning algorithms require large amounts of data and the currently available datasets do not address this matter.In this work, we propose UrOAC, a dataset of urban objects captured in a range of different lightning conditions, from bright daylight to low and poor night-time lighting conditions. In the latter, the objects are only lit by low ambient light, street lamps and headlights of passing-by vehicles. The dataset depicts the following objects: pedestrian crosswalks, green traffic lights and red traffic lights. The annotations include the category and the bounding-box of each object.This dataset could be used for improve the performance at night-time and under low-light conditions of any vision-based method that involves urban objects. For instance, guidance and object detection devices for the visually challenged or self-driving and intelligent vehicles.

Currently, state-of-the-art methods for 3D object recognition rely in a deep learning-pipeline. Nonetheless, these methods require a large amount of data that is not easy to obtain. In addition to that, the majority of them exploit features of the datasets, like the fact of being CAD models to create rendered representation which will not work in real life because the 3D sensors provide point clouds. We propose a novel global descriptor for point clouds which takes advantage of the fractal dimension of the objects. Our approach introduces many benefits, such as being agnostic to the density of points of the sample, number of points in the input cloud, sensor of choice, and noise up to a level, and it works on real life point cloud data provided by commercial sensors. We tested our descriptor for 3D object recognition using ModelNet, which is a well-known dataset for that task. Our approach achieves 92.84% accuracy on the ModelNet10, and 88.74% accuracy on the ModelNet40.

As it is well known, some versions of the Pepper robot provide poor depth perception due to the lenses it has in front of the tridimensional sensor. In this paper, we present a method to improving that faulty 3D perception. Our proposal is based on a combination of the actual depth readings of Pepper and a deep learning-based monocular depth estimation. As shown, the combination of both of them provides a better 3D representation of the scene. In previous works we made an initial approximation of this fusion technique, but it had some drawbacks. In this paper we analyze the pros and cons of the Pepper readings, the monocular depth estimation method and our previous fusion method. Finally, we demonstrate that the proposed fusion method outperforms them all.

In this work, we introduce HaReS, a hand rehabilitation system. Our proposal integrates a series of exercises, jointly developed with a foundation for those with motor and cognitive injuries, that are aimed at improving the skills of patients and the adherence to the rehabilitation plan. Our system takes advantage of a low-cost hand-tracking device to provide a quantitative analysis of the performance of the patient. It also integrates a low-cost surface electromyography (sEMG) sensor in order to provide insight about which muscles are being activated while completing the exercises. It is also modular and can be deployed on a social robot. We tested our proposal in two different facilities for rehabilitation with high success. The therapists and patients felt more motivation while using HaReS, which improved the adherence to the rehabilitation plan. In addition, the therapists were able to provide services to more patients than when they used their traditional methodology.

Deep learning-based methods have proven to be the best performers when it comes to object recognition cues both in images and tridimensional data. Nonetheless, when it comes to 3D object recognition, the authors tend to convert the 3D data to images and then perform their classification. However, despite its accuracy, this approach has some issues. In this work, we present a deep learning pipeline for object recognition that takes a point cloud as input and provides the classification probabilities as output. Our proposal is trained on synthetic CAD objects and is able to perform accurately when fed with real data provided by commercial sensors. Unlike most approaches, our method is specifically trained to work on partial views of the objects rather than on a full representation, which is not the representation of the objects as captured by commercial sensors. We trained our proposal with the ModelNet10 dataset and achieved a 78.39 % accuracy. We also tested it by adding noise to the dataset and against a number of datasets and real data with high success.

Every year, a significant number of people lose a body part in an accident, through sickness or in high-risk manual jobs. Several studies and research works have tried to reduce the constraints and risks in their lives through the use of technology. This work proposes a learning-based approach that performs gesture recognition using a surface electromyography-based device, the Myo Armband released by Thalmic Labs, which is a commercial device and has eight non-intrusive low-cost sensors. With 35 able-bodied subjects, and using the Myo Armband device, which is able to record data at about 200 MHz, we collected a dataset that includes six dissimilar hand gestures. We used a gated recurrent unit network to train a system that, as input, takes raw signals extracted from the surface electromyography sensors. The proposed approach obtained a 99.90% training accuracy and 99.75% validation accuracy. We also evaluated the proposed system on a test set (new subjects) obtaining an accuracy of 77.85%. In addition, we showed the test prediction results for each gesture separately and analyzed which gestures for the Myo armband with our suggested network can be difficult to distinguish accurately. Moreover, we studied for first time the gated recurrent unit network capability in gesture recognition approaches. Finally, we integrated our method in a system that is able to classify live hand gestures.