Search

Autonomously exploring and mapping is one of the open challenges of robotics and artificial intelligence. Especially when the environments are unknown, choosing the optimal navigation directive is not straightforward. In this paper, we propose a reinforcement learning framework for navigating, exploring, and mapping unknown environments. The reinforcement learning agent is in charge of selecting the commands for steering the mobile robot, while a SLAM algorithm estimates the robot pose and maps the environments. The agent, to select optimal actions, is trained to be curious about the world. This concept translates into the introduction of a curiosity-driven reward function that encourages the agent to steer the mobile robot towards unknown and unseen areas of the world and the map. We test our approach in explorations challenges in different indoor environments. The agent trained with the proposed reward function outperforms the agents trained with reward functions commonly used in the literature for solving such tasks.

Skin conductance or electrodermal activity (EDA) has been shown to be a reliable indicator of emotional arousal. With the current development to record EDA with wearable sensors attached to individuals who freely move around, detecting EDA artefacts is essential. Current practice of detecting EDA artefacts relies heavily on expert judgments. This is undesirable, especially when facing large amounts of EDA data for which the exact recording context is unknown or for real-time applications. We propose to exploit the similarity in EDA as measured from two hands to determine artefacts in a simple and objective way. The basic idea is that fast EDA fluctuations that are similar in both hands are caused by emotional processes, while dissimilar fluctuations indicate that the sensor at one of the hands suffers from an artefact. This idea was tested in an experiment where participants were asked to make certain movements with their right hand at designated times while not moving the left hand. Besides eliciting movement artefacts in this way, emotional sounds were presented to elicit arousal responses. When both hands were stationary, right-left hand difference signals were flatter compared to raw signals of the left hand, confirming the logic of our approach. We tested first versions of algorithms based on signal synchrony to automatically detect movement artefacts. To evaluate the performance of these algorithms, we did not rely on expert judgment but on the ground truth of no movement artefacts in the stationary epochs. We conclude that two-hand EDA can be valuable for more objective and less labor intensive EDA artefact detection, which is important for EDA studies out of the laboratory.

We introduce a new autonomous path planning algorithm for mobile robots for reaching target locations in an unknown environment where the robot relies on its on-board sensors. In particular, we describe the design and evaluation of a deep reinforcement learning motion planner with continuous linear and angular velocities to navigate to a desired target location based on deep deterministic policy gradient (DDPG). Additionally, the algorithm is enhanced by making use of the available knowledge of the environment provided by a grid-based SLAM with Rao-Blackwellized particle filter algorithm in order to shape the reward function in an attempt to improve the convergence rate, escape local optima and reduce the number of collisions with the obstacles. A comparison is made between a reward function shaped based on the map provided by the SLAM algorithm and a reward function when no knowledge of the map is available. Results show that the required learning time has been decreased in terms of number of episodes required to converge, which is 560 episodes compared to 1450 episodes in the standard RL algorithm, after adopting the proposed approach and the number of obstacle collision is reduced as well with a success ratio of 83% compared to 56% in the standard RL algorithm. The results are validated in a simulated experiment on a skid-steering mobile robot.

Reinforcement Learning has been able to solve many complicated robotics tasks without any need for feature engineering in an end-to-end fashion. However, learning the optimal policy directly from the sensory inputs, i.e the observations, often requires processing and storage of a huge amount of data. In the context of robotics, the cost of data from real robotics hardware is usually very high, thus solutions that achieve high sample-efficiency are needed. We propose a method that aims at learning a mapping from the observations into a lower-dimensional state space. This mapping is learned with unsupervised learning using loss functions shaped to incorporate prior knowledge of the environment and the task. Using the samples from the state space, the optimal policy is quickly and efficiently learned. We test the method on several mobile robot navigation tasks in a simulation environment and also on a real robot., Paper Accepted at ICPR2020

Most of the research in automated driving currently involves using the on-board sensors on the vehicle to collect information regarding surrounding vehicles to maneuver around them. In this paper we discuss how information communicated through vehicular networking can be used for controlling an autonomous vehicle in a multi-lane highway environment. A driving algorithm is designed using deep Q learning, a type of reinforcement learning. In order to train and test driving algorithms, we deploy a simulated traffic system, using SUMO (Simulation of Urban Mobility). The performance of the driving algorithm is tested for perfect knowledge regarding surrounding vehicles. Furthermore, the impact of limited communication range and random packet loss is investigated. Currently the performance of the driving algorithm is far from ideal with the collision ratios being quite high. We propose directions for additional research to improve the performance of the algorithm.

Autonomous navigation of robots in unknown environments from their current position to a desired target without colliding with obstacles represents an important aspect in the field of mobile robots. In literature, traditional methods do exist in case a complete knowledge of the environment is available. However, this is not the case in real-life applications where a complete knowledge about stochastic environments can hardly be obtained where the positions of the obstacles are unknown. Our main goal is to navigate a skid-steering mobile robot (SSMR) with non-holonomic constraints in an unknown environment to its desired target in minimum time while avoiding colliding with the obstacles. In the context of autonomous cart navigation, autonomous navigation of mobile robots in unstructured environments can be formulated as a Reinforcement learning (RL) problem. Reinforcement learning is a paradigm in which the agent (robot) learns its optimal path by interacting with the environment and receiving a reward based on its actions. Based on this action-reward scenario, the optimal action for each state can be discovered by maximizing a predefined accumulated reward that reflects the quality of the trajectory taken by the robot. Reinforcement learning can be categorized into two main methods (Bagnell J. Peters J. Kober, 2013), namely value based methods (Q-learning, Deep Q-learning) and policy based methods (REINFORCE with Policy Gradient). (i) Value based methods: In these methods, the value function that maps each state-action pair into a value is learned. Thanks to these methods, the best action to take for each state, the action with the biggest value, can be found. This works well in case of a finite set of actions. (ii) Policy based methods: In policy based methods, instead of learning value functions that give an indication of the expected total reward for each state-action pair, the policy function that maps the state into action is optimized directly (Brundage. M Arulkumaran. K and A., 2017). Policy search methods are more effective in high dimensional action spaces, or when using continuous actions like the case of mobile robot navigation (Neumann G. Peters J. Deisenroth, 2013). However, since both of these methods have their own drawbacks, there is hybrid method called Actor-Critic that employs both value functions and policy search (Brundage. M Arulkumaran. K and A., 2017). In this method, the actor adjust the policy parameters by a policy gradient ascent whereas the critic estimates the action value function using a policy evaluation algorithm such as temporal difference (TD) learning. This can be done through two neural networks running in parallel. Herein, we introduce a reinforcement learning based navigation system that enhance the navigation capabilities of a skid steering mobile robot. This goal will be achieved by using asynchronous deep deterministic policy gradient (ADDPG) through an actor-critic algorithm (Badia A. Mirza M. Graves A. Harley T. Lillicrap T. Silver D. Kavukcuoglu K. Mnih, 2016). Two deep neural-networks are introduced in this case. The first one (actornetwork) transfers the input vector that represents the states of the robot to linear and angular velocity commands that represent the actions. Whereas the second neural network (critic-network) is responsible for predicting the Q-value of the state and action pairs. Deterministic policies will be utilized since it was proposed that they outperform their stochastic counterparts in highdimensional action spaces (Lever G. Hess N. Degris T. Wierstra D. Riedmiller M. Silver, 2014). On the other hand, Simultaneous Localization and Mapping also known as SLAM techniques are to be integrated with reinforcement learning in an attempt to improve the learning rate by providing more accurate estimation of the robots states. SLAM has been an active topic for many years (C. Cadena, 2016) because it provides two fundamental components to robotics: where I am, and what I see. Herein, we introduce how the navigation problem of non-holonomic mobile robots can be formulated as a reinforcement learning problem that could be solved by using ADDPG actor-critic algorithm. Besides, we share the experimental results on how SLAM would help reinforcement learning at all by comparing the learning rate without SLAM (comparison when a partial map of the environment is given and when no map is available). For implementation of the parallel running neural networks, we do our initial experiments using the GPU cluster of University of Twente. However, when it comes to real-life implementations and demonstrations on real robots, the hardware becomes the key component to determine whether the goal can be achieved or not. Therefore, we implement the trained networks on NVIDIA Jetson TX2 in order to show real-life implementation possibilities when the algorithms need to run in a navigating robot (Jetson TX2 Module, online documentation, n.d.). In this sense, NVIDIA Jetson TX2 gives opportunity to run even multiple complex neural networks real-time on a credic card size processor unit which can easily be carried by a small (less than 20cm length) robot. We show our experiments on sensor selection and the OpenAI package in ROSGazebo platform as a possible simulation environment. Last but not least, we discuss the real-life implementation scenarios using our NVIDIA Jetson TX2 and how realistic the simulation results could be when the algorithms are implemented on the real robot. The early experiments indicate that, for an optimal design of a SLAM and reinforcement learning based system to truly work in real-world applications in order to reach a goal in an unknown environment, the hardware setup and the algorithms must be collaboratively designed.

Background: Most people experience low back pain (LBP) at least once in their life and for some patients this evolves into a chronic condition. One way to prevent acute LBP from transiting into chronic LBP, is to ensure that patients receive the right interventions at the right moment. We started research in the design of a clinical decision support system (CDSS) to support patients with LBP in their self-referral to primary care. For this, we explored the possibilities of using supervised machine learning. We compared the performances of the three classification models -i.e. 1. decision tree, 2. random forest, and 3. boosted tree - to get insight in which model performs best and whether it is already acceptable to use this model in real practice.Methods: The three models were generated by means of supervised machine learning with 70% of a training dataset (1288 cases with 65% GP, 33% physio, 2% self-care cases). The cases in the training dataset were fictive cases on low back pain collected during a vignette study with primary healthcare professionals. We also wanted to know the performance of the models on real-life low back pain cases that were not used to train the models. Therefore we also collected real-life cases on low back pain as test dataset. These cases were collected with the help of patients and healthcare professionals in primary care. For each model, the performance was measured during model validation - with 30% of the training dataset -as well as during model testing -with the test dataset containing real-life cases. The total observed accuracy as well as the kappa, and the sensitivity, specificity, and precision were used as performance measures to compare the models.Results: For the training dataset, the total observed accuracies of the decision tree, the random forest and boosted tree model were 70%, 69%, and 72% respectively. For the test dataset, the total observed accuracies were 71%, 53%, and 71% respectively. The boosted tree appeared to be the best for predicting a referral advice with a fair accuracy (Kappa between 0.2 and 0.4). Next to this, the measured evaluation measures show that all models provided a referral advice better than just a random guess. This means that all models learned some implicit knowledge of the provided referral advices in the training dataset.Conclusions: The study showed promising results on the possibility of using machine learning in the design of our CDSS. The boosted tree model performed best on the classification of low back pain cases, but still has to be improved. Therefore, new cases have to be collected, especially cases that are classified as self-care cases. This to be sure that also the self-care advice can be predicted well by the model.

In electronic health (eHealth) research, limited insight has been obtained on process outcomes or how the use of technology has contributed to the users' ability to have a healthier life, improved well-being, or activate new attitudes in their daily tasks. As a result, eHealth is often perceived as a black box. To open this black box of eHealth, methodologies must extend beyond the classic effect evaluations. The analyses of log data (anonymous records of real-time actions performed by each user) can provide continuous and objective insights into the actual usage of the technology. However, the possibilities of log data in eHealth research have not been exploited to their fullest extent. The aim of this paper is to describe how log data can be used to improve the evaluation and understand the use of eHealth technology with a broader approach than only descriptive statistics. This paper serves as a starting point for using log data analysis in eHealth research. Here, we describe what log data is and provide an overview of research questions to evaluate the system, the context, the users of a technology, as well as the underpinning theoretical constructs. We also explain the requirements for log data, the starting points for the data preparation, and methods for data collection. Finally, we describe methods for data analysis and draw a conclusion regarding the importance of the results for both scientific and practical applications. The analysis of log data can be of great value for opening the black box of eHealth. A deliberate log data analysis can give new insights into how the usage of the technology contributes to found effects and can thereby help to improve the persuasiveness and effectiveness of eHealth technology and the underpinning behavioral models.

Brain-computer interfaces (BCIs) are often evaluated in terms of performance and seldom for usability. However in some application domains, such as entertainment computing, user experience evaluation is vital. User experience evaluation in BCI systems, especially in entertainment applications such as games, can be biased due to the novelty of the interface. However, as the novelty will eventually vanish, what matters is the user experience related to the unique features offered by BCI. Therefore it is a viable approach to compare BCI to other novel modalities, such as a speech or motion recognizer, rather than the traditional mouse and keyboard. In the study that we present in this article, our participants played a computer game with a BCI and an automatic speech recognizer (ASR), and they rated their expectations and experiences for both modalities. Our analysis on subjective ratings revealed that both ASR and BCI were successful in satisfying participants' expectations in general. Participants found speech control easier to learn than BCI control. They indicated that BCI control induced more fatigue than they expected.

Background: In eHealth research, limited insights have been obtained on process outcomes or how the use of technology has contributed to the users’ ability to have a healthier life, improved wellbeing, or activate new attitudes in their daily tasks. As a result, eHealth is often perceived as a black box. To open this Black Box of eHealth, methodologies must extend beyond the classic effect evaluations. The analyses of log data (anonymous records of real-time actions performed by each user) can provide continuous and objective insights into the actual usage of the technology. However, until now the possibilities of log data in eHealth research has not been exploited to its fullest extent. Objectives: The aim of this paper is to describe how log data can be used to improve the evaluation and understand the use of an eHealth technology with a broader approach than only descriptive statistics. This paper serves as a starting point for using log data analysis in eHealth research. Methods: First, we describe what log data is and an overview is given of research questions to evaluate the system, the context, the users of a technology as well as the underpinning theoretical constructs. Secondly, requirements for log data, the starting points for the data preparation and methods for data collection are explained. Results: In the third part, some methods for data analysis are described. Finally, a conclusion is drawn regarding the importance of the results for both scientific and practical applications. Conclusion: The analysis of log data can be of great value for opening the black box of eHealth. A deliberate log data analysis can give new insights into how the usage of the technology contributed to the found effects and can thereby help to improve the persuasiveness and effectiveness of the eHealth technology and the underpinning behavioral models. Keywords: eHealth, black box, evaluation, log data analysis

Social robots should be able to automatically understand and respond to human touch. The meaning of touch does not only depend on the form of touch but also on the context in which the touch takes place. To gain more insight into the factors that are relevant to interpret the meaning of touch within a social context we elicited touch behaviors by letting participants interact with a robot pet companion in the context of different affective scenarios. In a contextualized lab setting, participants (n = 31) acted as if they were coming home in different emotional states (i.e., stressed, depressed, relaxed, and excited) without being given specific instructions on the kinds of behaviors that they should display. Based on video footage of the interactions and interviews we explored the use of touch behaviors, the expressed social messages, and the expected robot pet responses. Results show that emotional state influenced the social messages that were communicated to the robot pet as well as the expected responses. Furthermore, it was found that multimodal cues were used to communicate with the robot pet, that is, participants often talked to the robot pet while touching it and making eye contact. Additionally, the findings of this study indicate that the categorization of touch behaviors into discrete touch gesture categories based on dictionary definitions is not a suitable approach to capture the complex nature of touch behaviors in less controlled settings. These findings can inform the design of a behavioral model for robot pet companions and future directions to interpret touch behaviors in less controlled settings are discussed.

Health care insurance fraud is a pressing problem, causing substantial and increasing costs in medical insurance programs. Due to large amounts of claims submitted, estimated at 5 billion per day, review of individual claims or providers is a difficult task. This encourages the employment of automated pre-payment controls and better post-payment decision support tools to enable subject matter expert analysis. This paper presents how to apply unsupervised outlier techniques at post-payment stage to detect fraudulent patterns of received insurance claims. A special emphasis in this paper is put on the system architecture, the metrics designed for outlier detection and the flagging of suspicious providers which may support the fraud experts in evaluating providers and reveal fraud. The algorithms were tested on Medicaid data encompassing 650,000 health-care claims and 369 dentists of one state. Two health care fraud experts evaluated flagged cases and concluded that 12 of the top 17 providers (71%) submitted suspicious claim patterns and should be referred to officials for further investigation. The remaining 5 providers (29%) could be considered mis-classifications as their patterns could be explained by special characteristics of the provider. Selecting top flagged providers is demonstrated to be a valuable as an targeting method, and individual provider analysis revealed some cases of potential fraud. The study concludes that, through outlier detection, new patterns of potential fraud can be identified and possibly utilized in future automated detection mechanisms.

Item does not contain fulltext A brain-computer interface (BCI) enables direct communication from the brain to devices, bypassing the traditional pathway of peripheral nerves and muscles. Traditional approaches to BCIs require the user to train for weeks or even months to learn to control the BCI. In contrast, BCIs based on machine learning only require a calibration session of less than an hour before the system can be used, since the machine adapts to the user's existing brain signals. However, this calibration session has to be repeated before each use of the BCI due to inter-session variability, which makes using a BCI still a time-consuming and an error-prone enterprise. In this work, we present a second-order baselining procedure that reduces these variations, and enables the creation of a BCI that can be applied to new subjects without such a calibration session. The method was validated with a motor-imagery classification task performed by 109 subjects. Results showed that our subject-independent BCI without calibration performs as well as the popular common spatial patterns (CSP)-based BCI that does use a calibration session.

Currently the field of brain–computer interfacing is increasingly focused on developing usable brain–computer interfaces (BCIs) to better ensure technology transfer and acceptance. Many studies have investigated the usability of BCI applications as a whole. Here we aim to investigate one specific component of an electroencephalogram (EEG)-based BCI system: the acquisition component. This study compares on the usability of three different EEG headsets in the context of a P300-based BCI application for communication. Thirteen participants took part in a within-subject experiment. Participants were randomly given a Biosemi, Emotiv EPOC or g.Sahara headset. After every session offline classification accuracy (efficacy) was calculated and usability factors (perceived efficiency and user satisfaction) were measured using questionnaires. The 32-channel Biosemi headset offered the highest accuracy (88.5%) compared with the 8-channel g.Sahara (62.7%) and the 14-channel Emotiv (61.7%). There was no difference in accuracy between the Biosemi and the g.Sahara when comparing the same 8 channels. The Biosemi and g.Sahara were rated as more comfortable than the Emotiv. The Emotiv was rated as best for aesthetics. System setup time was highest for the Biosemi headset when compared with the g.Sahara and the Emotiv. Without information about the effectiveness, participants preferred the Emotiv. We recommend the use of a gelled headset for applications which require high accuracy and efficiency and water-based or dry headsets when aesthetics, easy setup and fun are important.

The multimodal, multi-paradigm brain-computer interfacing (BCI) game Bacteria Hunt was used to evaluate two aspects of BCI interaction in a gaming context. One goal was to examine the effect of feedback on the ability of the user to manipulate his mental state of relaxation. This was done by having one condition in which the subject played the game with real feedback, and another with sham feedback. The feedback did not seem to affect the game experience (such as sense of control and tension) or the objective indicators of relaxation, alpha activity and heart rate. The results are discussed with regard to clinical neurofeedback studies. The second goal was to look into possible interactions between the two BCI paradigms used in the game: steady-state visually-evoked potentials (SSVEP) as an indicator of concentration, and alpha activity as a measure of relaxation. SSVEP stimulation activates the cortex and can thus block the alpha rhythm. Despite this effect, subjects were able to keep their alpha power up, in compliance with the instructed relaxation task. In addition to the main goals, a new SSVEP detection algorithm was developed and evaluated. © 2010 The Author(s).