Search

Falling down is a serious problem for health and has become one of the major etiologies of accidental death for the elderly living alone. In recent years, many efforts have been paid to fall recognition based on wearable sensors or standard vision sensors. However, the prior methods have the risk of privacy leaks, and almost all these methods are based on video clips, which cannot localize where the falls occurred in long videos. For these reasons, in this article, the bioinspired vision sensor-based falls temporal localization framework is proposed. The bioinspired vision sensors, such as dynamic and active-pixel vision sensor (DAVIS) camera applied in this work responds to pixels' brightness change, and each pixel works independently and asynchronously compared to the standard vision sensors. This property makes it have a very high dynamic range and privacy preserving. First, to better represent event data, compared with the typical constant temporal window mechanism, an adaptive temporal window conversion mechanism is developed. The temporal localization framework follows a proven proposal and classification paradigm. Second, for the high-efficient and recall proposal generation, different from the traditional sliding window scheme, the event temporal density as the actionness score is set and the 1D-watershed algorithm to generate proposals is applied. In addition, we combine the temporal and spatial attention mechanism with our feature extraction network to temporally model the falls. Finally, to evaluate the performance of our framework, 30 volunteers are recruited to join the simulated fall experiments. According to the results of experiments, our framework can realize precise falls temporal localization and achieve the state-of-the-art performance.

Agile quadrotor flight in challenging environments has the potential to revolutionize shipping, transportation, and search and rescue applications. Nonlinear model predictive control (NMPC) has recently shown promising results for agile quadrotor control, but relies on highly accurate models for maximum performance. Hence, model uncertainties in the form of unmodeled complex aerodynamic effects, varying payloads and parameter mismatch will degrade overall system performance. In this paper, we propose L1-NMPC, a novel hybrid adaptive NMPC to learn model uncertainties online and immediately compensate for them, drastically improving performance over the non-adaptive baseline with minimal computational overhead. Our proposed architecture generalizes to many different environments from which we evaluate wind, unknown payloads, and highly agile flight conditions. The proposed method demonstrates immense flexibility and robustness, with more than 90% tracking error reduction over non-adaptive NMPC under large unknown disturbances and without any gain tuning. In addition, the same controller with identical gains can accurately fly highly agile racing trajectories exhibiting top speeds of 70 km/h, offering tracking performance improvements of around 50% relative to the non-adaptive NMPC baseline., 8 Pages, 6 figures, Accepted RAL 2021

Circulant matrices are an important tool widely used in coding theory and cryptography. A circulant matrix is a square matrix whose rows are the cyclic shifts of the first row. Such a matrix can be efficiently stored in memory because it is fully specified by its first row. The ring of $n \times n$ circulant matrices can be identified with the quotient ring $\mathbb{F}[x]/(x^n-1)$. In consequence, the strong algebraic structure of the ring $\mathbb{F}[x]/(x^n-1)$ can be used to study properties of the collection of all $n\times n$ circulant matrices. The ring $\mathbb{F}[x]/(x^n-1)$ is a special case of a group algebra and elements of any finite dimensional group algebra can be represented with square matrices which are specified by a single column. In this paper we study this representation and prove that it is an injective Hamming weight preserving homomorphism of $\mathbb{F}$-algebras and classify it in the case where the underlying group is abelian. Our work is motivated by the desire to generalize the BIKE cryptosystem (a contender in the NIST competition to get a new post-quantum standard for asymmetric cryptography). Group algebras can be used to design similar cryptosystems or, more generally, to construct low density or moderate density parity-check matrices for linear codes., Comment: A shortened version was submitted to MTNS 2022

Soft wearable robots to assist human movements, such as exosuits, have rapidly gained attention thanks to their compliance, low weight and accessibility. However, force measurement in exosuits still rely on load cells and rigid sensors that are not wearable or unsuitable for applications outside the lab. Soft, stretchable and lightweight sensors that become invisible when integrated in an exosuit and perfectly conform to the human body represent a promising alternative. In this work, we developed a wearable sensing system based on a soft stretchable silicone-based strain gauge to measure the forces acting in the passive elastic elements of an exosuit. To measure sensor's accuracy, two unimpaired participants walked on a treadmill at speeds between 0.9 and 2.1 $\text{m}\text{s}^{-1}$. When comparing our solution to a state-of-the-art motion capture system, we found an average root mean square error in force estimation of 12.5% and a standard deviation of 7.4%. Furthermore, we showed the portability of our sensory system by monitoring the forces exerted by the wearable robot during outdoor walking. Our study shows the potential of using stretchable sensors to monitor walking patterns in studies outside the lab and to control human-robot interaction.

Currently, therapists struggle with interaction of rehabilitation robots due to non-intuitive interfaces. Therefore their acceptance of these robots are limited. This paper presents the development of ARMStick, a lightweight and small robotic interface in the shape of a human arm with 4 actuated and 3 unactuated joints, to facilitate the interaction between therapists and rehabilitation robots. It allows therapists to intuitively perceive joint-dependent data as recorded by rehabilitation robots, and teach poses and trajectories to individualize therapy to the patient. It's range of motion (RoM) covers the RoM of a healthy human. The device's measuring accuracy of CI 95% $ \lt \pm 0.322^{\circ}$ and movement accuracy of CI 70% $ \lt \pm 5.23^{\circ}$ lie within the confidence interval of average visual perception. A demonstration of the device to 5 therapists indicated that it could indeed improve efficiency and efficacy bottlenecks in current robot-assisted therapy. Comparison of ARMStick to two visual user interfaces showed a decrease in mean adaptation time from 15s to 5s for three arm configurations presented to the therapists.

Hand gestures are one of the most natural and expressive way for humans to convey information, and thus hand gesture recognition has become a research hotspot in the human-machine interface (HMI) field. In particular, biological signals such as surface electromyography (sEMG) can be used to recognize hand gestures to implement intuitive control systems, but the decoding from the sEMG signal to actual control signals is non-trivial. Blind source separation (BSS)-based methods, such as convolutive independent component analysis (ICA), can be used to decompose the sEMG signal into its fundamental elements, the motor unit action potential trains (MUAPTs), which can then be processed with a classifier to predict hand gestures. However, ICA does not guarantee a consistent ordering of the extracted motor units (MUs), which poses a problem when considering multiple recording sessions and subjects; therefore, in this work we propose and validate three approaches to address this variability: two ordering criteria based on firing rate and negative entropy, and a re-calibration procedure, which allows the decomposition model to retain information about previous recording sessions when decomposing new data. In particular, we show that re-calibration is the most robust approach, yielding an accuracy up to 99.4%, and always greater than 85% across all the different scenarios that we tested. These results prove that our proposed system, which we publish open-source and which is based on biologically plausible features rather than on data-driven, black-box models, is capable of robust generalization.

Reliable detection of sleep positions is essential for the development of technical aids for patients with position-dependent sleep-related breathing disorders. We compare personalized and generalizable sleeping position classifiers using unobtrusive eight-channel pressure-sensing mats. Data of six male patients with confirmed position-dependent sleep apnea was recorded during three subsequent nights. Personalized position classifiers trained using leave-one-night-out cross-validation on average reached an F1-score of 61.3% for supine/non-supine and an F1-score of 46.2% for supine/lateral-left/lateral-right classification. The generalizable classifiers reached average F1-scores of 62.1% and 49.1% for supine/non-supine and supine/lateral-left/lateral-right classification, respectively. In-bed presence ('bed occupancy') could be detected with an average F1-score of 98.1%. This work shows that personalized sleep-position classifiers trained with data from two nights have comparable performance to classifiers trained with large interpatient datasets. Simple eight-channel sensor mattresses can be used to accurately detect in-bed presence required for closed-loop systems but their use to classify sleep-positions is limited.

Exoskeletons operate in continuous haptic interaction with a human limb. Thus, this interaction is a key factor to consider during the development of hardware and control policies for these devices. Physics simulations can complement real-world experiments for prototype validation, leading to higher efficiency in hardware and software development iterations as well as increased safety for participants and robot hardware. Here, we present a simulation framework of the full rigid-body dynamics of a coupled human and exoskeleton arm built to validate the full software stack. We present a method to model the human-robot interaction dynamics as decoupled spring-damper systems based on anthropometric data. Further, we demonstrate the application of the simulation framework to predict the closed-loop haptic-rendering performance of a 9-DOF exoskeleton in interaction with a human. The simulation was capable of simulating the closed-loop system's reaction to an impact on a haptic wall. The intrusion into the compliant walls was predicted with a relative accuracy of 6% to 13%. Admissible control gains could be predicted with an accuracy of around 14%, and higher prediction accuracy is indicated when modeling the torque tracking bandwidth of the actuators. Hence, the simulation is valuable for validating prototype software, developing intuition, and a better understanding of the complex characteristics of the coupled system dynamics, even though the quantitative prediction is limited.

Children affected by hand impairment due to cerebral palsy or stroke experience serious difficulties when performing activities of daily life (ADL), which reduces their quality of life and development. Wearable robots such as hand exoskeletons have been proposed to support people with hand impairment in therapy as well as daily tasks. While numerous actuated wearable robots have been developed, few designs support both fingers and wrist function, despite being mutually relevant for reach-to-grasp tasks. A recent feasibility study investigating the use of PEXO, a lightweight and fully wearable pediatric hand exoskeleton, showed that a wrist fixed in a slightly extended position may limit the user's ability to reach and grasp during ADL and restrict the user group. These insights and further interactions with clinicians inspired a novel design of PEXO that features an additional degree of freedom in the wrist. In this paper, we present a compliant wrist mechanism extending the existing leaf spring finger mechanism of the device. The novel design provides both wrist motion capability of 60° in flexion and extension and wrist stabilization at the same time while actively supporting finger motion. Preliminary results suggest that the adjustability in the wrist enables a larger variety of grasping gestures. The implemented wrist support has the potential to allow for a more versatile use of PEXO and increase the potential target user group.

Fault-tolerant control is crucial for safety-critical systems, such as quadrotors. State-of-art flight controllers can stabilize and control a quadrotor even when subjected to the complete loss of a rotor. However, these methods rely on external sensors, such as GPS or motion capture systems, for state estimation. To the best of our knowledge, this has not yet been achieved with only onboard sensors. In this paper, we propose the first algorithm that combines fault-tolerant control and onboard vision-based state estimation to achieve position control of a quadrotor subjected to complete failure of one rotor. Experimental validations show that our approach is able to accurately control the position of a quadrotor during a motor failure scenario, without the aid of any external sensors. The primary challenge to vision-based state estimation stems from the inevitable high-speed yaw rotation (over 20 rad/s) of the damaged quadrotor, causing motion blur to cameras, which is detrimental to visual inertial odometry (VIO). We compare two types of visual inputs to the vision-based state estimation algorithm: standard frames and events. Experimental results show the advantage of using an event camera especially in low light environments due to its inherent high dynamic range and high temporal resolution. We believe that our approach will render autonomous quadrotors safer in both GPS denied or degraded environments. We release both our controller and VIO algorithm open source., Comment: 8 pages, 10 figures

Neural networks predictions are unreliable when the input sample is out of the training distribution or corrupted by noise. Being able to detect such failures automatically is fundamental to integrate deep learning algorithms into robotics. Current approaches for uncertainty estimation of neural networks require changes to the network and optimization process, typically ignore prior knowledge about the data, and tend to make over-simplifying assumptions which underestimate uncertainty. To address these limitations, we propose a novel framework for uncertainty estimation. Based on Bayesian belief networks and Monte-Carlo sampling, our framework not only fully models the different sources of prediction uncertainty, but also incorporates prior data information, e.g. sensor noise. We show theoretically that this gives us the ability to capture uncertainty better than existing methods. In addition, our framework has several desirable properties: (i) it is agnostic to the network architecture and task; (ii) it does not require changes in the optimization process; (iii) it can be applied to already trained architectures. We thoroughly validate the proposed framework through extensive experiments on both computer vision and control tasks, where we outperform previous methods by up to 23% in accuracy., Accepted for publication in the Robotics and Automation Letters 2020, and for presentation at the International Conference on Robotics and Automation (ICRA) 2020

Multirotor aerial robots are becoming widely used for the inspection of powerlines. To enable continuous, robust inspection without human intervention, the robots must be able to perch on the powerlines to recharge their batteries. Highly versatile perching capabilities are necessary to adapt to the variety of configurations and constraints that are present in real powerline systems. This paper presents a novel perching trajectory generation framework that computes perception-aware, collision-free, and dynamically-feasible maneuvers to guide the robot to the desired final state. Trajectory generation is achieved via solving a Nonlinear Programming problem using the Primal-Dual Interior Point method. The problem considers the full dynamic model of the robot down to its single rotor thrusts and minimizes the final pose and velocity errors while avoiding collisions and maximizing the visibility of the powerline during the maneuver. The generated maneuvers consider both the perching and the posterior recovery trajectories. The framework adopts costs and constraints defined by efficient mathematical representations of powerlines, enabling online onboard execution in resource-constrained hardware. The method is validated on-board an agile quadrotor conducting powerline inspection and various perching maneuvers with final pitch values of up to 180 degrees. The developed code is available online at: https://github.com/grvcPerception/pa_powerline_perching, IEEE Robotics and Automation Letters (2022)

This MDPI open-access journal article is an enhanced version of the paper published previously with LCN 2021. This work implemented a TinyIPFIX platform using Espressif ESP-WROOM-32D devices. TinyIPFIX was selected as a wireless sensor network (WSN) data transport mechanism. As soon as data arrives at the sink, TinyIPFIX messages are provided to the Publish/Subscribe (Pub/Sub) engine, implemented with the help of the Zero Message Queue (ZMQ) message broker. Finally, two applications using the message broker were implemented. Two ESP-WROOM-32D devices were chosen as the hardware platform: Espressif ESP32 DevKitC V4 and Macchina SuperB\footnote. These devices were pre-paired for programming using MicroPython. Then, each ESP device was equipped with a Digi XBee board, which features the IEEE 802.15.4 standard, allowing for low-power communication among devices in the Wireless Sensor Network (WSN). Furthermore, all network components were implemented with the help of MicroPython (i.e., TinyIPFIX end devices and TinyIPFIX concentrators) or Python (i.e., the TinyIPFIX collector). Finally, the energy consumption of those devices running TinyIPFIX was evaluated successfully since the primary method of reducing the energy consumption in the WSN is to leave devices as long as possible in deep sleep mode (i.e., ESP32 and XBee devices). In conclusion, TinyIPFIX maintains a more negligible data overhead in specific application scenarios than regular type-length-value (TLV) data transfer. Furthermore, it was demonstrated experimentally that the Python-based TinyIPFIX works well in a home-based IEEE 802.15.4 network, providing almost a 100\% delivery ratio.Thus, this solution offers a uniform Python-based implementation spanning multiple elements of the system, including TinyIPFIX end devices, concentrators, and the collector, as well as the ZMQ broker and applications residing on the application server (i.e., the collector). The Python-based environment provides much faster value creation than older systems, depending on low-level programming languages. Soon, different improvements will be considered. A time synchronization-based solution will allow concentrators to enter deep sleep mode without losing data packets from end devices. As a result, this will reduce the energy consumed by concentrators. Additionally, the elevated deep sleep current currently experienced on ESP32 devices (e.g., 20 mW on the SuperB device in milliamperes) has to be decreased, allowing for an extensive network life span. &nbsp

Humans race drones faster than algorithms, despite being limited to a fixed camera angle, body rate control, and response latencies in the order of hundreds of milliseconds. A better understanding of the ability of human pilots of selecting appropriate motor commands from highly dynamic visual information may provide key insights for solving current challenges in vision-based autonomous navigation. This paper investigates the relationship between human eye movements, control behavior, and flight performance in a drone racing task. We collected a multimodal dataset from 21 experienced drone pilots using a highly realistic drone racing simulator, also used to recruit professional pilots. Our results show task-specific improvements in drone racing performance over time. In particular, we found that eye gaze tracks future waypoints (i.e., gates), with first fixations occurring on average 1.5 seconds and 16 meters before reaching the gate. Moreover, human pilots consistently looked at the inside of the future flight path for lateral (i.e., left and right turns) and vertical maneuvers (i.e., ascending and descending). Finally, we found a strong correlation between pilots eye movements and the commanded direction of quadrotor flight, with an average visual-motor response latency of 220 ms. These results highlight the importance of coordinated eye movements in human-piloted drone racing. We make our dataset publicly available., Comment: 8 pages, 6 figures

Autonomous car racing is a major challenge in robotics. It raises fundamental problems for classical approaches such as planning minimum-time trajectories under uncertain dynamics and controlling the car at the limits of its handling. Besides, the requirement of minimizing the lap time, which is a sparse objective, and the difficulty of collecting training data from human experts have also hindered researchers from directly applying learning-based approaches to solve the problem. In the present work, we propose a learning-based system for autonomous car racing by leveraging a high-fidelity physical car simulation, a course-progress proxy reward, and deep reinforcement learning. We deploy our system in Gran Turismo Sport, a world-leading car simulator known for its realistic physics simulation of different race cars and tracks, which is even used to recruit human race car drivers. Our trained policy achieves autonomous racing performance that goes beyond what had been achieved so far by the built-in AI, and, at the same time, outperforms the fastest driver in a dataset of over 50,000 human players., Comment: Accepted for Publication at the IEEE Robotics and Automation Letters (RA-L) 2021, and International Conference on Robots and Automation (ICRA) 2021

It is well known that there is a correspondence between convolutional codes and discrete-time linear systems over finite fields. In this paper, we employ the linear systems representation of a convolutional code to develop a decoding algorithm for convolutional codes over the erasure channel. In this kind of channel, which is important due to its use for data transmission over the Internet, the receiver knows if a received symbol is correct. We study the decoding problem using the state space description of a convolutional code, and this provides in a natural way additional information. With respect to previously known decoding algorithms, our new algorithm has the advantage that it is able to reduce the decoding delay as well as the computational effort in the erasure recovery process. We describe which properties a convolutional code should have in order to obtain a good decoding performance and illustrate it with an example.

Traditional approaches to stereo visual simultaneous localization and mapping (SLAM) rely on point features to estimate the camera trajectory and build a map of the environment. In low-textured environments, though, it is often difficult to find a sufficient number of reliable point features and, as a consequence, the performance of such algorithms degrades. This paper proposes PL-SLAM, a stereo visual SLAM system that combines both points and line segments to work robustly in a wider variety of scenarios, particularly in those where point features are scarce or not well-distributed in the image. PL-SLAM leverages both points and line segments at all the instances of the process: visual odometry, keyframe selection, bundle adjustment, etc. We contribute also with a loop-closure procedure through a novel bag-of-words approach that exploits the combined descriptive power of the two kinds of features. Additionally, the resulting map is richer and more diverse in three-dimensional elements, which can be exploited to infer valuable, high-level scene structures, such as planes, empty spaces, ground plane, etc. (not addressed in this paper). Our proposal has been tested with several popular datasets (such as EuRoC or KITTI), and is compared with state-of-the-art methods such as ORB-SLAM2, revealing a more robust performance in most of the experiments while still running in real time. An open-source version of the PL-SLAM C++ code has been released for the benefit of the community.

In this letter, we study the effects that perception latency has on the maximum speed a robot can reach to safely navigate through an unknown cluttered environment. We provide a general analysis that can serve as a baseline for future quantitative reasoning for design tradeoffs in autonomous robot navigation. We consider the case where the robot is modeled as a linear second-order system with bounded input and navigates through static obstacles. Also, we focus on a scenario where the robot wants to reach a target destination in as little time as possible, and therefore cannot change its longitudinal velocity to avoid obstacles. We show how the maximum latency that the robot can tolerate to guarantee safety is related to the desired speed, the range of its sensing pipeline, and the actuation limitations of the platform (i.e., the maximum acceleration it can produce). As a particular case study, we compare monocular and stereo frame-based cameras against novel, low-latency sensors, such as event cameras, in the case of quadrotor flight. To validate our analysis, we conduct experiments on a quadrotor platform equipped with an event camera to detect and avoid obstacles thrown towards the robot. To the best of our knowledge, this is the first theoretical work in which perception and actuation limitations are jointly considered to study the performance of a robotic platform in high-speed navigation.

Data analysis in high-dimensional spaces aims at obtaining a synthetic description of a data set, revealing its main structure and its salient features. We here introduce an approach providing this description in the form of a topography of the data, namely a human-readable chart of the probability density from which the data are harvested. The approach is based on an unsupervised extension of Density Peak clustering and a non-parametric density estimator that measures the probability density in the manifold containing the data. This allows finding automatically the number and the height of the peaks of the probability density, and the depth of the "valleys" separating them. Importantly, the density estimator provides a measure of the error, which allows distinguishing genuine density peaks from density fluctuations due to finite sampling. The approach thus provides robust and visual information about the density peaks' height, their statistical reliability, and their hierarchical organization, offering a conceptually powerful extension of the standard clustering partitions. We show that this framework is particularly useful in the analysis of complex data sets., There is a Supplementary Information document in the ancillary files folder

Once an academic venture, autonomous driving has received unparalleled corporate funding in the last decade. Still, the operating conditions of current autonomous cars are mostly restricted to ideal scenarios. This means that driving in challenging illumination conditions such as night, sunrise, and sunset remains an open problem. In these cases, standard cameras are being pushed to their limits in terms of low light and high dynamic range performance. To address these challenges, we propose, DSEC, a new dataset that contains such demanding illumination conditions and provides a rich set of sensory data. DSEC offers data from a wide-baseline stereo setup of two color frame cameras and two high-resolution monochrome event cameras. In addition, we collect lidar data and RTK GPS measurements, both hardware synchronized with all camera data. One of the distinctive features of this dataset is the inclusion of high-resolution event cameras. Event cameras have received increasing attention for their high temporal resolution and high dynamic range performance. However, due to their novelty, event camera datasets in driving scenarios are rare. This work presents the first high-resolution, large-scale stereo dataset with event cameras. The dataset contains 53 sequences collected by driving in a variety of illumination conditions and provides ground truth disparity for the development and evaluation of event-based stereo algorithms., Comment: IEEE Robotics and Automation Letters

Event cameras are novel vision sensors that report per-pixel brightness changes as a stream of asynchronous "events". They offer significant advantages compared to standard cameras due to their high temporal resolution, high dynamic range and lack of motion blur. However, events only measure the varying component of the visual signal, which limits their ability to encode scene context. By contrast, standard cameras measure absolute intensity frames, which capture a much richer representation of the scene. Both sensors are thus complementary. However, due to the asynchronous nature of events, combining them with synchronous images remains challenging, especially for learning-based methods. This is because traditional recurrent neural networks (RNNs) are not designed for asynchronous and irregular data from additional sensors. To address this challenge, we introduce Recurrent Asynchronous Multimodal (RAM) networks, which generalize traditional RNNs to handle asynchronous and irregular data from multiple sensors. Inspired by traditional RNNs, RAM networks maintain a hidden state that is updated asynchronously and can be queried at any time to generate a prediction. We apply this novel architecture to monocular depth estimation with events and frames where we show an improvement over state-of-the-art methods by up to 30% in terms of mean absolute depth error. To enable further research on multimodal learning with events, we release EventScape, a new dataset with events, intensity frames, semantic labels, and depth maps recorded in the CARLA simulator.

This MDPI open-access journal articleis an enhanced version of the paper published previously with CNSM 2021 of the same title. This paper provides an in-depth description of an access management system that utilizes Artificial Intelligence (AI), Blockchains (BC), and Internet-of-Things (IoT) in an integrated use case. This use case centers around access management without direct human intervention. Thus, the user must present their face to a camera to access a resource. The system takes an image of their face and checks whether this user has the right to access a given resource. For that decision to be taken, face detection and recognition are performed directly on the IoT device. A trained MTCNN (Multi-Task Cascaded Convolutional Network) with MobileNets (MN) was deployed to detect faces. Furthermore, the ESP32-based Face Recognition model is based on a Convolutional Neural Network (FRMN) was used. To establish transparency, AI decisions on access rights and images taken by the sensor are stored in the immutable, tamper-resistant storage implemented with the help of the HyperLedger Fabric (HLF). The system's performance was evaluated at an excellent level, where a 5.3 s end-to-end delay was reached. This value reads to be outstanding compared to well-established BC systems, such as Bitcoin (BTC) or Ethereum (ETH), having a block time configured at the level of 10 min and 10 s, respectively. Additionally, the size of data to be persisted inside a block does not lead to high costs, as for a BTC or ETH case, since HLF does offer "unlimited" storage capacity due to its private, permissioned characteristic. At the same time, data privacy is ensured since the private ledger within the HLF implementation will only allow for authorized access by definition. As this approach taken is considered to be a prototype, two current drawbacks will have to be taken care of in the next generation: All face descriptors of recognized faces are hard-encoded on Esp Eye devices. The HLF Transaction (TX) signing is not performed directly on Esp Eye devices since it is delegated to the IoT Gateway device supporting this process on behalf of the sensor. Apart from solving these aspects, future work will not only close the gaps as identified in the currently specified and prototyped system but will further develop the IoT-BC integration into operational and reusable software functionality: The face descriptor management will allow an IoT device to retrieve a remote directory of recognized faces. Porting the HLF Software Development Kit (SDK) to IoT devices will enable an HLF TX submission directly from IoT devices.

Reliable perception during fast motion maneuvers or in high dynamic range environments is crucial for robotic systems. Since event cameras are robust to these challenging conditions, they have great potential to increase the reliability of robot vision. However, event-based vision has been held back by the shortage of labeled datasets due to the novelty of event cameras. To overcome this drawback, we propose a task transfer method to train models directly with labeled images and unlabeled event data. Compared to previous approaches, (i) our method transfers from single images to events instead of high frame rate videos, and (ii) does not rely on paired sensor data. To achieve this, we leverage the generative event model to split event features into content and motion features. This split enables efficient matching between latent spaces for events and images, which is crucial for successful task transfer. Thus, our approach unlocks the vast amount of existing image datasets for the training of event-based neural networks. Our task transfer method consistently outperforms methods targeting Unsupervised Domain Adaptation for object detection by 0.26 mAP (increase by 93%) and classification by 2.7% accuracy.

Lower leg prostheses could improve the life quality of amputees by increasing comfort and reducing energy to locomote, but currently control methods are limited in modulating behaviors based upon the human's experience. This paper describes the first steps toward learning complex controllers for dynamical robotic assistive devices. We provide the first example of behavioral cloning to control a powered transfemoral prostheses using a Gated Recurrent Unit (GRU) based recurrent neural network (RNN) running on a custom hardware accelerator that exploits temporal sparsity. The RNN is trained on data collected from the original prosthesis controller. The RNN inference is realized by a novel EdgeDRNN accelerator in real-time. Experimental results show that the RNN can replace the nominal PD controller to realize end-to-end control of the AMPRO3 prosthetic leg walking on flat ground and unforeseen slopes with comparable tracking accuracy. EdgeDRNN computes the RNN about 240 times faster than real time, opening the possibility of running larger networks for more complex tasks in the future. Implementing an RNN on this real-time dynamical system with impacts sets the ground work to incorporate other learned elements of the human-prosthesis system into prosthesis control., Comment: Accepted at 2020 International Conference on Robotics and Automation (ICRA 2020)

In modern visual SLAM systems, it is a standard practice to retrieve potential candidate map points from overlapping keyframes for further feature matching or direct tracking. In this work, we argue that keyframes are not the optimal choice for this task, due to several inherent limitations, such as weak geometric reasoning and poor scalability. We propose a voxel-map representation to efficiently retrieve map points for visual SLAM. In particular, we organize the map points in a regular voxel grid. Visible points from a camera pose are queried by sampling the camera frustum in a raycasting manner, which can be done in constant time using an efficient voxel hashing method. Compared with keyframes, the retrieved points using our method are geometrically guaranteed to fall in the camera field-of-view, and occluded points can be identified and removed to a certain extend. This method also naturally scales up to large scenes and complicated multi-camera configurations. Experimental results show that our voxel map representation is as efficient as a keyframe map with 5 keyframes and provides significantly higher localization accuracy (average 46% improvement in RMSE) on the EuRoC dataset. The proposed voxel-map representation is a general approach to a fundamental functionality in visual SLAM and widely applicable.

Spiking Neural Networks (SNNs) are bio-inspired networks that process information conveyed as temporal spikes rather than numeric values. An example of a sensor providing such data is the event-camera. It only produces an event when a pixel reports a significant brightness change. Similarly, the spiking neuron of an SNN only produces a spike whenever a significant number of spikes occur within a short period of time. Due to their spike-based computational model, SNNs can process output from event-based, asynchronous sensors without any pre-processing at extremely lower power unlike standard artificial neural networks. This is possible due to specialized neuromorphic hardware that implements the highly-parallelizable concept of SNNs in silicon. Yet, SNNs have not enjoyed the same rise of popularity as artificial neural networks. This not only stems from the fact that their input format is rather unconventional but also due to the challenges in training spiking networks. Despite their temporal nature and recent algorithmic advances, they have been mostly evaluated on classification problems. We propose, for the first time, a temporal regression problem of numerical values given events from an event-camera. We specifically investigate the prediction of the 3- DOF angular velocity of a rotating event-camera with an SNN. The difficulty of this problem arises from the prediction of angular velocities continuously in time directly from irregular, asynchronous event-based input. Directly utilising the output of event-cameras without any pre-processing ensures that we inherit all the benefits that they provide over conventional cameras. That is high-temporal resolution, high-dynamic range and no motion blur. To assess the performance of SNNs on this task, we introduce a synthetic event-camera dataset generated from real-world panoramic images and show that we can successfully train an SNN to perform angular velocity regression.

Adding more cameras to SLAM systems improves robustness and accuracy but complicates the design of the visual front-end significantly. Thus, most systems in the literature are tailored for specific camera configurations. In this work, we aim at an adaptive SLAM system that works for arbitrary multi-camera setups. To this end, we revisit several common building blocks in visual SLAM. In particular, we propose an adaptive initialization scheme, a sensor-agnostic, information- theoretic keyframe selection algorithm, and a scalable voxel- based map. These techniques make little assumption about the actual camera setups and prefer theoretically grounded methods over heuristics. We adapt a state-of-the-art visual- inertial odometry with these modifications, and experimental results show that the modified pipeline can adapt to a wide range of camera setups (e.g., 2 to 6 cameras in one experiment) without the need of sensor-specific modifications or tuning.

In this paper, we propose to augment image-based place recognition with structural cues. Specifically, these structural cues are obtained using structure-from-motion, such that no additional sensors are needed for place recognition. This is achieved by augmenting the 2D convolutional neural network (CNN) typically used for image-based place recognition with a 3D CNN that takes as input a voxel grid derived from the structure-from-motion point cloud. We evaluate different methods for fusing the 2D and 3D features and obtain best performance with global average pooling and simple concatenation. On the Oxford RobotCar dataset, the resulting descriptor exhibits superior recognition performance compared to descriptors extracted from only one of the input modalities, including state-of-the-art image-based descriptors. Especially at low descriptor dimensionalities, we outperform state-of-the-art descriptors by up to 90%., Comment: 8 pages, published in RA-L & IROS 2020

Whether or not blockchain technology adds value to supply chains is a highly controversial topic. It has been lauded as the technology that brings transparency and efficiency to supply chains. However, others find this view overly optimistic. In this paper, we aim to add more insight into the issue by applying an action design research approach. We develop and evaluate two artifacts in collaboration with companies and use the results to conclude how blockchain technology can help overcome issues in the current supply chain information flow. It shows that blockchains can provide a shared trust base and a common standard that allows for increased information exchange between supply chain partners. Based on our findings, we propose a research agenda for studying blockchain technology with a stronger focus on governance mechanism and transparency issues.

Motivated by the astonishing capabilities of natural intelligent agents and inspired by theories from psychology, this paper explores the idea that perception gets coupled to 3D properties of the world via interaction with the environment. Existing works for depth estimation require either massive amounts of annotated training data or some form of hard-coded geometrical constraint. This paper explores a new approach to learning depth perception requiring neither of those. Specifically, we train a specialized global-local network architecture with what would be available to a robot interacting with the environment: from extremely sparse depth measurements down to even a single pixel per image. From a pair of consecutive images, our proposed network outputs a latent representation of the observer's motion between the images and a dense depth map. Experiments on several datasets show that, when ground truth is available even for just one of the image pixels, the proposed network can learn monocular dense depth estimation up to 22.5% more accurately than state-of-the-art approaches. We believe that this work, despite its scientific interest, lays the foundations to learn depth from extremely sparse supervision, which can be valuable to all robotic systems acting under severe bandwidth or sensing constraints., Comment: Accepted for Publication at the IEEE Robotics and Automation Letters (RA-L) 2020, and International Conference on Intelligent Robots and Systems (IROS) 2020

Civilian drones are soon expected to be used in a wide variety of tasks, such as aerial surveillance, delivery, or monitoring of existing architectures. Nevertheless, their deployment in urban environments has so far been limited. Indeed, in unstructured and highly dynamic scenarios, drones face numerous challenges to navigate autonomously in a feasible and safe way. In contrast to traditional “map-localize-plan” methods, this letter explores a data-driven approach to cope with the above challenges. To accomplish this, we propose DroNet: a convolutional neural network that can safely drive a drone through the streets of a city. Designed as a fast eight-layers residual network, DroNet produces two outputs for each single input image: A steering angle to keep the drone navigating while avoiding obstacles, and a collision probability to let the UAV recognize dangerous situations and promptly react to them. The challenge is however to collect enough data in an unstructured outdoor environment such as a city. Clearly, having an expert pilot providing training trajectories is not an option given the large amount of data required and, above all, the risk that it involves for other vehicles or pedestrians moving in the streets. Therefore, we propose to train a UAV from data collected by cars and bicycles, which, already integrated into the urban environment, would not endanger other vehicles and pedestrians. Although trained on city streets from the viewpoint of urban vehicles, the navigation policy learned by DroNet is highly generalizable. Indeed, it allows a UAV to successfully fly at relative high altitudes and even in indoor environments, such as parking lots and corridors. To share our findings with the robotics community, we publicly release all our datasets, code, and trained networks.

Transport mode detection is an emerging research area in different domains such as urban planning, context-aware mobile computing, and intelligent transportation systems. Current approaches are mostly data-driven, based on machine learning approaches. However, machine learning models require substantial training data and cannot explain the reasoning procedure. Data-driven approaches also fall short while interpreting trajectories where ground truth information is limited. Therefore, this paper develops a novel knowledge-based approach for interpreting smartphone global positioning system trajectories by detecting various transport modes used during travel. The proposed model is based on an expert system that can work without any training, based solely on expert knowledge. Core is a fuzzy multiple-input multiple-output expert system using kinematic and spatial information with a well explained fuzzy reasoning scheme through a fuzzy rule base. The model can provide alternate predictions with varied certainty factors. Different membership function combinations have been evaluated in terms of accuracy and ambiguity, and the result demonstrates that the model performs best using a Gaussian–Gaussian combination, comparable to the existing machine learning approaches.

When we search online for content, we are constantly exposed to rankings. For example, web search results are presented as a ranking, and online bookstores often show us lists of best-selling books. While popularity-based ranking algorithms (like Google's PageRank) have been extensively studied in previous works, we still lack a clear understanding of their potential systemic consequences. In this work, we fill this gap by introducing a new model of network growth that allows us to compare the properties of the networks generated under the influence of different ranking algorithms. We show that by correcting for the omnipresent age bias of popularity-based ranking algorithms, the resulting networks exhibit a significantly larger agreement between the nodes' inherent quality and their long-term popularity, and a less concentrated popularity distribution. To further promote popularity diversity, we introduce and validate a perturbation of the original rankings where a small number of randomly-selected nodes are promoted to the top of the ranking. Our findings move the first steps toward a model-based understanding of the long-term impact of popularity-based ranking algorithms, and could be used as an informative tool for the design of improved information filtering tools., Main text (pp. 1-27) and Supplementary Material (pp. 28-49)

Event cameras are novel bio-inspired vision sensors that output pixel-level intensity changes, called “events”, instead of traditional video images. These asynchronous sensors naturally respond to motion in the scene with very low latency (in the order of microseconds) and have a very high dynamic range. These features, along with a very low power consumption, make event cameras an ideal sensor for fast robot localization and wearable applications, such as AR/VR and gaming. Considering these applications, we present a method to track the 6-DOF pose of an event camera in a known environment, which we contemplate to be described by a photometric 3D map (i.e., intensity plus depth information) built via classic dense 3D reconstruction algorithms. Our approach uses the raw events, directly, without intermediate features, within a maximum-likelihood framework to estimate the camera motion that best explains the events via a generative model. We successfully evaluate the method using both simulated and real data, and show improved results over the state of the art. We release the datasets to the public to foster reproducibility and research in this topic.

In recent years, many approaches to visual-inertial odometry (VIO) have become available. However, they neither exploit the robot's dynamics and known actuation inputs, nor differentiate between the desired motion due to actuation and the unwanted perturbation due to external force. For many robotic applications, it is often essential to sense the external force acting on the system due to, for example, interactions, contacts, and disturbances. Adding a motion constraint to an estimator leads to discrepancy between the model-predicted motion and the actual motion. Our approach exploits this discrepancy and resolves it by simultaneously estimating the motion and external force. We propose a relative motion constraint combining the robot's dynamics and the external force in a preintegrated residual, resulting in a tightly coupled, sliding-window estimator exploiting all correlations among all variables. We implement our visual inertial model-based odometry system into a state-of-the-art VIO approach and evaluate it against the original pipeline without motion constraints both on simulated and real-world data. The results show that our approach increases the accuracy of the estimator up to 29% as compared to the original VIO, and it provides external force estimates at no extra computational cost. To the best of our knowledge, this is the first approach that exploits model dynamics by jointly estimating motion and external force. Our implementation will be made available open-source.

Children with hand motor impairment due to cerebral palsy, traumatic brain injury, or pediatric stroke are considerably affected in their independence, development, and quality of life. Treatment conventionally includes task-oriented training in occupational therapy. While dose and intensity of hand therapy can be promoted through technology, these approaches are mostly limited to large stationary robotic devices for non-task-oriented training, or passive wearable devices for children with mild impairments. Here we present PEXO, a fully wearable actuated pediatric hand exoskeleton to cover the special needs of children (6 to 12 years of age) with strong impairments in hand function. Through three degrees of freedom, PEXO provides assistance in various grasp types needed for the execution of functional tasks. It is lightweight, water proof, and inherently interacts safely with the user. It meets mechanical requirements such as force, fast closing movement, and battery lifetime derived from literature and discussions with clinicians. Appealing appearance, user-friendly design, and intuitive control with visual feedback of forearm muscle activity should keep the user motivated during training in the clinic or at home. A pilot test with a 6-years old child with stroke showed that PEXO can provide assistance in grasping various objects weighing up to 0.5 kg. These are promising first results on the way to make hand exoskeletons accessible for children with neuromotor disorders.

Wearable robots for the legs have been developed for gait rehabilitation training and as assistive devices. Most devices have been rigid exoskeletons designed to substitute the function of users who are completely paralyzed. While effective for this target group, exoskeletons limit their users' contributions to movements. Soft wearable robots have been suggested as an alternative that allows, and requires, active contributions from users with residual mobility. In this work, we first tested if the MyoSuit, a lightweight, lower-limb soft wearable robot, affected the walking kinematics of unimpaired users. Secondly, we evaluated the assistance delivered to a patient with a gait impairment. In our first study, 10 unimpaired participants walked on a treadmill at speeds between 0.5 and 1.3 m/s. We found that wearing the MyoSuit in its transparency mode did not affect the participants' walking kinematics (RMS difference of joint angles

Neuromorphic computing is a new paradigm for design of both the computing hardware and algorithms inspired by biological neural networks. The event-based nature and the inherent parallelism make neuromorphic computing a promising paradigm for building efficient neural network based architectures for control of fast and agile robots. In this paper, we present a spiking neural network architecture that uses sensory feedback to control rotational velocity of a robotic vehicle. When the velocity reaches the target value, the mapping from the target velocity of the vehicle to the correct motor command, both represented in the spiking neural network on the neuromorphic device, is autonomously stored on the device using on-chip plastic synaptic weights. We validate the controller using a wheel motor of a miniature mobile vehicle and inertia measurement unit as the sensory feedback and demonstrate online learning of a simple 'inverse model' in a two-layer spiking neural network on the neuromorphic chip. The prototype neuromorphic device that features 256 spiking neurons allows us to realise a simple proof of concept architecture for the purely neuromorphic motor control and learning. The architecture can be easily scaled-up if a larger neuromorphic device is available., Comment: 6+1 pages, 4 figures, will appear in one of the Robotics conferences

Dynamically changing environments, unreliable state estimation, and operation under severe resource constraints are fundamental challenges that limit the deployment of small autonomous drones. We address these challenges in the context of autonomous, vision-based drone racing in dynamic environments. A racing drone must traverse a track with possibly moving gates at high speed. We enable this functionality by combining the performance of a state-of-the-art planning and control system with the perceptual awareness of a convolutional neural network (CNN). The resulting modular system is both platform- and domain-independent: it is trained in simulation and deployed on a physical quadrotor without any fine-tuning. The abundance of simulated data, generated via domain randomization, makes our system robust to changes of illumination and gate appearance. To the best of our knowledge, our approach is the first to demonstrate zero-shot sim-to-real transfer on the task of agile drone flight. We extensively test the precision and robustness of our system, both in simulation and on a physical platform, and show significant improvements over the state of the art., Accepted as a Regular Paper to the IEEE Transactions on Robotics Journal. arXiv admin note: substantial text overlap with arXiv:1806.08548

Despite impressive results in visual-inertial state estimation in recent years, high speed trajectories with six degree of freedom motion remain challenging for existing estimation algorithms. Aggressive trajectories feature large accelerations and rapid rotational motions, and when they pass close to objects in the environment, this induces large apparent motions in the vision sensors, all of which increase the difficulty in estimation. Existing benchmark datasets do not address these types of trajectories, instead focusing on slow speed or constrained trajectories, targeting other tasks such as inspection or driving. We introduce the UZH-FPV Drone Racing dataset, consisting of over 27 sequences, with more than 10 km of flight distance, captured on a first-person-view (FPV) racing quadrotor flown by an expert pilot. The dataset features camera images, inertial measurements, event-camera data, and precise ground truth poses. These sequences are faster and more challenging, in terms of apparent scene motion, than any existing dataset. Our goal is to enable advancement of the state of the art in aggressive motion estimation by providing a dataset that is beyond the capabilities of existing state estimation algorithms.

For mobile robots to localize robustly, actively considering the perception requirement at the planning stage is essential. In this paper, we propose a novel representation for active visual localization. By formulating the Fisher information and sensor visibility carefully, we are able to summarize the localization information into a discrete grid, namely the Fisher information field. The information for arbitrary poses can then be computed from the field in constant time, without the need of costly iterating all the 3D landmarks. Experimental results on simulated and real-world data show the great potential of our method in efficient active localization and perception-aware planning. To benefit related research, we release our implementation of the information field to the public.

Event cameras are bio-inspired vision sensors that output pixel-level brightness changes instead of standard intensity frames. They offer significant advantages over standard cameras, namely a very high dynamic range, no motion blur, and a latency in the order of microseconds. However, due to the fundamentally different structure of the sensor's output, new algorithms that exploit the high temporal resolution and the asynchronous nature of the sensor are required. Recent work has shown that a continuous-time representation of the event camera pose can deal with the high temporal resolution and asynchronous nature of this sensor in a principled way. In this paper, we leverage such a continuous-time representation to perform visual-inertial odometry with an event camera. This representation allows direct integration of the asynchronous events with micro-second accuracy and the inertial measurements at high frequency. The event camera trajectory is approximated by a smooth curve in the space of rigid-body motions using cubic splines. This formulation significantly reduces the number of variables in trajectory estimation problems. We evaluate our method on real data from several scenes and compare the results against ground truth from a motion-capture system. We show that our method provides improved accuracy over the result of a state-of-the-art visual odometry method for event cameras. We also show that both the map orientation and scale can be recovered accurately by fusing events and inertial data. To the best of our knowledge, this is the first work on visual-inertial fusion with event cameras using a continuous-time framework., 15 pages, 12 figures, 6 tables, IEEE Transactions on Robotics, 2018

IEEE Robotics and Automation Letters, 4 (2), ISSN:2377-3766

The recent advances in state estimation, perception, and navigation algorithms have significantly contributed to the ubiquitous use of quadrotors for inspection, mapping, and aerial imaging. To further increase the versatility of quadrotors, recent works investigated the use of an adaptive morphology, which consists of modifying the shape of the vehicle during flight to suit a specific task or environment. However, these works either increase the complexity of the platform or decrease its controllability. In this letter, we propose a novel, simpler, yet effective morphing design for quadrotors consisting of a frame with four independently rotating arms that fold around the main frame. To guarantee stable flight at all times, we exploit an optimal control strategy that adapts on the fly to the drone morphology. We demonstrate the versatility of the proposed adaptive morphology in different tasks, such as negotiation of narrow gaps, close inspection of vertical surfaces, and object grasping and transportation. The experiments are performed on an actual, fully autonomous quadrotor relying solely on onboard visual-inertial sensors and compute. No external motion tracking systems and computers are used. This is the first work showing stable flight without requiring any symmetry of the morphology.

When investigating the spreading of a piece of information or the diffusion of an innovation, we often lack information on the underlying propagation network. Reconstructing the hidden propagation paths based on the observed diffusion process is a challenging problem which has recently attracted attention from diverse research fields. To address this reconstruction problem, based on static similarity metrics commonly used in the link prediction literature, we introduce new node-node temporal similarity metrics. The new metrics take as input the time-series of multiple independent spreading processes, based on the hypothesis that two nodes are more likely to be connected if they were often infected at similar points in time. This hypothesis is implemented by introducing a time-lag function which penalizes distant infection times. We find that the choice of this time-lag strongly affects the metrics' reconstruction accuracy, depending on the network's clustering coefficient and we provide an extensive comparative analysis of static and temporal similarity metrics for network reconstruction. Our findings shed new light on the notion of similarity between pairs of nodes in complex networks.