Search

The cockpit is evolving from passive, reactive interaction toward proactive, cognitive interaction, making precise predictions of driver intent a key factor in enhancing proactive interaction experiences. This paper introduces Cockpit-Llama, a novel language model specifically designed for predicting driver behavior intent. Cockpit-Llama predicts driver intent based on the relationship between current driver actions, historical interactions, and the states of the driver and cockpit environment, thereby supporting further proactive interaction decisions. To improve the accuracy and rationality of Cockpit-Llama's predictions, we construct a new multi-attribute cockpit dataset that includes extensive historical interactions and multi-attribute states, such as driver emotional states, driving activity scenarios, vehicle motion states, body states and external environment, to support the fine-tuning of Cockpit-Llama. During fine-tuning, we adopt the Low-Rank Adaptation (LoRA) method to efficiently optimize the parameters of the Llama3-8b-Instruct model, significantly reducing training costs. Extensive experiments on the multi-attribute cockpit dataset demonstrate that Cockpit-Llama's prediction performance surpasses other advanced methods, achieving BLEU-4, ROUGE-1, ROUGE-2, and ROUGE-L scores of 71.32, 80.01, 76.89, and 81.42, respectively, with relative improvements of 92.34%, 183.61%, 95.54%, and 201.27% compared to ChatGPT-4. This significantly enhances the reasoning and interpretative capabilities of intelligent cockpits. [ABSTRACT FROM AUTHOR]

The paradigm of Industry 5.0 pushes the transition from the traditional to a novel, smart, digital, and connected industry, where well-being is key to enhance productivity, optimize man–machine interaction and guarantee workers' safety. This work aims to conduct a systematic review of current methodologies for monitoring and analyzing physical and cognitive ergonomics. Three research questions are addressed: (1) which technologies are used to assess the physical and cognitive well-being of workers in the workplace, (2) how the acquired data are processed, and (3) what purpose this well-being is evaluated for. This way, individual factors within the holistic assessment of worker well-being are highlighted, and information is provided synthetically. The analysis was conducted following the PRISMA 2020 statement guidelines. From the sixty-five articles collected, the most adopted (1) technological solutions, (2) parameters, and (3) data analysis and processing were identified. Wearable inertial measurement units and RGB-D cameras are the most prevalent devices used for physical monitoring; in the cognitive ergonomics, and cardiac activity is the most adopted physiological parameter. Furthermore, insights on practical issues and future developments are provided. Future research should focus on developing multi-modal systems that combine these aspects with particular emphasis on their practical application in real industrial settings. [ABSTRACT FROM AUTHOR]

Accurately capturing human movements is a crucial element of health status monitoring and a necessary precondition for realizing future virtual reality/augmented reality applications. Flexible motion sensors with exceptional sensitivity are capable of detecting physical activities by converting them into resistance fluctuations. Silver nanowires (AgNWs) have become a preferred choice for the development of various types of sensors due to their outstanding electrical conductivity, transparency, and flexibility within polymer composites. Herein, we present the design and fabrication of a flexible strain sensor based on silver nanowires. Suitable substrate materials were selected, and the sensor's sensitivity and fatigue properties were characterized and tested, with the sensor maintaining reliability after 5000 deformation cycles. Different sensors were prepared by controlling the concentration of silver nanowires to achieve the collection of motion signals from various parts of the human body. Additionally, we explored potential applications of these sensors in fields such as health monitoring and virtual reality. In summary, this work integrated the acquisition of different human motion signals, demonstrating great potential for future multifunctional wearable electronic devices. [ABSTRACT FROM AUTHOR]

The emergence of the Metaverse is raising important questions in the field of human–machine interaction that must be addressed for a successful implementation of the new paradigm. Therefore, the exploration and integration of both technology and human interaction within this new framework are needed. This paper describes an innovative and technically viable proposal for virtual shopping in the fashion field. Virtual hands directly scanned from the real world have been integrated, after a retopology process, in a virtual environment created for the Metaverse, and have been integrated with digital nails. Human interaction with the Metaverse has been carried out through the acquisition of the real posture of the user's hands using an infrared-based sensor and mapping it in its virtualized version, achieving natural identification. The technique has been successfully tested in an immersive shopping experience with the Meta Quest 2 headset as a pilot project, where a transactions mechanism based on the blockchain technology (non-fungible tokens, NFTs) has allowed for the development of a feasible solution for massive audiences. The consumers' reactions were extremely positive, with a total of 250 in-person participants and 120 remote accesses to the Metaverse. Very interesting technical guidelines are raised in this project, the resolution of which may be useful for future implementations. [ABSTRACT FROM AUTHOR]

This article presents the Network Empower and Prototyping Platform (NEP+), a flexible framework purposefully crafted to simplify the process of interactive application development, catering to both technical and non-technical users. The name "NEP+" encapsulates the platform's dual mission: to empower the network-related capabilities of ZeroMQ and to provide software tools and interfaces for prototyping and integration. NEP+ accomplishes this through a comprehensive quality model and an integrated software ecosystem encompassing middleware, user-friendly graphical interfaces, a command-line tool, and an accessible end-user programming interface. This article primarily focuses on presenting the proposed quality model and software architecture, illustrating how they can empower developers to craft cross-platform, accessible, and user-friendly interfaces for various applications, with a particular emphasis on robotics and the Internet of Things (IoT). Additionally, we provide practical insights into the applicability of NEP+ by briefly presenting real-world user cases where human-centered projects have successfully utilized NEP+ to develop robotics systems. To further emphasize the suitability of NEP+ tools and interfaces for developer use, we conduct a pilot study that delves into usability and workload assessment. The outcomes of this study highlight the user-friendly features of NEP+ tools, along with their ease of adoption and cross-platform capabilities. The novelty of NEP+ fundamentally lies in its holistic approach, acting as a bridge across diverse user groups, fostering inclusivity, and promoting collaboration. [ABSTRACT FROM AUTHOR]

Humans' performance varies due to the mental resources that are available to successfully pursue a task. To monitor users' current cognitive resources in naturalistic scenarios, it is essential to not only measure demands induced by the task itself but also consider situational and environmental influences. We conducted a multimodal study with 18 participants (nine female, M = 25.9 with SD = 3.8 years). In this study, we recorded respiratory, ocular, cardiac, and brain activity using functional near-infrared spectroscopy (fNIRS) while participants performed an adapted version of the warship commander task with concurrent emotional speech distraction. We tested the feasibility of decoding the experienced mental effort with a multimodal machine learning architecture. The architecture comprised feature engineering, model optimisation, and model selection to combine multimodal measurements in a cross-subject classification. Our approach reduces possible overfitting and reliably distinguishes two different levels of mental effort. These findings contribute to the prediction of different states of mental effort and pave the way toward generalised state monitoring across individuals in realistic applications. [ABSTRACT FROM AUTHOR]

Sensing the interaction between the pilot and the control inceptors can provide important information about the pilot's activity during flight, potentially enabling the objective measurement of the pilot workload, the application of preventive actions against loss of situational awareness, and the identification of the insurgence of adverse couplings with the vehicle dynamics. This work presents an innovative pressure-sensing device developed to be seamlessly integrated into the grips of conventional aircraft control inceptors. The sensor, based on frustrated total internal reflection of light, is composed of low-cost elements and can be easily manufactured to be applicable to different hand pressure ranges. The characteristics of the sensor are first demonstrated in laboratory calibration tests. Subsequently, applications in flight simulator testing are presented, focusing on the objective representation of the pilot's instantaneous workload. [ABSTRACT FROM AUTHOR]

In the billions of faces that are shaped by thousands of different cultures and ethnicities, one thing remains universal: the way emotions are expressed. To take the next step in human–machine interactions, a machine (e.g., a humanoid robot) must be able to clarify facial emotions. Allowing systems to recognize micro-expressions affords the machine a deeper dive into a person's true feelings, which will take human emotion into account while making optimal decisions. For instance, these machines will be able to detect dangerous situations, alert caregivers to challenges, and provide appropriate responses. Micro-expressions are involuntary and transient facial expressions capable of revealing genuine emotions. We propose a new hybrid neural network (NN) model capable of micro-expression recognition in real-time applications. Several NN models are first compared in this study. Then, a hybrid NN model is created by combining a convolutional neural network (CNN), a recurrent neural network (RNN, e.g., long short-term memory (LSTM)), and a vision transformer. The CNN can extract spatial features (within a neighborhood of an image), whereas the LSTM can summarize temporal features. In addition, a transformer with an attention mechanism can capture sparse spatial relations residing in an image or between frames in a video clip. The inputs of the model are short facial videos, while the outputs are the micro-expressions recognized from the videos. The NN models are trained and tested with publicly available facial micro-expression datasets to recognize different micro-expressions (e.g., happiness, fear, anger, surprise, disgust, sadness). Score fusion and improvement metrics are also presented in our experiments. The results of our proposed models are compared with that of literature-reported methods tested on the same datasets. The proposed hybrid model performs the best, where score fusion can dramatically increase recognition performance. [ABSTRACT FROM AUTHOR]

This article is devoted to the study of the correlation between the emotional state of a person and the posture of his or her body in the sitting position. In order to carry out the study, we developed the first version of the hardware-software system based on a posturometric armchair, allowing the characteristics of the posture of a sitting person to be evaluated using strain gauges. Using this system, we revealed the correlation between sensor readings and human emotional states. We showed that certain readings of a sensor group are formed for a certain emotional state of a person. We also found that the groups of triggered sensors, their composition, their number, and their location are related to the states of a particular person, which led to the need to build personalized digital pose models for each person. The intellectual component of our hardware–software complex is based on the concept of co-evolutionary hybrid intelligence. The system can be used during medical diagnostic procedures and rehabilitation processes, as well as in controlling people whose professional activity is connected with increased psycho-emotional load and can cause cognitive disorders, fatigue, and professional burnout and can lead to the development of diseases. [ABSTRACT FROM AUTHOR]

Multimodal user interfaces promise natural and intuitive human–machine interactions. However, is the extra effort for the development of a complex multisensor system justified, or can users also be satisfied with only one input modality? This study investigates interactions in an industrial weld inspection workstation. Three unimodal interfaces, including spatial interaction with buttons augmented on a workpiece or a worktable, and speech commands, were tested individually and in a multimodal combination. Within the unimodal conditions, users preferred the augmented worktable, but overall, the interindividual usage of all input technologies in the multimodal condition was ranked best. Our findings indicate that the implementation and the use of multiple input modalities is valuable and that it is difficult to predict the usability of individual input modalities for complex systems. [ABSTRACT FROM AUTHOR]

The article presents a novel idea of Interaction Quality Sensor (IQS), introduced in the complete solution of Hybrid INTelligence (HINT) architecture for intelligent control systems. The proposed system is designed to use and prioritize multiple information channels (speech, images, videos) in order to optimize the information flow efficiency of interaction in HMI systems. The proposed architecture is implemented and validated in a real-world application of training unskilled workers—new employees (with lower competencies and/or a language barrier). With the help of the HINT system, the man–machine communication information channels are deliberately chosen based on IQS readouts to enable an untrained, inexperienced, foreign employee candidate to become a good worker, while not requiring the presence of either an interpreter or an expert during training. The proposed implementation is in line with the labor market trend, which displays significant fluctuations. The HINT system is designed to activate human resources and support organizations/enterprises in the effective assimilation of employees to the tasks performed on the production assembly line. The market need of solving this noticeable problem was caused by a large migration of employees within (and between) enterprises. The research results presented in the work show significant benefits of the methods used, while supporting multilingualism and optimizing the preselection of information channels. [ABSTRACT FROM AUTHOR]

Adaptive human–computer systems require the recognition of human behavior states to provide real-time feedback to scaffold skill learning. These systems are being researched extensively for intervention and training in individuals with autism spectrum disorder (ASD). Autistic individuals are prone to social communication and behavioral differences that contribute to their high rate of unemployment. Teamwork training, which is beneficial for all people, can be a pivotal step in securing employment for these individuals. To broaden the reach of the training, virtual reality is a good option. However, adaptive virtual reality systems require real-time detection of behavior. Manual labeling of data is time-consuming and resource-intensive, making automated data annotation essential. In this paper, we propose a semi-supervised machine learning method to supplement manual data labeling of multimodal data in a collaborative virtual environment (CVE) used to train teamwork skills. With as little as 2.5% of the data manually labeled, the proposed semi-supervised learning model predicted labels for the remaining unlabeled data with an average accuracy of 81.3%, validating the use of semi-supervised learning to predict human behavior. [ABSTRACT FROM AUTHOR]

Designing human–machine interactive systems requires cooperation between different disciplines is required. In this work, we present a Dialogue Manager and a Language Generator that are the core modules of a Voice-based Spoken Dialogue System (SDS) capable of carrying out challenging, long and complex coaching conversations. We also develop an efficient integration procedure of the whole system that will act as an intelligent and robust Virtual Coach. The coaching task significantly differs from the classical applications of SDSs, resulting in a much higher degree of complexity and difficulty. The Virtual Coach has been successfully tested and validated in a user study with independent elderly, in three different countries with three different languages and cultures: Spain, France and Norway. [ABSTRACT FROM AUTHOR]

The COVID-19 pandemic created the need for telerehabilitation development, while Industry 4.0 brought the key technology. As motor therapy often requires the physical support of a patient's motion, combining robot-aided workouts with remote control is a promising solution. This may be realised with the use of the device's digital twin, so as to give it an immersive operation. This paper presents an extensive overview of this technology's applications within the fields of industry and health. It is followed by the in-depth analysis of needs in rehabilitation based on questionnaire research and bibliography review. As a result of these sections, the original concept of controlling a rehabilitation exoskeleton via its digital twin in the virtual reality is presented. The idea is assessed in terms of benefits and significant challenges regarding its application in real life. The presented aspects prove that it may be potentially used for manual remote kinesiotherapy, combined with the safety systems predicting potentially harmful situations. The concept is universally applicable to rehabilitation robots. [ABSTRACT FROM AUTHOR]

Connected and automated vehicles have the potential to deliver significant environmental, safety, economic and social benefits. The key advancement for automated vehicles with higher levels of automation (SAE Level 4 and over) is fail-operational. One possible solution for the failsafe mode of automated vehicles is a 5G-enabled teleoperation system controlled by remote drivers. However, knowledge is missing regarding understanding of the human–machine interaction in teleoperation from the perspective of remote drivers. To address this research gap, this study qualitatively investigated the acceptance, attitudes, needs and requirements of remote drivers when teleoperating a 5G-enabled Level 4 automated vehicle (5G L4 AV) in the real world. The results showed that remote drivers are positive towards the 5G L4 AV. They would like to constantly monitor the driving when they are not controlling the vehicle remotely. Improving their field of vision for driving and enhancing the perception of physical motion feedback are the two key supports required by remote drivers in 5G L4 AVs. The knowledge gained in this study provides new insights into facilitating the design and development of safe, effective and user-friendly teleoperation systems in vehicle automation. [ABSTRACT FROM AUTHOR]

The article deals with the problems of improving modern human-machine interaction systems. Such systems are called biocybernetic systems. It is shown that a significant increase in their efficiency can be achieved by stabilising their work according to the automation control theory. An analysis of the structural schemes of the systems showed that one of the most significantly influencing factors in these systems is a poor "digitization" of the human condition. "Digitization" here is the identification of a person as a participant in the interaction with a cybernetic or cyber-physical system. The main problem of a biocybernetic system construction is the non-stationarity of such human characteristics as time of the reaction to external disturbances, physical or nervous fatigue, the ability to perform the required amount of work, etc. At the same time, as a rule, there is no objective assessment of this non-stationarity. Under these conditions, ensuring the controllability and efficiency of biocybernetic systems is a very difficult task. It is proposed to solve this problem with the help of electrocardiogram signals: the most accessible and accurate information about a human's current state. Herein, several examples of such solutions and the results of theoretical studies and experiments are discussed. [ABSTRACT FROM AUTHOR]

At airports, security officers (screeners) inspect X-ray images of passenger baggage in order to prevent threat items (bombs, guns, knives, etc.) from being brought onto an aircraft. Because threat items rarely occur, many airports use a threat-image-projection (TIP) system, which projects pre-recorded X-ray images of threat items onto some of the X-ray baggage images in order to improve the threat detection of screeners. TIP is regulatorily mandated in many countries and is also used to identify officers with insufficient threat-detection performance. However, TIP images sometimes look unrealistic because of artifacts and unrealistic scenarios, which could reduce the efficacy of TIP. Screeners rated a representative sample of TIP images regarding artifacts identified in a pre-study. We also evaluated whether specific image characteristics affect the occurrence rate of artifacts. 24% of the TIP images were rated to display artifacts and 26% to depict unrealistic scenarios, with 34% showing at least one of the two. With two-thirds of the TIP images having been perceived as realistic, we argue that TIP still serves its purpose, but artifacts and unrealistic scenarios should be reduced. Recommendations on how to improve the efficacy of TIP by considering image characteristics are provided. [ABSTRACT FROM AUTHOR]

This paper is devoted to a new method of using Microsoft (MS) Kinect sensors for non-contact monitoring of breathing and heart rate estimation to detect possible medical and neurological disorders. Video sequences of facial features and thorax movements are recorded by MS Kinect image, depth and infrared sensors to enable their time analysis in selected regions of interest. The proposed methodology includes the use of computational methods and functional transforms for data selection, as well as their denoising, spectral analysis and visualization, in order to determine specific biomedical features. The results that were obtained verify the correspondence between the evaluation of the breathing frequency that was obtained from the image and infrared data of the mouth area and from the thorax movement that was recorded by the depth sensor. Spectral analysis of the time evolution of the mouth area video frames was also used for heart rate estimation. Results estimated from the image and infrared data of the mouth area were compared with those obtained by contact measurements by Garmin sensors (www.garmin.com). The study proves that simple image and depth sensors can be used to efficiently record biomedical multidimensional data with sufficient accuracy to detect selected biomedical features using specific methods of computational intelligence. The achieved accuracy for non-contact detection of breathing rate was 0.26% and the accuracy of heart rate estimation was 1.47% for the infrared sensor. The following results show how video frames with depth data can be used to differentiate different kinds of breathing. The proposed method enables us to obtain and analyse data for diagnostic purposes in the home environment or during physical activities, enabling efficient human–machine interaction. [ABSTRACT FROM AUTHOR]

Current "Internet of Things"concepts point to a future where connected objects gather meaningful information about their environment and share it with other objects and people. In particular, objects embedding Human Machine Interaction (HMI), such as mobile devices and, increasingly, connected vehicles, home appliances, urban interactive infrastructures, etc., may not only be conceived as sources of sensor information, but, through interaction with their users, they can also produce highly valuable context-aware human-generated observations. We believe that the great promise offered by combining and sharing all of the different sources of information available can be realized through the integration of HMI and Semantic Sensor Web technologies. This paper presents a technological framework that harmonizes two of the most influential HMI and Sensor Web initiatives: the W3C's Multimodal Architecture and Interfaces (MMI) and the Open Geospatial Consortium (OGC) Sensor Web Enablement (SWE) with its semantic extension, respectively. Although the proposed framework is general enough to be applied in a variety of connected objects integrating HMI, a particular development is presented for a connected car scenario where drivers' observations about the traffic or their environment are shared across the Semantic Sensor Web. For implementation and evaluation purposes an on-board OSGi (Open Services Gateway Initiative) architecture was built, integrating several available HMI, Sensor Web and Semantic Web technologies. A technical performance test and a conceptual validation of the scenario with potential users are reported, with results suggesting the approach is sound. [ABSTRACT FROM AUTHOR]

This paper reports the design of a tactile sensor patch to cover large areas of robots and machines that interact with human beings. Many devices have been proposed to meet such a demand. These realizations are mostly custom-built or developed in the lab. The sensor of this paper is implemented with commercial force sensors. This has the benefit of a more foreseeable response of the sensor if its behavior is understood as the aggregation of readings from all the individual force sensors in the array. A few reported large area tactile sensors are also based on commercial sensors. However, the one in this paper is the first of this kind based on the use of polymeric commercial force sensing resistors (FSR) as unit elements of the array or tactels, which results in a robust sensor. The paper discusses design issues related to some necessary modifications of the force sensor, its assembly in an array, and the signal conditioning. The patch has 16 × 9 force sensors mounted on a flexible printed circuit board with a spatial resolution of 18.5 mm. The force range of a tactel is 6 N and its sensitivity is 0.6 V/N. The array is read at a rate of 78 frames per second. Finally, two simple application examples are also carried out with the sensor mounted on the forearm of a rescue robot that communicates with the sensor through a CAN bus. [ABSTRACT FROM AUTHOR]

Sensors are becoming ubiquitous in all areas of science, technology, and society. In this Special Issue on “Sensors for Entertainment”, developments in progress and the current state of application scenarios for sensors in the field of entertainment is explored. [ABSTRACT FROM AUTHOR]

(1) Background: Using autonomous social robots in selected areas of care for community-dwelling older adults is one of the promising approaches to address the problem of the widening care gap. We posed the question of whether a possibility to interact with the technology to be used had an impact on the scores given by the respondents in various domains of needs and requirements for social robots to be deployed in care for older individuals. (2) Methods: During the study, the opinions of older people (65+; n = 113; with no severe cognitive impairment) living in six social care institutions about a robot in care for older people were collected twice using the Users' Needs, Requirements and Abilities Questionnaire (UNRAQ): after seeing a photo of the robot only and after a 90–150 min interaction with the TIAGo robot. (3) Results: Mean total scores for both assistive and social functions were higher after the interaction (p < 0.05). A positive correlation was found between opinion changes in social and assistive functions (r = 0.4842; p = 0.0000). (4) Conclusions: Preimplementation studies and assessments should include the possibility to interact with the robot to provide its future users with a clear idea of the technology and facilitate necessary customisations of the machine. [ABSTRACT FROM AUTHOR]

In modern times, the collaboration between humans and machines increasingly rises, combining their respective benefits. The direct physical support causes interaction forces in human–machine interfaces, whereas their form determines both the effectiveness and comfort of the collaboration. However, their correct detection requires various sensor characteristics and remains challenging. Thus, this paper presents a developed low-cost sensor pad working with a silicone capsule and a piezoresistive pressure sensor. Its measurement accuracy is validated in both an isolated testing environment and a laboratory study with four test subjects (gender-balanced), and an application integrated in interfaces of an active upper-body exoskeleton. In the material-testing machine, it becomes apparent that the sensor pad generally features the capability of reliably determining normal forces on its surface until a certain threshold. This is also proven in the real application, where the measurement data of three sensor pads spatially embedded in the exoskeletal interface are compared to the data of an installed multi-axis load cell and a high-resolution flexible pressure map. Here, the consideration of three sensor pads potentially enables detection of exoskeletal support on the upper arm as well as "poor" fit conditions such as uneven pressure distributions that recommend immediate system adjustments for ergonomic improvements. [ABSTRACT FROM AUTHOR]

Changeover times are an important element when evaluating the Overall Equipment Effectiveness (OEE) of a production machine. The article presents a machine learning (ML) approach that is based on an external sensor setup to automatically detect changeovers in a shopfloor environment. The door statuses, coolant flow, power consumption, and operator indoor GPS data of a milling machine were used in the ML approach. As ML methods, Decision Trees, Support Vector Machines, (Balanced) Random Forest algorithms, and Neural Networks were chosen, and their performance was compared. The best results were achieved with the Random Forest ML model (97% F1 score, 99.72% AUC score). It was also carried out that model performance is optimal when only a binary classification of a changeover phase and a production phase is considered and less subphases of the changeover process are applied. [ABSTRACT FROM AUTHOR]

This work describes an assistive technology development for cognitive support and training to be used by children of special education schools in Spain. Design is based on and guided by cognitive support findings coming from a long-term collaboration of a team of engineers (University of Zaragoza) and special education teachers (Alborada Special Education School, Zaragoza). The description starts by providing a structure of such findings in five cognitive-social areas (interface usability, virtual representations understanding, time orientation, self-awareness, and social interaction). Design requirements are extracted by applying those findings to four support services (home control, time orientation, behavior contention, and context anticipation). Technological system description follows, together with the degree of implementation and testing for each service. A major result is the benefit of using a services interface with the same structure and appearance as the alternative and augmentative communication system that children are already acquainted with. Based on regulatory conditions, the needed flexibility, and reduced available budgets, this support platform is built on mainstream technology and low-cost single-board computers with standard databases and free software packages. Results show evidence of benefit in children's cognitive and social performance, in addition to offering a technological tool for deeper educational research. [ABSTRACT FROM AUTHOR]

With the further development of highly automated vehicles, drivers will engage in non-related tasks while being driven. Still, drivers have to take over control when requested by the car. Here, the question arises, how potentially distracted drivers get back into the control-loop quickly and safely when the car requests a takeover. To investigate effective human–machine interactions, a mobile, versatile, and cost-efficient setup is needed. Here, we describe a virtual reality toolkit for the Unity 3D game engine containing all the necessary code and assets to enable fast adaptations to various human–machine interaction experiments, including closely monitoring the subject. The presented project contains all the needed functionalities for realistic traffic behavior, cars, pedestrians, and a large, open-source, scriptable, and modular VR environment. It covers roughly 25 km2, a package of 125 animated pedestrians, and numerous vehicles, including motorbikes, trucks, and cars. It also contains all the needed nature assets to make it both highly dynamic and realistic. The presented repository contains a C++ library made for LoopAR that enables force feedback for gaming steering wheels as a fully supported component. It also includes all necessary scripts for eye-tracking in the used devices. All the main functions are integrated into the graphical user interface of the Unity® editor or are available as prefab variants to ease the use of the embedded functionalities. This project's primary purpose is to serve as an open-access, cost-efficient toolkit that enables interested researchers to conduct realistic virtual reality research studies without costly and immobile simulators. To ensure the accessibility and usability of the mentioned toolkit, we performed a user experience report, also included in this paper. [ABSTRACT FROM AUTHOR]

With the help of wearable robotics, the lower limb exoskeleton becomes a promising solution for spinal cord injury (SCI) patients to recover lower body locomotion ability. However, fewer exoskeleton gait planning methods can meet the needs of patient in real time, e.g., stride length or step width, etc., which may lead to human-machine incoordination, limit comfort, and increase the risk of falling. This work presents a human-exoskeleton-crutch system with the center of pressure (CoP)-based gait planning method to enable the balance control during the exoskeleton-assisted walking with crutches. The CoP generated by crutches and human-machine feet makes it possible to obtain the overall stability conditions of the system in the process of exoskeleton-assisted quasi-static walking, and therefore, to determine the next stride length and ensure the balance of the next step. Thus, the exoskeleton gait is planned with the guidance of stride length. It is worth emphasizing that the nominal reference gait is adopted as a reference to ensure that the trajectory of the swing ankle mimics the reference one well. This gait planning method enables the patient to adaptively interact with the exoskeleton gait. The online gait planning walking tests with five healthy volunteers proved the method's feasibility. Experimental results indicate that the algorithm can deal with the sensed signals and plan the landing point of the swing leg to ensure balanced and smooth walking. The results suggest that the method is an effective means to improve human–machine interaction. Additionally, it is meaningful for the further training of independent walking stability control in exoskeletons for SCI patients with less assistance of crutches. [ABSTRACT FROM AUTHOR]

This paper introduces a prototype of ClothFace technology, a battery-free textile-based handwriting recognition platform that includes an e-textile antenna and a 10 × 10 array of radio frequency identification (RFID) integrated circuits (ICs), each with a unique ID. Touching the textile platform surface creates an electrical connection from specific ICs to the antenna, which enables the connected ICs to be read with an external UHF (ultra-haigh frequency) RFID reader. In this paper, the platform is demonstrated to recognize handwritten numbers 0–9. The raw data collected by the platform are a sequence of IDs from the touched ICs. The system converts the data into bitmaps and their details are increased by interpolating between neighboring samples using the sequential information of IDs. These images of digits written on the platform can be classified, with enough accuracy for practical use, by deep learning. The recognition system was trained and tested with samples from six volunteers using the platform. The real-time number recognition ability of the ClothFace technology is demonstrated to work successfully with a very low error rate. The overall recognition accuracy of the platform is 94.6% and the accuracy for each digit is between 91.1% and 98.3%. As the solution is fully passive and gets all the needed energy from the external RFID reader, it enables a maintenance-free and cost-effective user interface that can be integrated into clothing and into textiles around us. [ABSTRACT FROM AUTHOR]

This paper is devoted to proving two goals, to show that various depth sensors can be used to record breathing rate with the same accuracy as contact sensors used in polysomnography (PSG), in addition to proving that breathing signals from depth sensors have the same sensitivity to breathing changes as in PSG records. The breathing signal from depth sensors can be used for classification of sleep apnea events with the same success rate as with PSG data. The recent development of computational technologies has led to a big leap in the usability of range imaging sensors. New depth sensors are smaller, have a higher sampling rate, with better resolution, and have bigger precision. They are widely used for computer vision in robotics, but they can be used as non-contact and non-invasive systems for monitoring breathing and its features. The breathing rate can be easily represented as the frequency of a recorded signal. All tested depth sensors (MS Kinect v2, RealSense SR300, R200, D415 and D435) are capable of recording depth data with enough precision in depth sensing and sampling frequency in time (20–35 frames per second (FPS)) to capture breathing rate. The spectral analysis shows a breathing rate between 0.2 Hz and 0.33 Hz, which corresponds to the breathing rate of an adult person during sleep. To test the quality of breathing signal processed by the proposed workflow, a neural network classifier (simple competitive NN) was trained on a set of 57 whole night polysomnographic records with a classification of sleep apneas by a sleep specialist. The resulting classifier can mark all apnea events with 100% accuracy when compared to the classification of a sleep specialist, which is useful to estimate the number of events per hour. When compared to the classification of polysomnographic breathing signal segments by a sleep specialist, which is used for calculating length of the event, the classifier has an F 1 score of 92.2% Accuracy of 96.8% (sensitivity 89.1% and specificity 98.8%). The classifier also proves successful when tested on breathing signals from MS Kinect v2 and RealSense R200 with simulated sleep apnea events. The whole process can be fully automatic after implementation of automatic chest area segmentation of depth data. [ABSTRACT FROM AUTHOR]

Current consumer wearable devices such as smartwatches mostly rely on touchscreen-based user interfaces. Even though touch-based user interfaces help smartphone users quickly adapt to wearable devices with touchscreens, there exist several limitations. In this paper, we propose a non-touchscreen tactile wearable interface as an alternative to touchscreens on wearable devices. We designed and implemented a joystick-integrated smartwatch prototype to demonstrate our non-touchscreen tactile wearable interface. We iteratively improved and updated our prototype to improve and polish interaction ideas and prototype integration. To show feasibility of our approach, we compared and contrasted form factors of our prototype against the latest nine commercial smartwatches in terms of their dimensions. We also show response time and accuracy of our wearable interface to discuss our rationale for an alternative and usable wearable UI. With the proposed tactile wearable user interface, we believe our approach may serve as a cohesive single interaction device to enable various cross-device interaction scenarios and applications. [ABSTRACT FROM AUTHOR]

AMiCUS is a human–robot interface that enables tetraplegics to control an assistive robotic arm in real-time using only head motion, allowing them to perform simple manipulation tasks independently. The interface may be used as a standalone system or to provide direct control as part of a semi-autonomous system. Within this work, we present our new gesture-free prototype AMiCUS 2.0, which has been designed with special attention to accessibility and ergonomics. As such, AMiCUS 2.0 addresses the needs of tetraplegics with additional impairments that may come along with multiple sclerosis. In an experimental setup, both AMiCUS 1.0 and 2.0 are compared with each other, showing higher accessibility and usability for AMiCUS 2.0. Moreover, in an activity of daily living, a proof-of-concept is provided that an individual with progressed multiple sclerosis is able to operate the robotic arm in a temporal and functional scope, as would be necessary to perform direct control tasks for use in a commercial semi-autonomous system. The results indicate that AMiCUS 2.0 makes an important step towards closing the gaps of assistive technology, being accessible to those who rely on such technology the most. [ABSTRACT FROM AUTHOR]

The recent advancements in the fields of artificial intelligence (AI) and machine learning (ML) have affected several research fields, leading to improvements that could not have been possible with conventional optimization techniques. Among the sectors where AI/ML enables a plethora of opportunities, industrial manufacturing can expect significant gains from the increased process automation. At the same time, the introduction of the Industrial Internet of Things (IIoT), providing improved wireless connectivity for real-time manufacturing data collection and processing, has resulted in the culmination of the fourth industrial revolution, also known as Industry 4.0. In this survey, we focus on the vital processes of fault detection, prediction and prevention in Industry 4.0 and present recent developments in ML-based solutions. We start by examining various proposed cloud/fog/edge architectures, highlighting their importance for acquiring manufacturing data in order to train the ML algorithms. In addition, as faults might also occur from sources beyond machine degradation, the potential of ML in safeguarding cyber-security is thoroughly discussed. Moreover, a major concern in the Industry 4.0 ecosystem is the role of human operators and workers. Towards this end, a detailed overview of ML-based human–machine interaction techniques is provided, allowing humans to be in-the-loop of the manufacturing processes in a symbiotic manner with minimal errors. Finally, open issues in these relevant fields are given, stimulating further research. [ABSTRACT FROM AUTHOR]

This paper presents a framework for the evaluation of system complexity and utility and the identification of bottlenecks in the deployment of field-based, high-throughput phenotyping (FB-HTP) systems. Although the capabilities of technology used for high-throughput phenotyping has improved and costs decreased, there have been few, if any, successful attempts at developing turnkey field-based phenotyping systems. To identify areas for future improvement in developing turnkey FB-HTP solutions, a framework for evaluating their complexity and utility was developed and applied to total of 10 case studies to highlight potential barriers in their development and adoption. The framework performs system factorization and rates the complexity and utility of subsystem factors, as well as each FB-HTP system as a whole, and provides data related to the trends and relationships within the complexity and utility factors. This work suggests that additional research and development are needed focused around the following areas: (i) data handling and management, specifically data transfer from the field to the data processing pipeline, (ii) improved human-machine interaction to facilitate usability across multiple users, and (iii) design standardization of the factors common across all FB-HTP systems to limit the competing drivers of system complexity and utility. This framework can be used to evaluate both previously developed and future proposed systems to approximate the overall system complexity and identify areas for improvement prior to implementation. [ABSTRACT FROM AUTHOR]

Within this work we present AMiCUS, a Human-Robot Interface that enables tetraplegics to control a multi-degree of freedom robot arm in real-time using solely head motion, empowering them to perform simple manipulation tasks independently. The article describes the hardware, software and signal processing of AMiCUS and presents the results of a volunteer study with 13 able-bodied subjects and 6 tetraplegics with severe head motion limitations. As part of the study, the subjects performed two different pick-and-place tasks. The usability was assessed with a questionnaire. The overall performance and the main control elements were evaluated with objective measures such as completion rate and interaction time. The results show that the mapping of head motion onto robot motion is intuitive and the given feedback is useful, enabling smooth, precise and efficient robot control and resulting in high user-acceptance. Furthermore, it could be demonstrated that the robot did not move unintendedly, giving a positive prognosis for safety requirements in the framework of a certification of a product prototype. On top of that, AMiCUS enabled every subject to control the robot arm, independent of prior experience and degree of head motion limitation, making the system available for a wide range of motion impaired users. [ABSTRACT FROM AUTHOR]