Search

As in manufacturing with its Industry 4.0 transformation, the enormous potential of artificial intelligence (AI) is also being recognized in the construction industry. Specifically, the equipment-intensive construction industry can benefit from using AI. AI applications can leverage the data recorded by the numerous sensors on machines and mirror them in a digital twin. Analyzing the digital twin can help optimize processes on the construction site and increase productivity. We present a case from special foundation engineering: the machine production of bored piles. We introduce a hierarchical classification for activity recognition and apply a hybrid deep learning model based on convolutional and recurrent neural networks. Then, based on the results from the activity detection, we use discrete-event simulation to predict construction progress. We highlight the difficulty of defining the appropriate modeling granularity. While activity detection requires equipment movement, simulation requires knowledge of the production flow. Therefore, we present a flow-based production model that can be captured in a modularized process catalog. Overall, this paper aims to illustrate modeling using digital-twin technologies to increase construction process improvement in practice. [ABSTRACT FROM AUTHOR]

Kinematic motion detection aims to determine a person's actions based on activity data. Human kinematic motion detection has many valuable applications in health care, such as health monitoring, preventing obesity, virtual reality, daily life monitoring, assisting workers during industry manufacturing, caring for the elderly. Computer vision-based activity recognition is challenging due to problems such as partial occlusion, background clutter, appearance, lighting, viewpoint, and changes in scale. Our research aims to detect human kinematic motions such as walking or running using smartphones' sensor data within a high-performance framework. An existing dataset based on smartphones' gyroscope and accelerometer sensor values is utilized for the experiments in our study. Sensor exploratory data analysis was conducted in order to identify valuable patterns and insights from sensor values. The six hyperparameters, tunned artificial indigence-based machine learning, and deep learning techniques were applied for comparison. Extensive experimentation showed that the ensemble learning-based novel ERD (ensemble random forest decision tree) method outperformed other state-of-the-art studies with high-performance accuracy scores. The proposed ERD method combines the random forest and decision tree models, which achieved a 99% classification accuracy score. The proposed method was successfully validated with the k-fold cross-validation approach. [ABSTRACT FROM AUTHOR]

Recent advances in deep neural networks have been successfully demonstrated with fairly good accuracy for multi-class activity identification. However, existing methods have limitations in achieving complex spatial-temporal dependencies. In this work, we design two stream fusion attention (2SFA) connected to a temporal bidirectional gated recurrent unit (GRU) one-layer model and classified by prediction voting classifier (PVC) to recognize the action in a video. Particularly in the proposed deep neural network (DNN), we present 2SFA for capturing appearance information from red green blue (RGB) and motion from optical flow, where both streams are correlated by proposed fusion attention (FA) as the input of a temporal network. On the other hand, the temporal network with a bi-directional temporal layer using a GRU single layer is preferred for temporal understanding because it yields practical merits against six topologies of temporal networks in the UCF101 dataset. Meanwhile, the new proposed classifier scheme called PVC employs multiple nearest class mean (NCM) and the SoftMax function to yield multiple features outputted from temporal networks, and then votes their properties for high-performance classifications. The experiments achieve the best average accuracy of 70.8% in HMDB51 and 91.9%, the second best in UCF101 in terms of 2DConvNet for action recognition. [ABSTRACT FROM AUTHOR]

Building classification models in activity recognition is based on the concept of exchangeability. While splitting the dataset into training and test sets, we assume that the training set is exchangeable with the test set and expect good classification performance. However, this assumption is invalid due to subject variability of the training and test sets due to age differences. This happens when the classification models are trained with adult dataset and tested it with elderly dataset. This study investigates the effects of subject variability on activity recognition using inertial sensor. Two different datasets—one locally collected from 15 elders and another public from 30 adults with eight types of activities—were used to evaluate the assessment techniques using ten-fold cross-validation. Three sets of experiments have been conducted: experiments on the public dataset only, experiments on the local dataset only, and experiments on public (as training) and local (as test) datasets using machine learning and deep learning classifiers including single classifiers (Support Vector Machine, Decision Tree, K-Nearest Neighbors), ensemble classifiers (Adaboost, Random Forest, and XGBoost), and Convolutional Neural Network. The experimental results show that there is a significant performance drop in activity recognition on different subjects with different age groups. It demonstrates that on average the drop in recognition accuracy is 9.75 and 12% for machine learning and deep learning models respectively. This confirms that subject variability concerning age is a valid problem that degrades the performance of activity recognition models. [ABSTRACT FROM AUTHOR]

Aging in place has garnered a lot of interest in the past decade among researchers and politicians. It presents itself as a humane and cost-efficient solution to the worsening problem related to the financing and staffing of our fragile healthcare ecosystem. In that regard, smart environments could help monitor the daily activities of the person and provide information regarding the health status and any potential problem necessitating immediate assistance. To do so, many teams, including ours, have been working on human activity recognition from distributed sensors. Nevertheless, it is still very challenging due to the difficulty of amassing enough data to learn activity models that generalizes well across different residents and different smart environments. Moreover, whenever one wants to add a new activity to the set of recognizable activities, it requires to gather additional data with label. The whole process is prohibitively costly and time consuming. Therefore, in this paper, our team proposed to explore web mining in order to build activity model in an unsupervised fashion. More specifically, using the results of two popular image search engines with automated queries and with the help of object detection/classification models, we learned the set of key objects involved in the realization of activities of daily life. A total of 108 configurations were tested. The experiments showed that the key objects related to activities can be easily extracted with a good accuracy. [ABSTRACT FROM AUTHOR]

Human activity recognition (HAR) has the potential to significantly impact applications such as health monitoring, context-aware systems, transportation, robotics, and smart cities. Because of the prevalence of wireless devices, the Wi-Fi-based approach has attracted a lot of attention among other existing methods such as sensor-based and vision-based HAR. Wi-Fi devices can be used to distinguish between daily activities such as "walking," "running," and "sleeping," which affect Wi-Fi signal propagation. This paper proposes a Deep Learning method for HAR tasks that makes use of channel state information (CSI). We convert the CSI data to RGB images and classify the activity recognition using a 2D-Convolutional Neural Network (CNN). We evaluate the performance of the proposed method on two publicly available datasets for CSI data. Our experiments show that converting data into RGB images improves performance and accuracy compared to our previous method by at least 5%. [ABSTRACT FROM AUTHOR]

Human-robot teams collaborating to achieve tasks under various conditions, especially in unstructured, dynamic environments will require robots to adapt autonomously to a human teammate’s state. An important element of such adaptation is the robot’s ability to infer the human teammate’s tasks. Environmentally embedded sensors (e.g., motion capture and cameras) are infeasible in such environments for task recognition, but wearable sensors are a viable task recognition alternative. Human-robot teams will perform a wide variety of composite and atomic tasks, involving multiple activity components (i.e., gross motor, fine-grained motor, tactile, visual, cognitive, speech and auditory) that may occur concurrently. A robot’s ability to recognize the human’s composite, concurrent tasks is a key requirement for realizing successful teaming. Over a hundred task recognition algorithms across multiple activity components are evaluated based on six criteria: sensitivity, suitability, generalizability, composite factor, concurrency and anomaly awareness. The majority of the reviewed task recognition algorithms are not viable for human-robot teams in unstructured, dynamic environments, as they only detect tasks from a subset of activity components, incorporate non-wearable sensors, and rarely detect composite, concurrent tasks across multiple activity components.

Due to its low cost and privacy protection, Channel-State-Information (CSI)-based activity detection has gained interest recently. However, to achieve high accuracy, which is challenging in practice, a significant number of training samples are required. To address the issues of the small sample size and cross-scenario in neural network training, this paper proposes a WiFi human activity-recognition system based on transfer learning using an antenna array: Wi-AR. First, the Intel5300 network card collects CSI signal measurements through an antenna array and processes them with a low-pass filter to reduce noise. Then, a threshold-based sliding window method is applied to extract the signal of independent activities, which is further transformed into time–frequency diagrams. Finally, the produced diagrams are used as input to a pretrained ResNet18 to recognize human activities. The proposed Wi-AR was evaluated using a dataset collected in three different room layouts. The testing results showed that the suggested Wi-AR recognizes human activities with a consistent accuracy of about 94%, outperforming the other conventional convolutional neural network approach.

Parkinson’s disease (PD) is the second most prevalent dementia in the world. Wearable technology has been useful in the computer-aided diagnosis and long-term monitoring of PD in recent years. The fundamental issue remains how to assess the severity of PD using wearable devices in an efficient and accurate manner. However, in the real-world free-living environment, there are two difficult issues, poor annotation and class imbalance, both of which could potentially impede the automatic assessment of PD. To address these challenges, we propose a novel framework for assessing the severity of PD patient’s in a free-living environment. Specifically, we use clustering methods to learn latent categories from the same activities, while latent Dirichlet allocation (LDA) topic models are utilized to capture latent features from multiple activities. Then, to mitigate the impact of data imbalance, we augment bag-level data while retaining key instance prototypes. To comprehensively demonstrate the efficacy of our proposed framework, we collected a dataset containing wearable-sensor signals from 83 individuals in real-life free-living conditions. The experimental results show that our framework achieves an astounding 73.48% accuracy in the fine-grained (normal, mild, moderate, severe) classification of PD severity based on hand movements. Overall, this study contributes to more accurate PD self-diagnosis in the wild, allowing doctors to provide remote drug intervention guidance.

Computer vision (CV)-based recognition approaches have accelerated the automation of safety and progress monitoring on construction sites. However, limited studies have explored its application in process-based quality control of construction works, especially for concealed work. In this study, a framework is developed to facilitate process-based quality control utilizing Spatial–Temporal Graph Convolutional Networks (ST-GCNs). To test this model experimentally, we used an on-site collected plastering work video dataset to recognize construction activities. An ST-GCN model was constructed to identify the four primary activities in plastering works, which attained 99.48% accuracy on the validation set. Then, the ST-GCN model was employed to recognize the activities of three extra videos, which represented a process with four activities in the correct order, a process without the activity of fiberglass mesh covering, and a process with four activities but in the wrong order, respectively. The results indicated that activity order could be clearly withdrawn from the activity recognition result of the model. Hence, it was convenient to judge whether key activities were missing or in the wrong order. This study has identified a promising framework that has the potential to the development of active, real-time, process-based quality control at construction sites.

Mobile fitness applications provide the opportunity to show users real-time feedback on their current fitness activity. For such applications, it is essential to accurately track the user’s current fitness activity using available mobile sensors, such as inertial measurement units (IMUs). Convolutional neural networks (CNNs) have been shown to produce strong results in different time series classification tasks, including the recognition of daily living activities. However, fitness activities can present unique challenges to the human activity recognition task (HAR), including greater similarity between individual activities and fewer available data for model training. In this paper, we evaluate the applicability of CNNs to the fitness activity recognition task (FAR) using IMU data and determine the impact of input data size and sensor count on performance. For this purpose, we adapted three existing CNN architectures to the FAR task and designed a fourth CNN variant, which we call the scaling fully convolutional network (Scaling-FCN). We designed a preprocessing pipeline and recorded a running exercise data set with 20 participants, in which we evaluated the respective recognition performances of the four networks, comparing them with three traditional machine learning (ML) methods commonly used in HAR. Although CNN architectures achieve at least 94% test accuracy in all scenarios, two traditional ML architectures surpass them in the default scenario, with support vector machines (SVMs) achieving 99.00 ± 0.34% test accuracy. The removal of all sensors except one foot sensor reduced the performance of traditional ML architectures but improved the performance of CNN architectures on our data set, with our Scaling-FCN reaching the highest accuracy of 99.86 ± 0.11% on the test set. Our results suggest that CNNs are generally well suited for fitness activity recognition, and noticeable performance improvements can be achieved if sensors are dropped selectively, although traditional ML architectures can still compete with or even surpass CNNs when favorable input data are utilized.

Abstract Animal behavioural responses to the environment ultimately affect their survival. Monitoring animal fine-scale behaviour may improve understanding of animal functional response to the environment and provide an important indicator of the welfare of both wild and domesticated species. In this study, we illustrate the application of collar-attached acceleration sensors for investigating reindeer fine-scale behaviour. Using data from 19 reindeer, we tested the supervised machine learning algorithms Random forests, Support vector machines, and hidden Markov models to classify reindeer behaviour into seven classes: grazing, browsing low from shrubs or browsing high from trees, inactivity, walking, trotting, and other behaviours. We implemented leave-one-subject-out cross-validation to assess generalizable results on new individuals. Our main results illustrated that hidden Markov models were able to classify collar-attached accelerometer data into all our pre-defined behaviours of reindeer with reasonable accuracy while Random forests and Support vector machines were biased towards dominant classes. Random forests using 5-s windows had the highest overall accuracy (85%), while hidden Markov models were able to best predict individual behaviours and handle rare behaviours such as trotting and browsing high. We conclude that hidden Markov models provide a useful tool to remotely monitor reindeer and potentially other large herbivore species behaviour. These methods will allow us to quantify fine-scale behavioural processes in relation to environmental events.

Inertial sensors have become increasingly popular in human activity classification due to their ease of use and affordability. This paper proposes a novel algorithm for human activity recognition that is a combination of a high-gain observer and deep learning computer vision classification algorithms. The nonlinear high-gain observer designed using Lyapunov analysis accurately estimates the attitude of the chest of a human subject using measurements from a single Inertial Measurement Unit (IMU). The signals processed by the observer are then converted into spectrograms to obtain “images” of the frequency response of the signals. The images for activities from a dataset of 7 human subjects are annotated and used for training/ fine-tuning of several well-known deep learning algorithms for image processing. The results from the best combination of our algorithms shows an exceptional accuracy of 98% for activity recognition. Using deep learning computer vision algorithms, this paper shows how to perform transfer learning from networks pre-trained on millions of images, thus showing how we can train a powerful deep learning network for activity recognition even with just small datasets. The algorithm that uses the high gain observer is shown to perform significantly better than an algorithm based on raw accelerometer and gyro signals.

Remote health monitoring plays a significant role in research areas related to medicine, neurology, rehabilitation, and robotic systems. These applications include Human Activity Recognition (HAR) using wearable sensors, signal processing, mathematical methods, and machine learning to improve the accuracy of remote health monitoring systems. To improve the detection and accuracy of human activity recognition, we create a novel method to reduce the complexities of extracting features using the HuGaDB dataset. Our model extracts power spectra; due to the high dimensionality of features, sliding windows techniques are used to determine frequency bandwidth automatically, where an improved QRS algorithm selects the first dominant spectrum amplitude. In addition, the bandwidth algorithm has been used to reduce the dimensionality of data, remove redundant dimensions, and improve feature extraction. In this work, we have considered widely used machine learning classifiers. Our proposed method was evaluated using the accelerometer angles spectrum installed in six parts of the body and then reducing the bandwidth to know the evolution. Our approach attains an accuracy rate of 95.1% in the HuGaDB dataset with 70% of bandwidth, outperforming others in the human activity recognition accuracy. [ABSTRACT FROM AUTHOR]

Activity recognition in unmanned aerial vehicle (UAV) surveillance is addressed in various computer vision applications such as image retrieval, pose estimation, object detection, object detection in videos, object detection in still images, object detection in video frames, face recognition, and video action recognition. In the UAV-based surveillance technology, video segments captured from aerial vehicles make it challenging to recognize and distinguish human behavior. In this research, to recognize a single and multi-human activity using aerial data, a hybrid model of histogram of oriented gradient (HOG), mask-regional convolutional neural network (Mask-RCNN), and bidirectional long short-term memory (Bi-LSTM) is employed. The HOG algorithm extracts patterns, Mask-RCNN extracts feature maps from the raw aerial image data, and the Bi-LSTM network exploits the temporal relationship between the frames for the underlying action in the scene. This Bi-LSTM network reduces the error rate to the greatest extent due to its bidirectional process. This novel architecture generates enhanced segmentation by utilizing the histogram gradient-based instance segmentation and improves the accuracy of classifying human activities using the Bi-LSTM approach. Experimental outcomes demonstrate that the proposed model outperforms the other state-of-the-art models and has achieved 99.25% accuracy on the YouTube-Aerial dataset. [ABSTRACT FROM AUTHOR]

Smart home automation is protective and preventive measures that are taken to monitor elderly people in a non-intrusive manner using simple and pervasive sensors termed Ambient Assistive Living. The smart home produces a large volume of sensor activations to predict an elder's health status to improve the quality of life and independent living. Machine learning techniques are very familiar and popular in recognizing single resident activities using such a sensor reading. However, multi-resident activities are more complex, and no correlation exists between sensor readings and activities. Recently deep learning and graphical models have been proposed to solve this problem, but it consumes more time to train the model. Moreover, models are primarily executed in parallel, or multi-resident activity recognition has been converted to single for classifying events. This paper proposes a Multi-label Multi-output Hybrid Sequential Model (MLMO-HSM), a feature engineering approach with a hybrid sequential model to recognize the multi-resident activities concurrently in the shortest time. Experimentation has been performed with various machine learning, graphical and deep learning models at the different sizes of the ARAS dataset to validate the efficiency of the proposed model in terms of activity recognition and computation time. [ABSTRACT FROM AUTHOR]

Human pose estimation (HPE) is a procedure for determining the structure of the body pose and it is considered a challenging issue in the computer vision (CV) communities. HPE finds its applications in several fields namely activity recognition and human-computer interface. Despite the benefits of HPE, it is still a challenging process due to the variations in visual appearances, lighting, occlusions, dimensionality, etc. To resolve these issues, this paper presents a squirrel search optimization with a deep convolutional neural network for HPE (SSDCNN-HPE) technique. The major intention of the SSDCNN-HPE technique is to identify the human pose accurately and efficiently. Primarily, the video frame conversion process is performed and pre-processing takes place via bilateral filtering-based noise removal process. Then, the EfficientNet model is applied to identify the body points of a person with no problem constraints. Besides, the hyperparameter tuning of the EfficientNet model takes place by the use of the squirrel search algorithm (SSA). In the final stage, the multiclass support vector machine (M-SVM) technique was utilized for the identification and classification of human poses. The design of bilateral filtering followed by SSA based EfficientNet model for HPE depicts the novelty of the work. To demonstrate the enhanced outcomes of the SSDCNN-HPE approach, a series of simulations are executed. The experimental results reported the betterment of the SSDCNN-HPE system over the recent existing techniques in terms of different measures. [ABSTRACT FROM AUTHOR]

Laying hen activities in modern intensive housing systems can dramatically influence the policies needed for the optimal management of such systems. Intermittent monitoring of different behaviors during daytime cannot provide a good overview, since daily behaviors are not equally distributed over the day. This paper investigates the application of deep learning technology in the automatic recognition of laying hen behaviors equipped with body-worn inertial measurement unit (IMU) modules in poultry systems. Motivated by the human activity recognition literature, a sophisticated preprocessing method is tailored on the time-series data of IMU, transforming it into the form of so-called activity images to be recognized by the deep learning models. The diverse range of behaviors a laying hen can exhibit are categorized into three classes: low-, medium-, and high-intensity activities, and various recognition models are trained to recognize these behaviors in real-time. Several ablation studies are conducted to assess the efficacy and robustness of the developed models against variations and limitations common for an in situ practical implementation. Overall, the best trained model on the full-feature acquired data achieves a mean accuracy of almost 100%, where the whole process of inference by the model takes less than 30 milliseconds. The results suggest that the application of deep learning technology for activity recognition of individual hens has the potential to accurately aid successful management of modern poultry systems. [ABSTRACT FROM AUTHOR]

Human activity recognition (HAR) is a complex and multifaceted problem. The research community has reported numerous approaches to perform HAR. Along with HAR approaches, various surveys have revealed HAR trends in various environments and applications. HAR is linked to a variety of technology-dependent daily life systems, such as human–computer interaction systems, security surveillance, video surveillance, healthcare surveillance, robotics, content-based information retrieval, and monitoring systems. Because of technological advancements, HAR trends change quickly and necessitate an up-to-date and broader perspective. This study offers an HAR taxonomy, which includes online/offline HAR, multimodal/unimodal HAR, handcrafted feature-based, and learning-based approaches. This study attempts to present the multidisciplinary nature of HAR, such as application areas, activity types, task complexities, benchmark datasets, and/methods. This research includes a comparative analysis of state-of-the-art HAR methods and a discussion of popular datasets. The selected studies have been categorized using taxonomy, and different attributes such as activity complexity, dataset size, and recognition rate have been used for their analysis. The comparative analysis of HAR approaches has also helped to highlight domain challenges and open research directions for HAR researchers to follow. [ABSTRACT FROM AUTHOR]

Smart workplace Internet of Things (IoT) solutions rely on several sensors deployed efficiently in the workplace environment to collect accurate data to meet system goals. A vital issue for these sensor-based IoT solutions is privacy. Ideally, the occupants must be monitored discreetly, and the strategies for maintaining privacy are dependent on the nature of the data required. This paper proposes a new sensor design approach for IoT solutions in the workplace that protects occupants' privacy. We focus on a novel sensor that autonomously detects and captures human movements in the office to monitor a person's sedentary behavior. The sensor guides an eHealth solution that uses continuous feedback about desk behaviors to prompt healthy movement breaks for seated workers. The proposed sensor and its privacy-preserving characteristics can enhance the eHealth solution system's performance. Compared to self-reporting, intrusive, and other data collection techniques, this sensor can collect the information reliably and timely. We also present the data analysis specific to this new sensor that measures two physical distance parameters in real-time and uses their difference to determine human actions. This architecture aims to collect precise data at the sensor design level rather than to protect privacy during the data analysis phase. [ABSTRACT FROM AUTHOR]

The electromyogram (EMG) is a waveform representation of the action potential generated by muscle cells using electrodes. EMG acquired using surface electrodes is called surface EMG (sEMG), and it is the acquisition of muscle action potentials transmitted by volume conduction from the skin. Surface electrodes require disposable conductive gel or adhesive tape to be attached to the skin, which is costly to run, and the tape is hard on the skin when it is removed. Muscle activity can be evaluated by acquiring muscle potentials and analyzing quantitative, temporal, and frequency factors. It is also possible to evaluate muscle fatigue because the frequency of the EMG becomes lower as the muscle becomes fatigued. Research on human activity recognition from EMG signals has been actively conducted and applied to systems that support arm and hand functions. This paper proposes a method for recognizing the muscle activity state of the arm using pulse wave data (PPG: Photoplethysmography) and a method for estimating EMG using pulse wave data. This paper assumes that the PPG sensor is worn on the user's wrist to measure the heart rate. The user also attaches an elastic band to the upper arm, and when the user exerts a force on the arm, the muscles of the upper arm contract. The arteries are then constricted, and the pulse wave measured at the wrist becomes weak. From the change in the pulse wave, the muscle activity of the arm can be recognized and the number of action potentials of the muscle can be estimated. From the evaluation experiment with five subjects, three types of muscle activity were recognized with 80+%, and EMG was estimated with approximately 20% error rate. [ABSTRACT FROM AUTHOR]

Activity monitoring has become a necessary demand for weak people to guarantee their safety. The paper proposed a Parallel Training Logical Execution (PTLE) system using machine learning (ML) models on a microelectromechanical system (MEMS) accelerometer to detect coughs, falls, and other normal activities. When there are many categories, the ML prediction can be confused between these activities with each other. The PTLE system trains several models in parallel with more specific activity classes in each dataset. The shared tasks between parallel models relieve the complexity for a single one. There are six additional parameters for accelerometer characteristics, which were calculated from three axes accelerations as input features to improve the ML's consciousness. Once all models were trained, the system was ready to receive the input accelerations and activated the logical flow to manage link operation between these ML models for output predictions. Random Forest (RF) had the highest potential among the ML classification algorithms after the validation. In the experiment, the comparison between the PTLE model and the regular ML model were carried out with real-time data from an M5stickC wearable device on the user's chest to the trained models on PC. The result showed the advancement of the proposed method in term of precision, recall, F1-score with an overall accuracy of 98% in the real-time test. The accelerations from the wearable device were sent to ML models via Wi-Fi with Message Queue Telemetry Transport (MQTT) broker, and the activity predictions were transferred to the cloud for the family members or doctor care based on Internet of Things (IoT) communication. [ABSTRACT FROM AUTHOR]

Recently, the scientific community has placed great emphasis on the recognition of human activity, especially in the area of health and care for the elderly. There are already practical applications of activity recognition and unusual conditions that use body sensors such as wrist-worn devices or neck pendants. These relatively simple devices may be prone to errors, might be uncomfortable to wear, might be forgotten or not worn, and are unable to detect more subtle conditions such as incorrect postures. Therefore, other proposed methods are based on the use of images and videos to carry out human activity recognition, even in open spaces and with multiple people. However, the resulting increase in the size and complexity involved when using image data requires the use of the most recent advanced machine learning and deep learning techniques. This paper presents an approach based on deep learning with attention to the recognition of activities from multiple frames. Feature extraction is performed by estimating the pose of the human skeleton, and classification is performed using a neural network based on Bidirectional Encoder Representation of Transformers (BERT). This algorithm was trained with the UP-Fall public dataset, generating more balanced artificial data with a Generative Adversarial Neural network (GAN), and evaluated with real data, outperforming the results of other activity recognition methods using the same dataset. [ABSTRACT FROM AUTHOR]

Most industrial workplaces involving robots and other apparatus operate behind the fences to remove defects, hazards, or casualties. Recent advancements in machine learning can enable robots to co-operate with human co-workers while retaining safety, flexibility, and robustness. This article focuses on the computation model, which provides a collaborative environment through intuitive and adaptive human–robot interaction (HRI). In essence, one layer of the model can be expressed as a set of useful information utilized by an intelligent agent. Within this construction, a vision-sensing modality can be broken down into multiple layers. The authors propose a human-skeleton-based trainable model for the recognition of spatiotemporal human worker activity using LSTM networks, which can achieve a training accuracy of 91.365%, based on the InHARD dataset. Together with the training results, results related to aspects of the simulation environment and future improvements of the system are discussed. By combining human worker upper body positions with actions, the perceptual potential of the system is increased, and human–robot collaboration becomes context-aware. Based on the acquired information, the intelligent agent gains the ability to adapt its behavior according to its dynamic and stochastic surroundings. [ABSTRACT FROM AUTHOR]

Deep convolutional neural networks (CNNs) achieve state-of-the-art performance in wearable human activity recognition (HAR), which has become a new research trend in ubiquitous computing scenario. Increasing network depth or width can further improve accuracy. However, in order to obtain the optimal HAR performance on mobile platform, it has to consider a reasonable tradeoff between recognition accuracy and resource consumption. Improving the performance of CNNs without increasing memory and computational burden is more beneficial for HAR. In this article, we first propose a new CNN that uses hierarchical-split (HS) idea for a large variety of HAR tasks, which is able to enhance multiscale feature representation ability via capturing a wider range of receptive fields of human activities within one feature layer. Experiments conducted on benchmarks demonstrate that the proposed HS module is an impressive alternative to baseline models with similar model complexity, and can achieve higher recognition performance (e.g., 97.28%, 93.75%, 99.02%, and 79.02% classification accuracies) on UCI-HAR, PAMAP2, WISDM, and UNIMIB-SHAR. Extensive ablation studies are performed to evaluate the effect of the variations of receptive fields on classification performance. Finally, we demonstrate that multiscale receptive fields can help to learn more discriminative features (achieving 94.10% SOTA accuracy) in weakly labeled HAR dataset. [ABSTRACT FROM AUTHOR]

The human pose estimation is a significant issue that has been taken into consideration in the computer vision network for recent decades. It is a vital advance toward understanding individuals in videos and still images. In simple terms, a human pose estimation model takes in an image or video and estimates the position of a person's skeletal joints in either 2D or 3D space. Several studies on human posture estimation can be found in the literature, however, they center around a specific class; for instance, model-based methodologies or human movement investigation, and so on. Later, various Deep Learning (DL) algorithms came into existence to overcome the difficulties which were there in the earlier approaches. In this study, an exhaustive review of human pose estimation (HPE), including milestone work and recent advancements is carried out. This survey discusses the different two-dimensional (2D) and three-dimensional human (3D) pose estimation techniques along with their classical and deep learning approaches which provide the solution to the various computer vision problems. Moreover, the paper also considers the different deep learning models used in pose estimation, and the analysis of 2D and 3D datasets is done. Some of the evaluation metrics used for estimating human poses are also discussed here. By knowing the direction of the individuals, HPE opens a road for a few real-life applications some of which are talked about in this study. [ABSTRACT FROM AUTHOR]

Human activity recognition (HAR) extensively uses wearable inertial sensors since this data source provides the most information for non-visual datasets' time series. HAR research has advanced significantly in recent years due to the proliferation of wearable devices with sensors. To improve recognition performance, HAR researchers have extensively investigated other sources of biosignals, such as a photoplethysmograph (PPG), for this task. PPG sensors measure the rate at which blood flows through the body, and this rate is regulated by the heart's pumping action, which constantly occurs throughout the body. Even though detecting body movement and gestures was not initially the primary purpose of PPG signals, we propose an innovative method for extracting relevant features from the PPG signal and use deep learning (DL) to predict physical activities. To accomplish the purpose of our study, we developed a deep residual network referred to as PPG-NeXt, designed based on convolutional operation, shortcut connections, and aggregated multi-branch transformation to efficiently identify different types of daily life activities from the raw PPG signal. The proposed model achieved more than 90% prediction F1-score from experimental results using only PPG data on the three benchmark datasets. Moreover, our results indicate that combining PPG and acceleration signals can enhance activity recognition. Although, both biosignals—electrocardiography (ECG) and PPG—can differentiate between stationary activities (such as sitting) and non-stationary activities (such as cycling and walking) with a level of success that is considered sufficient. Overall, our results propose that combining features from the ECG signal can be helpful in situations where pure tri-axial acceleration (3D-ACC) models have trouble differentiating between activities with relative motion (e.g., walking, stair climbing) but significant differences in their heart rate signatures. [ABSTRACT FROM AUTHOR]

Federated Learning (FL) has recently received significant interest, thanks to its capability of protecting data privacy. However, existing FL paradigms yield unsatisfactory performance for a wide class of human activity recognition (HAR) applications, since they are oblivious to the intrinsic relationship between data of different users. We propose ClusterFL, a clustering-based federated learning system that can provide high model accuracy and low communication overhead for HAR applications. ClusterFL features a novel clustered multitask federated learning framework that minimizes the empirical training loss of multiple learned models while automatically capturing the intrinsic clustering relationship among the nodes. We theoretically prove the convergence of proposed FL framework for non-convex and strongly convex models and provide the guidance on selection of hyper-parameters for achieving such convergence. Based on the learned cluster relationship, ClusterFL can efficiently drop the nodes that converge slower or have little correlations with others in each cluster, significantly speeding up the convergence while maintaining the accuracy performance. We evaluate the performance of ClusterFL on an NVIDIA edge testbed using four new HAR datasets collected from 145 users. The results show that ClusterFL outperforms several state-of-the-art FL paradigms in terms of overall accuracy and can save more than 50% communication overhead. [ABSTRACT FROM AUTHOR]

Human gait activity recognition is an emerging field of motion analysis that can be applied in various application domains. One of the most attractive applications includes monitoring of gait disorder patients, tracking their disease progression and the modification/evaluation of drugs. This paper proposes a robust, wearable gait motion data acquisition system that allows either the classification of recorded gait data into desirable activities or the identification of common risk factors, thus enhancing the subject's quality of life. Gait motion information was acquired using accelerometers and gyroscopes mounted on the lower limbs, where the sensors were exposed to inertial forces during gait. Additionally, leg muscle activity was measured using strain gauge sensors. As a matter of fact, we wanted to identify different gait activities within each gait recording by utilizing Machine Learning algorithms. In line with this, various Machine Learning methods were tested and compared to establish the best-performing algorithm for the classification of the recorded gait information. The combination of attention-based convolutional and recurrent neural networks algorithms outperformed the other tested algorithms and was individually tested further on the datasets of five subjects and delivered the following averaged results of classification: 98.9% accuracy, 96.8% precision, 97.8% sensitivity, 99.1% specificity and 97.3% F1-score. Moreover, the algorithm's robustness was also verified with the successful detection of freezing gait episodes in a Parkinson's disease patient. The results of this study indicate a feasible gait event classification method capable of complete algorithm personalization. [ABSTRACT FROM AUTHOR]

The boundaries of tracking and sensing solutions are continuously being pushed. A stimulation in this field over recent years is exploiting the properties of millimeter wave (mmWave) radar to achieve simultaneous tracking and sensing of multiple objects. This paper aims to provide a critical analysis of the current literature surrounding multi-object tracking and sensing with short-range mmWave radar. There is significant literature available regarding single-object tracking using mmWave radar, demonstrating the maturity of single-object tracking systems. However, innovative research and advancements are also needed in the field of mmWave radar multi-object tracking, specifically with respect to uniquely identifying multiple target tracks across an interrupted field of view. In this article, we aim to provide an overview of the latest progress in multi-target tracking. In particular, an attempt to phrase the problem space is made by firstly defining a typical multi-object tracking architecture. We then highlight the areas for potential advancements. These areas include sensor fusion, micro-Doppler feature analysis, specialized and generalized activity recognition, gait, tagging and shape profile. Potential multi-object tracking advancements are reviewed and compared with respect to adaptability, performance, accuracy and specificity. Although the majority of the literature reviewed has a focus on human targets, most of the methodologies can be applied to targets consisting of different profiles and characteristics to that of humans. Lastly, future research directions are also discussed to shed light on research opportunities and potential approaches in the open research areas. [ABSTRACT FROM AUTHOR]

Identification of human activity with decent precision is a challenging task in the field of computer vision, especially when applying for surveillance purpose. A rule-based classifier method is proposed in this paper, which is capable of recognizing a view-invariant multiple human activity recognition in real time. A single Kinect sensor is used for the input of RGB-D data in real time. Initially, a skeleton-tracking algorithm is applied. After tracking the skeletons, activities are recognized from each individually tracked skeleton independently. Different rules are defined to recognize discrete skeleton positions and classify a particular order of multiple postures into activities. During the experimentation, we examine about 14 activities and found that the proposed method is robust and efficient concerning multiple views, scaling and phase variation activities during different realistic acts. A self-generated dataset in the controlled environment is used for the experiment. About 2 min of data was collected. Data from two different males were collected for multiple human activities. Experimental results show that the proposed method is flexible and efficient for multiple view activities as well as scale and phase variation activities. It provides a detection accuracy of 98%. [ABSTRACT FROM AUTHOR]

Human activity recognition (HAR) applications have received much attention due to their necessary implementations in various domains, including Industry 5.0 applications such as smart homes, e-health, and various Internet of Things applications. Deep learning (DL) techniques have shown impressive performance in different classification tasks, including HAR. Accordingly, in this article, we develop a comprehensive HAR system based on a novel DL architecture called Multi-ResAtt (multilevel residual network with attention). This model incorporates initial blocks and residual modules aligned in parallel. Multi-ResAtt learns data representations on the inertial measurement units level. Multi-ResAtt integrates a recurrent neural network with attention to extract time-series features and perform activity recognition. We consider complex human activities collected from wearable sensors to evaluate the Multi-ResAtt using three public datasets, Opportunity; UniMiB-SHAR; and PAMAP2. Additionally, we compared the proposed Multi-ResAtt to several DL models and existing HAR systems, and it achieved significant performance. [ABSTRACT FROM AUTHOR]

Sensor-based activity recognition involves the automatic recognition of a user's activity in a smart environment using computational methods. The use of wearable devices and video-based approaches have attracted considerable interest in ubiquitous computing. Nevertheless, these methods have limitations such as issues with privacy invasion, ethics, comfort and obtrusiveness. Environmental sensors are an increasingly promising consideration in the ubiquitous computing domain for long-term monitoring, as these devices are non-invasive to inhabitants, yet certain challenges remain with activity recognition in sensorised environments, for example, addressing the challenge of intraclass variation between activities and reasoning from low-level uncertain information. In an effort to address these challenges, this paper proposes and evaluates the performance of a Radial Basis Function Neural Network approach for activity recognition with environmental sensors. The model is trained using the Localized Generalization Error and focuses on the generalization ability by considering both the training error and stochastic sensitivity measure. This measures the network output fluctuation with respect to the minor perturbation of input, to address the tolerance of the low-level uncertain sensor data. This approach is compared with three benchmark Neural Network approaches, including a popular deep learning approach using an Autoencoder, and it is evaluated with a simulated dataset as well as a number of publicly available datasets. The proposed method has shown advantages over the other models for all four evaluated datasets. This paper provides insights into the importance of model generalization abilities and an initial analysis of the limitation of deep Neural Networks with respect to sensor-based activity recognition. [ABSTRACT FROM AUTHOR]

In this article we present a multi-modal machine learning-based system, which we call ACORN, to analyze videos of school classrooms for the Positive Climate (PC) and Negative Climate (NC) dimensions of the CLASS [1] observation protocol that is widely used in educational research. ACORN uses convolutional neural networks to analyze spectral audio features, the faces of teachers and students, and the pixels of each image frame, and then integrates this information over time using Temporal Convolutional Networks. The audiovisual ACORN's PC and NC predictions have Pearson correlations of 0.55 and 0.63 with ground-truth scores provided by expert CLASS coders on the UVA Toddler dataset (cross-validation on $n=300$ n = 300 15-min video segments), and a purely auditory ACORN predicts PC and NC with correlations of 0.36 and 0.41 on the MET dataset (test set of $n=2000$ n = 2000 videos segments). These numbers are similar to inter-coder reliability of human coders. Finally, using Graph Convolutional Networks we make early strides (AUC=0.70) toward predicting the specific moments (45-90sec clips) when the PC is particularly weak/strong. Our findings inform the design of automatic classroom observation and also more general video activity recognition and summary recognition systems. [ABSTRACT FROM AUTHOR]

WiFi sensing has received recent and significant interest from academia, industry, healthcare professionals, and other caregivers (including family members) as a potential mechanism to monitor our aging population at a distance without deploying devices on users’ bodies. In particular, these methods have the potential to detect critical events such as falls, sleep disturbances, wandering behavior, respiratory disorders, and abnormal cardiac activity experienced by vulnerable people. The interest in such WiFi-based sensing systems arises from practical advantages including its ease of operation indoors as well as ready compliance from monitored individuals. Unlike other sensing methods, such as wearables, camera-based imaging, and acoustic-based solutions, WiFi technology is easy to implement and unobtrusive. This paper reviews the current state-of-the-art research on collecting and analyzing channel state information extracted using ubiquitous WiFi signals, describing a range of healthcare applications and identifying a series of open research challenges, including untapped areas of research and related trends. This work aims to provide an overarching view in understanding the technology and discusses its use-cases from a perspective that considers hardware, advanced signal processing, and data acquisition. [ABSTRACT FROM AUTHOR]

Multi-label activity recognition is extremely challenging, where multiple activities may appear simultaneously or sequentially in a video. While previous works have realized the temporal co-occurrence of activities, the sequential order of activities have been largely overlooked. However, we argue that the sequential order of activities should also be preserved in correlation modeling, because shuffling the order might not form a semantically meaningful video. In this work, we present plug-and-play Simultaneous and Sequential Transformer (SiSe) modules for multi-label activity recognition. Upon frame features of all time steps, SiSe enhances spatio-temporal feature learning for multi-label activity recognition, by capturing the simultaneous and sequential activity correlations. Specifically, we employ a Simultaneous Transformer module to connect multiple activities that probably appear at each frame, and a hierarchical Sequential Transformer module to efficiently capture the sequential activity correlations in an order-preserved manner. Despite the straightforward and class-agnostic design of SiSe, it can outperform state-of-the-art approaches on three multi-label activity recognition benchmarks. In particular, we verify the significance of preserving the sequential order of activities with our Sequential Transformer in correlation modeling. We also conduct ablation studies and visual analysis for better understanding of our SiSe. • A novel SiSe method is proposed for capturing the simultaneous and sequential activity correlations. • Capturing sequential correlations in an order-preserved manner. • Capturing the simultaneous activity correlations. • Efficiency and effectiveness, outperforming existing state-of-the-art methods. [ABSTRACT FROM AUTHOR]

The construction industry still suffers from a low degree of transparency and bears health and safety risks for workers. A well-known disease is the Hand-Arm Vibration Syndrome, which results from the use of high-vibration tools. The recognition of worker activities to calculate a worker's exposure to vibration while operating tools in different modes has received little attention so far. This paper presents a machine-learning-based approach to recognize worker activities involving high-vibration hand-held power tools using acceleration data recorded with a smart-watch. The data is collected in a laboratory utilizing a combi-hammer in various tool settings. A Decision Tree and a Convolutional Pooling Long Short Term Memory (ConvPoolLSTM) model are developed to recognize worker activities on different levels. The ConvPoolLSTM architecture achieves accuracies between 89.1% and 96.1%. This research contributes towards the development of a more accurate, automated and low-cost monitoring system for vibration exposure time calculation and work process monitoring. • Machine learning models can recognize worker activities including power tools. • Collected data includes 202 min of smart-watch sensor data in various settings. • Models show high performance to recognize activities on three levels of details. • The ConvPoolLSTM model achieves up to 96.1% accuracy for tool runtime recognition. • The models can enable automated vibration time calculation on tool usage mode level. [ABSTRACT FROM AUTHOR]

Conducting an analysis of human behavior is an intriguing topic for many researchers. Within this field, machine learning can be applied to classify activities and emotions by analyzing physiological signals. However, the limited size of available databases poses challenges for the generalization of classifiers. This paper proposes a method to enhance the generalization of neural network-based classifiers by intelligently initializing weights for emotion and activity recognition. The signals under consideration are electrocardiogram, thoracic electrical bioimpedance, and electrodermal activity. The database used comprises recordings from 40 subjects performing various tasks that induce emotions and activities. The performance of the proposed method is compared with several standard machine learning and deep learning classifiers typically employed in emotion and activity recognition. This study involves two primary assessments. First is the activity recognition task, encompassing classes such as neutral, emotional, mental, and physical activity, where results close to 20% accuracy are achieved using the three physiological signals. Second, the emotion recognition assessment aims to differentiate between emotions like neutral, sadness, and disgust. An error probability close to 15% is obtained using thoracic electrical bioimpedance and electrodermal activity. The proposed method yields the best results among the approaches evaluated. • Database size limits the classification of the activity and emotions from biosignals. • Smart initialization of the weights in neural networks improves the generalization. • Our artificial network pre-training reduces local minima problems and overfitting. • The method is compared with several machine and deep learning classifiers. [ABSTRACT FROM AUTHOR]

Recent advancements in construction equipment activity recognition are increasingly leveraging multi-sensor signals, indicating a promising future trend. However, existing methods struggle to simultaneously meet high accuracy and real-time performance on resource-limited edge devices. To address this challenge, a Privilege-guided Knowledge Distillation Network (PGKDN) is introduced for excavator activity recognition. This approach harnesses the accuracy benefits of multi-sensor signal input networks while maintaining the deployment advantages of single-sensor signal input networks. Privileged information is transmitted to the student network through knowledge distillation. High-quality transfer of knowledge between networks is achieved with Semantic Guidance Loss (SGL). Consequently, the method not only achieves an optimal balance between recognition accuracy, model complexity, and computational efficiency on edge devices but also presents a viable solution for activity recognition in resource-limited conditions. • The body signals of the excavator are used to recognize its activities. • Transmitting multi-sensor information to the student network using privilege-guided knowledge distillation. • High-quality transfer of information in the knowledge distillation process is achieved. • Higher recognition accuracy while ensuring computational efficiency on edge devices. [ABSTRACT FROM AUTHOR]

The proposal of Integrated Sensing and Communications has once again drawn researchers' attention to WiFi sensing, propelling applications based on WiFi sensing into an advanced stage. However, the current field of activity recognition only identifies fixed categories of activities, neglecting the growing demand for perceiving activity types in real applications over time. In response to the issue, we present WiCAR, a WiFi activity recognition system designed for class incremental scenarios. WiCAR takes antenna array-fused image data as input, employing the Wi-RA model with parallel stacked activation functions as its backbone network. To alleviate the typical catastrophic forgetting issue in class-incremental learning, WiCAR employs a strategy of replaying known data. Additionally, we adopts knowledge distillation to improve accuracy among old samples during the incremental process. To tackle the imbalance in the number of samples between old and new classes, the model is updated through weight alignment. This serious of strategies endows the system with the capability to progressively learn and handle new classes. We conducted extensive experiments to evaluate the system performance. The experimental results demonstrate that our system exhibits excellent performance regardless of the number of tasks, whether tasks are uniform or non-uniform, and the order of task arrivals. The highest average accuracy reaches 96.429%, and even in the presence of six incremental stages, the average accuracy remains at 92.867%. [ABSTRACT FROM AUTHOR]

Participatory exposure research, which tracks behaviour and assesses exposure to stressors like air pollution, traditionally relies on time-activity diaries. This study introduces a novel approach, employing machine learning (ML) to empower laypersons in human activity recognition (HAR), aiming to reduce dependence on manual recording by leveraging data from wearable sensors. Recognising complex activities such as smoking and cooking presents unique challenges due to specific environmental conditions. In this research, we combined wearable environment/ambient and wrist-worn activity/biometric sensors for complex activity recognition in an urban stressor exposure study, measuring parameters like particulate matter concentrations, temperature, and humidity. Two groups, Group H (88 individuals) and Group M (18 individuals), wore the devices and manually logged their activities hourly and minutely, respectively. Prioritising accessibility and inclusivity, we selected three classification algorithms: k-nearest neighbours (IBk), decision trees (J48), and random forests (RF), based on: (1) proven efficacy in existing literature, (2) understandability and transparency for laypersons, (3) availability on user-friendly platforms like WEKA, and (4) efficiency on basic devices such as office laptops or smartphones. Accuracy improved with finer temporal resolution and detailed activity categories. However, when compared to other published human activity recognition research, our accuracy rates, particularly for less complex activities, were not as competitive. Misclassifications were higher for vague activities (resting, playing), while well-defined activities (smoking, cooking, running) had few errors. Including environmental sensor data increased accuracy for all activities, especially playing, smoking, and running. Future work should consider exploring other explainable algorithms available on diverse tools and platforms. Our findings underscore ML’s potential in exposure studies, emphasising its adaptability and significance for laypersons while also highlighting areas for improvement.

Force-based human posture estimation (FPE) provides a valuable alternative when camera-based human motion capturing is impractical. It offers new opportunities for sensor integration in smart products for patient monitoring, ergonomic optimization and sports science. Due to the interdisciplinary research on the topic, an overview of existing methods and the required expertise for their utilization is lacking. This paper presents a systematic review by the PRISMA 2020 review process. In total, 82 studies are selected (59 machine learning (ML)-based and 23 digital human model (DHM)-based posture estimation methods). The ML-based methods use input data from hardware sensors—mostly pressure mapping sensors—and trained ML models for estimating human posture. The ML-based human posture estimation algorithms mostly reach an accuracy above 90%. DHMs, which represent the structure and kinematics of the human body, adjust posture to minimize physical stress. The required expert knowledge for the utilization of these methods and their resulting benefits are analyzed and discussed. DHM-based methods have shown their general applicability without the need for application-specific training but require expertise in human physiology. ML-based methods can be used with less domain-specific expertise, but an application-specific training of these models is necessary.

By observing the actions taken by operators, it is possible to determine the risk level of a work task. One method for achieving this is the recognition of human activity using biosignals and inertial measurements provided to a machine learning algorithm performing such recognition. The aim of this research is to propose a method to automatically recognize physical exertion and reduce noise as much as possible towards the automation of the Job Strain Index (JSI) assessment by using a motion capture wearable device (MindRove armband) and training a quadratic support vector machine (QSVM) model, which is responsible for predicting the exertion depending on the patterns identified. The highest accuracy of the QSVM model was 95.7%, which was achieved by filtering the data, removing outliers and offsets, and performing zero calibration; in addition, EMG signals were normalized. It was determined that, given the job strain index’s purpose, physical exertion detection is crucial to computing its intensity in future work.

In recent years, human activity monitoring and recognition have gained importance in providing valuable information to improve the quality of life. A lack of activity can cause health problems including falling, depression, and decreased mobility. Continuous activity monitoring can be useful to prevent progressive health problems. With this purpose, this study presents a wireless smart insole with four force-sensitive resistors (FSRs) that monitor foot contact states during activities for both indoor and outdoor use. The designed insole is a compact solution and provides walking comfort with a slim and flexible structure. Moreover, the inertial measurement unit (IMU) sensors designed in our previous study were used to collect 3-axis accelerometer and 3-axis gyroscope outputs. Smart insoles were located in the shoe sole for both right and left feet, and two IMU sensors were attached to the thigh area of each leg. The sensor outputs were collected and recorded from forty healthy volunteers for eight different gait-based activities including walking uphill and descending stairs. The obtained datasets were separated into three categories; foot contact states, the combination of acceleration and gyroscope outputs, and a set of all sensor outputs. The dataset for each category was separately fed into deep learning algorithms, namely, convolutional long–short-term memory neural networks. The performance of each neural network for each category type was examined. The results show that the neural network using only foot contact states presents 90.1% accuracy and provides better performance than the combination of acceleration and gyroscope datasets for activity recognition. Moreover, the neural network presents the best results with 93.4% accuracy using a combination of all the data compared with the other two categories.