Search

Researchers have been developing Hand Gesture Recognition (HGR) systems to enhance natural, efficient, and authentic human-computer interaction, especially benefiting those who rely solely on hand gestures for communication. Despite significant progress, automatic and precise identification of hand gestures remains a considerable challenge in computer vision. Recent studies have focused on specific modalities like RGB images, skeleton data, and spatiotemporal interest points. This paper comprehensively reviews HGR techniques and data modalities from 2014 to 2024, exploring advancements in sensor technology and computer vision. We highlight accomplishments using various modalities, including RGB, Skeleton, Depth, Audio, Electromyography (EMG), Electroencephalography (EEG), and Multimodal approaches and identify areas needing further research. We reviewed over 250 articles from prominent databases, focusing on data collection, data settings, and gesture representation. Our review assesses the efficacy of HGR systems through their recognition accuracy and identifies a gap in research on continuous gesture recognition, indicating the need for improved vision-based gesture systems. The field has experienced steady research progress, including advancements in hand-crafted features and deep learning (DL) techniques. Additionally, we report on the promising developments in HGR methods and the area of multimodal approaches. We hope this survey will serve as a potential guideline for diverse data modality-based HGR research.

Gesture recognition has emerged as a significant research domain in computer vision and human-computer interaction. One of the key challenges in gesture recognition is how to select the most useful channels that can effectively represent gesture movements. In this study, we have developed a channel selection algorithm that determines the number and placement of sensors that are critical to gesture classification. To validate this algorithm, we constructed a Force Myography (FMG)-based signal acquisition system. The algorithm considers each sensor as a distinct channel, with the most effective channel combinations and recognition accuracy determined through assessing the correlation between each channel and the target gesture, as well as the redundant correlation between different channels. The database was created by collecting experimental data from 10 healthy individuals who wore 16 sensors to perform 13 unique hand gestures. The results indicate that the average number of channels across the 10 participants was 3, corresponding to an 75% decrease in the initial channel count, with an average recognition accuracy of 94.46%. This outperforms four widely adopted feature selection algorithms, including Relief-F, mRMR, CFS, and ILFS. Moreover, we have established a universal model for the position of gesture measurement points and verified it with an additional five participants, resulting in an average recognition accuracy of 96.3%. This study provides a sound basis for identifying the optimal and minimum number and location of channels on the forearm and designing specialized arm rings with unique shapes.

Automatic modulation recognition (AMR) is an essential topic of cognitive radio, which is of great significance for the analysis of wireless signals and is one of the current research hotspots. Traditional AMR approaches predominantly utilize raw in-phase/quadrature symbols (I/Q), amplitude/phase (A/P), or pre-processed data (e.g., high-order cumulates, spectrum images, or constellation diagrams) as inputs for the recognition model. However, it is difficult to achieve superior performance with only a single type of data as input. This paper proposes a novel multi-channel hybrid learning framework that integrates convolutional layers, Long Short-Term Memory (LSTM) layers, fully connected layers and classification layers. The model is built for modeling spatial-temporal correlations from four signal cues (including I/Q signals, A/P signals, I, and Q signals), which aims to explore various differences and leverage the complements from multiple data-form. Two functions employed during the data conversion process further enhance the non-linear representational capacity of the model, thereby boosting the recognition accuracy of the model. Experimental results demonstrate that the proposed framework effectively addresses the classification challenges of QAM16 and QAM64. For the RAML2016A dataset, our model achieves an impressive recognition accuracy of 95% at an SNR of 0 dB. Extensive experiments indicate that the proposed framework outperforms other current networks in terms of recognition accuracy.

We developed an optical neural network (ONN) for efficient processing and recognition of 2-dimensional (2D) images, employing a conventional liquid crystal display panel as optical neurons and synapses. This configuration allowed for optical signal outputs proportional to matrix-vector multiplication for 2D image inputs. However, our experimental results revealed a 26.6% decrease in the optical classification accuracy, despite utilizing digitally pre-trained parameters with 100% accuracy for 500 handwritten digits. This decline can be attributed to system imperfections associated with non-ideal functions of optical components and optical alignment. Rather than pursuing an elusive, imperfection-free ONN or attempting to calibrate these defects individually, we addressed these challenges by introducing a self-correction mechanism that utilizes a machine learning algorithm. This approach effectively restored the recognition accuracy and minimized loss of our ONN to levels comparable to the digitally pre-trained model. This study underscores the potential of constructing defect-tolerant hardware in ONNs through the application of machine learning techniques.

Coal gangue recognition of top coal caving is one of the important links in the process of intelligent coal mine construction. However, the recognition accuracy of this technology in practical application is still challenging, because the recognition model is not perfect in the following aspects: 1) the filtering scale is not suitable for signal noise reduction;, 2) the selected features have no obvious difference in vibration signals of coal and gangue; and 3) the overall relevance of the model input data to the target classification is insufficient. The purpose of this paper is to establish a coal gangue recognition model with effective filtering, feature extraction and classification capabilities, which can adaptively carry out purposive feature extraction while retaining relevant information to improve the recognition accuracy. Firstly, hierarchical filtering (HF) method was proposed. Secondly, an effective information correlation fusion based coal gangue recognition model (EICF-coal gangue recognition model) was established by coupling wrapper feature selection method and recognition algorithm. Then, 2223 groups of vibration impact tests were carried out on the coal gangue mixture with gangue content of 0 to 50%, and two kinds of coal gangue recognition sample sets of “caving category” and “shutdown category” were established. Finally, coal gangue recognition experiments were carried out on 9 hierarchical filter sample sets by coupling wrapper and 5 recognition algorithms. Under the combined effect of the HF method, wrapper feature selection method and Stacking, coal gangue recognition accuracy reaches 99%. This paper demonstrates the effectiveness of the EICF-coal gangue recognition model.

The purpose is to minimize subjective errors in the manual analysis of volleyball game videos and improve the traditional Human Behavior Recognition (HBR) algorithms’ excessive calculation, high hardware requirements, and poor long-stream video modeling ability. Firstly, this paper expounds on the relevant theories. Secondly, a fusion Convolutional Neural Network (CNN)-based HBR model is implemented. It combines the two-stream CNN (TSCNN), Three-Dimensional (3D) CNN, and Long Short-Term Memory (LSTM) and gives full play to the LSTM’s long-term Dynamic Information Extraction (DIE) ability. Finally, the public dataset is selected to verify the model’s volleyball-game-video-oriented HBR performance. Here are the experimental results. (1) The optimal key parameters of the proposed fusion-CNN-based HBR model are determined as follows: the number of video segments is three, the average method is used for feature fusion, and then the HBR accuracy is the highest when the fusion ratio of spatial feature map and temporal feature map is 4:6, and the learning rate is 0.0014. (2) The average recognition accuracy of the proposed fusion-CNN-based HBR model on three different datasets is 4%, 2.7%, and 3% higher than other popular networks, respectively. The improvement effect of the model is remarkable, and it is suitable for studying Human Behavior Analysis (HBA) in volleyball match videos. Finally, the proposed HBR model can provide more accurate results for volleyball videos’ HBR, which is significant in promoting the rapid development of volleyball sports. The proposed model can classify and label videos and understand and describe video behaviors to simplify video processing procedures and save social resources.

This paper reviewed the sign language research in the vision-based hand gesture recognition system from 2014 to 2020. Its objective is to identify the progress and what needs more attention. We have extracted a total of 98 articles from well-known online databases using selected keywords. The review shows that the vision-based hand gesture recognition research is an active field of research, with many studies conducted, resulting in dozens of articles published annually in journals and conference proceedings. Most of the articles focus on three critical aspects of the vision-based hand gesture recognition system, namely: data acquisition, data environment, and hand gesture representation. We have also reviewed the performance of the vision-based hand gesture recognition system in terms of recognition accuracy. For the signer dependent, the recognition accuracy ranges from 69% to 98%, with an average of 88.8% among the selected studies. On the other hand, the signer independent’s recognition accuracy reported in the selected studies ranges from 48% to 97%, with an average recognition accuracy of 78.2%. The lack in the progress of continuous gesture recognition could indicate that more work is needed towards a practical vision-based gesture recognition system.

Deep-learning-based modulation recognition methods can extract the features of signals automatically with the usage of the deep neural network (DNN). However, the background noises might lower the recognition accuracy and induce longer convergence time. In order to improve the recognition accuracy and to reduce the computational complexity, in this article, we propose to construct the dataset based on spectral correlation function (SCF), which has the special property of relatively being insensitive to background noises. Moreover, we propose to add a convolutional neural network denoising module to combat the noises for better feature extraction performances. Furthermore, in order to reduce the dataset size, we propose to use the two-dimensional (2-D) cross-section SCF profiles for the DNN. Then, we propose a DNN architecture based on the back-propagation neural network. Thanks to the noise resistant SCF feature extractions and the intelligent denoising as well as DNN-based classification design, the recognition accuracy can be improved. Simulation results show that the proposed method can effectively classify modulation schemes at the signal-to-noise ratio (SNR) as low as $-$10 dB. Moreover, the proposed intelligent system achieves higher recognition accuracy than counterpart schemes in low-SNR regions. [ABSTRACT FROM AUTHOR]

Despite the great proven advantages of imaging techniques in percutaneous coronary interventions (PCIs), the recent rapid increase in the number of PCI procedures exposes both surgical experts and patients to more radiation and orthopedic hazards in the intervention room. While patients are minimally exposed, many appearance of interventionists subject them to frequent exposure of the operational hazards. Despite promoting the use of robot-assisted intravascular PCI in the Cath-Labs, cognitive and technical skills of interventionists' are yet to be explored towards reducing procedure time and minimal exposure of surgeon to operational hazards. In this study, a random forest classification framework is developed for proper identification of technical manipulation skills of surgeons along with underlying motion patterns of the flexible intravascular tools (viz. guidewire and catheter) during PCI catheterization. For this purpose, analysis of interventionists' muscular activities and related hand motion were decoded from physiological signals recorded with sensors. Surface electromyography, electromagnetic, and tactile force signals were acquired concurrently from seven interventionists during specified guidewire movement viz. pull, push, rotate-pull (clockwise rotation with pull), rotate-push (counterclockwise rotation with push), CR (clockwise rotation), and CCR (counterclockwise rotation). While relations were established between force and muscle activities in the surgeon groups using Spearman's Rank-Order statistical method, Wilcoxon test and Kruskal-Wallis one-way ANOVA test were employed to identify intra-group and inter-group differences. From the experimental results obtained, guidewire delivery patterns exhibit stable characteristics with overall classification accuracies of 94.11% based on nineteen features subset from muscle activities and hand motion, followed by 88.01% based on twelve features subset from muscle activities, 71.97% based on seven features subset from hand motion, 84.56% based on ten features subset from part muscle activities and hand motion. Thus, this study shows existence of significant correlation (p

Tomato leaf disease seriously affects the yield of tomato. It is extremely vital for agricultural economy to identify agricultural diseases. The traditional data augmentation methods, such as rotation, flip and translation, are severely limited, which cannot achieve good generalization results. To improve the recognition accuracy of tomato leaf diseases, a new method of data augmentation by generative adversarial networks (GANs) is proposed for leaf disease recognition in this work. Generated images augmented by deep convolutional generative adversarial networks (DCGAN) and original images as the input of GoogLeNet, this model can achieve a top-1 average identification accuracy of 94.33%. By adjusting the hyper-parameters, modifying the architecture of the convolutional neural networks, and selecting different generative adversarial networks, an improved model for training and testing 5 classes of tomato leaf images was obtained. Meanwhile, images generated by DCGAN not only enlarge the size of the data set, but also have the characteristics of diversity, which makes the model have a good generalization effect. We have also visually confirmed that the images generated by DCGAN have much better quality and are more convincing through the t-Distributed Stochastic Neighbor Embedding (t-SNE) and Visual Turing Test. Experiments with tomato leaf disease identification show that DCGAN can generate data that approximate to real images, which can be used to (1) provide a larger data set for the training of large neural networks, and improve the performance of the recognition model through highly discriminating image generation technology; (2) reduce the cost of data collection; (3) enhance the diversity of data and the generalization ability of the recognition models.

With the rapid development of neuroinformatics and related intelligent algorithms, the research of recognition and classification based on EEG signals is becoming more and more important and valuable. With the progress of science and technology, the related research of EEG signal recognition and processing has been gradually applied to rehabilitation medicine, intelligent information processing, and other cross-cutting fields. As one of the most important research directions in the field of the brain-computer interface, motor imagery EEG has a wide range of applications. At the same time, it shows a good application effect in the process of application practice. At present, the main EEG recognition and analysis algorithms always have some problems and defects in data processing, such as low signal-to-noise ratio, unclean noise filtering, and high data dimension. In this paper, based on the improved BP neural network algorithm, weight splitting technology is added to the traditional BP neural network algorithm. In order to solve the filtering problem, this paper uses the non-linear mapping function of the traditional BP neural network, and intelligently trains the small weight particles by combining the particle swarm filter algorithm, so as to improve the filtering performance of the whole BP algorithm. Based on the above two algorithms, the problem of low signal-to-noise ratio (SNR) and unclean filtering in EEG data processing caused by fast weight degradation in traditional BP algorithm can be solved. Finally, according to the actual data of brain-computer interface, this paper compares the improved BP neural network algorithm with the traditional BP neural network algorithm in recognition and analysis of motor imagery EEG signals. The experiment shows that the proposed algorithm has obvious advantages in recognition accuracy and analysis effect.

Chronic low back pain (CLBP) is a common musculoskeletal disorder and a major source of disability in adults. The assessment of lumbar muscle functioning has proven as an appropriate approach for early identification of CLBP when significant pathological signs and symptoms are absent. Thus, earlier therapy or rehabilitation can be administered to prevent further deterioration, such as spinal stenosis or disk herniation. In this paper, surface electromyography (sEMG) signal analysis was explored for the recognition of low back pain in subjects with non-specific symptoms; 88 CLBP subjects and a control group of 83 subjects were recruited for sEMG data acquisition. Subjects were asked to perform four specific movements, namely forward bending, backward bending, right lateral flexion, and left lateral flexion. While performing each movement, sEMG signals from three pairs of lumbar muscles were captured, and 31 features from both the time and frequency domains were extracted from the signal. Finally, the main feature group and four subsets, derived from it, were explored. The suggested method allowed to achieve CLBP recognition accuracy of 98.04% based on subset C for forward bending, followed by 96.15% based on subset E for right lateral flexion, 93.33% based on subset E for left lateral flexion, and 91.30% based on subset B for backward bending. A combination of support vector machine classifiers and optimal feature selection allowed for improved classification performance. The main aim of this paper is to recognize CLBP in subjects with non-specific pathology during the four types of movement. The major steps carried out to achieve this are pre-processing, feature selection, and classification of the sEMG signals acquired from 171 subjects. Results suggest CLBP recognition based on sEMG as a promising alternative to the conventional methods. Therefore, this paper could inspire the design of appropriate programs that can ensure effective rehabilitation of CLBP patients.

Developing an intrusion event identification scheme, which concurrently possesses high recognition accuracy and high efficiency, has been an intractable task in optical fiber perimeter security systems. To achieve high recognition accuracy, we apply the all-phase filter (APF) bank in frequency division and choose the envelope fluctuation parameter to describe the waveform feature of APFs’ outputs. To achieve high recognition efficiency, we introduce the random forest classifier to recognize intrusion types, which not only alleviates the negative effect arising from occasionality or randomness of intrusions, but also bypasses tedious computation of existing classifiers applied to optical fiber dual Mach-Zehnder Interferometry based perimeter security system. Experimental results demonstrate that the proposed system can distinguish 6 typical patterns (kicking the fence, cutting the fence, waggling the fence, knocking the fence, climbing the fence, and no intrusion) with the average recognition rate of 96.92%. Moreover, the consumed training time is reduced to about 40% of the Support Vector Machine. Therefore, the proposed scheme has vast potentials in actual applications. [ABSTRACT FROM AUTHOR]

This paper presents three different gesture recognition models which are capable of recognizing seven hand gestures, i.e., up, down, left, right, tick, circle, and cross, based on the input signals from MEMS 3-axes accelerometers. The accelerations of a hand in motion in three perpendicular directions are detected by three accelerometers respectively and transmitted to a PC via Bluetooth wireless protocol. An automatic gesture segmentation algorithm is developed to identify individual gestures in a sequence. To compress data and to minimize the influence of variations resulted from gestures made by different users, a basic feature based on sign sequence of gesture acceleration is extracted. This method reduces hundreds of data values of a single gesture to a gesture code of 8 numbers. Finally, the gesture is recognized by comparing the gesture code with the stored templates. Results based on 72 experiments, each containing a sequence of hand gestures (totaling 628 gestures), show that the best of the three models discussed in this paper achieves an overall recognition accuracy of 95.6%, with the correct recognition accuracy of each gesture ranging from 91% to 100%. We conclude that a recognition algorithm based on sign sequence and template matching as presented in this paper can be used for nonspecific-users hand-gesture recognition without the time consuming user-training process prior to gesture recognition. [ABSTRACT FROM PUBLISHER]

Speech recognition and speaker recognition by machine are crucial ingredients for many important applications such as natural and flexible human-machine interfaces. Most developments in speech-based automatic recognition have relied on acoustic speech as the sole input signal, disregarding its visual counterpart. However, recognition based on acoustic speech alone can be afflicted with deficiencies that preclude its use in many real-world applications, particularly under adverse conditions. The combination of auditory and visual modalities promises higher recognition accuracy and robustness than can be obtained with a single modality. Multimodal recognition is therefore acknowledged as a vital component of the next generation of spoken language systems. The paper reviews the components of bimodal recognizers, discusses the accuracy of bimodal recognition, and highlights some outstanding research issues as well as possible application domains [ABSTRACT FROM PUBLISHER]