Search

Modern network systems (unmanned aerial vehicles groups, social networks, network production chains, transport and logistics networks, communication networks, cryptocurrency networks) are distinguished by their multi-element nature and the dynamics of connections between its elements. A number of discrete problems on the construction of optimal substructures of network systems described in the form of various classes of graphs are NP-complete problems. In this case, the variability and dynamism of the structures of network systems leads to an "additional" complication of the search for solutions to discrete optimization problems. At the same time, for some subclasses of dynamical graphs, which are used to model the structures of network systems, conditions for the solvability of a number of NP-complete problems can be distinguished. This subclass of dynamic graphs includes pre-fractal graphs. The article investigates NP-complete problems on pre-fractal graphs: a Hamiltonian cycle, a skeleton with the maximum number of pendant vertices, a monochromatic triangle, a clique, an independent set. The conditions under which for some problems it is possible to obtain an answer about the existence and to construct polynomial (when fixing the number of seed vertices) algorithms for finding solutions are identified.

The purpose of action prediction is to recognize an action before it is completed to reduce recognition latency. Because action prediction has lower latency than action recognition, it can be applied to a variety of surveillance scenarios and responds faster. However, action prediction is more difficult because it cannot obtain the complete action execution. In this paper, we study the action prediction which is based on skeleton data and propose a new network called adaptive graph convolutional network with adversarial learning (AGCN-AL) for it. The AGCN-AL uses adversarial learning to make the features of the partial sequences as similar as possible to the features of the full sequences to learn the potential global information in the partial sequences. Besides, partial sequences with different numbers of frames contain different amounts of information. We introduce temporal-dependent loss functions to prevent the network from paying too much attention to partial sequences whose observation ratios are small, and ignoring partial sequences whose observation ratios are large. Moreover, the AGCN-AL is combined with the local AGCN into a two-stream network to enhance the prediction, proving that the local information and the potential global information in partial sequences are complementary. We evaluate the proposed approach on two datasets and show excellent performance.

With the prevalence of accessible multi-modal sensors and the maturity of pose estimation algorithms, skeleton-based action recognition has gradually become the mainstream of human action recognition (HAR). The key issue is to mine the correlations and dependencies between different joints and bones. In this paper, we propose a cross view learning approach. First, the static and dynamic representations of skeletons, from two different views (joints and bones), are calculated and aggregated respectively. Then, the integrated representations of these two views are used as parallel inputs to the cross view learning model, which mainly includes two blocks, namely a multi-scale learning block and a multi-view fusion block. The former is used to excavate the intra-view’s discriminative and comprehensive features, and the latter is utilized to capture the complementary representations of the inter-view. Finally, the fused representations are input to the classifier for action recognition. It has been experimentally proven that our proposed approach outperforms several state-of-the-art baseline methods and achieves a very competitive performance.

Skeleton-based human action recognition has received extensive attention due to its efficiency and robustness to complex backgrounds. Though the human skeleton can accurately capture the dynamics of human poses, it fails to recognize human actions induced by the interaction between human and objects, making it is of great importance to further explore the interaction between the human and objects for human action recognition. In this paper, we devise the multi-stream interaction networks (MSIN), to simultaneously explore the dynamics of human skeleton, objects, and the interaction between human and objects. Specifically, apart from the traditional human skeleton stream, 1) the second stream explores the dynamics of object appearance from the objects surrounding the human body joints; and 2) the third stream captures the dynamics of object position in regard to the distance between the object and different human body joints. Experimental results on three popular skeleton-based human action recognition datasets, NTU RGB+D, NTU RGB+D 120, and SYSU, demonstrate the effectiveness of the proposed method, especially for recognizing the human actions with human-object interactions.

In complicated product industry such as aircraft manufacturing, an assembly model contains abundant engineering information for use in design, manufacture, and maintenance. Assembly retrieval can be used to find relevant models for knowledge reuse. However, an assembly with rotatable joints may have many poses, which brings difficulty to assembly retrieval, since there is no pose principle for assembly design. Therefore, focused on rotatable joints in assembly, a skeleton-based descriptor for pose-free assembly retrieval is proposed. The centroid points of part surfaces and contact faces in an assembly are extracted to construct a spatial-contact skeleton. The skeleton-based distance is proposed to measure the distance between two surface points, which is invariant to the rotatable joints. The distribution of skeleton distances between two parts is used to describe the pair. Considering a part paired with all other parts in the assembly, the set of part pairs is used to represent a part, and the modified Hausdorff distance is used to measure the dissimilarity between parts for assembly retrieval. Experiments are conducted to compare the accuracy of the proposed descriptor to holistic and structureless descriptors. The proposed method is shown to retrieve assemblies with similar parts and structures regardless of their rotatable joints.

Skeleton representation has attracted a great deal of attention recently as an extremely robust feature for human action recognition. However, its non-Euclidean structural characteristics raise new challenges for conventional solutions. Recent studies have shown that there is a native superiority in modeling spatiotemporal skeleton information with a Graph Convolutional Network (GCN). Nevertheless, the skeleton graph modeling normally focuses on the physical adjacency of the elements of the human skeleton sequence, which contrasts with the requirement to provide a perceptually meaningful representation. To address this problem, in this paper, we propose a perceptually-enriched graph learning method by introducing innovative features to spatial and temporal skeleton graph modeling. For the spatial information modeling, we incorporate a Local-Global Graph Convolutional Network (LG-GCN) that builds a multifaceted spatial perceptual representation. This helps to overcome the limitations caused by over-reliance on the spatial adjacency relationships in the skeleton. For temporal modeling, we present a Region-Aware Graph Convolutional Network (RA-GCN), which directly embeds the regional relationships conveyed by a skeleton sequence into a temporal graph model. This innovation mitigates the deficiency of the original skeleton graph models. In addition, we strengthened the ability of the proposed channel modeling methods to extract multi-scale representations. These innovations result in a lightweight graph convolutional model, referred to as Graph2Net, that simultaneously extends the spatial and temporal perceptual fields, and thus enhances the capacity of the graph model to represent skeleton sequences. We conduct extensive experiments on NTU-RGB+D 60&120, Northwestern-UCLA, and Kinetics-400 datasets to show that our results surpass the performance of several mainstream methods while limiting the model complexity and computational overhead.

Accurate lower-limb pose estimation is a prerequisite of skeleton based pathological gait analysis. To achieve this goal in free-living environments for long-term monitoring, single depth sensor has been proposed in research. However, the depth map acquired from a single viewpoint encodes only partial geometric information of the lower limbs and exhibits large variations across different viewpoints. Existing off-the-shelf three-dimensional (3D) pose tracking algorithms and public datasets for depth based human pose estimation are mainly targeted at activity recognition applications. They are relatively insensitive to skeleton estimation accuracy, especially at the foot segments. Furthermore, acquiring ground truth skeleton data for detailed biomechanics analysis also requires considerable effort. To address these issues, we propose a novel cross-domain self-supervised complete geometric representation learning framework, with knowledge transfer from the unlabelled synthetic point clouds of full lower-limb surfaces. The proposed method can significantly reduce the number of ground truth skeletons (with only 1\%) in the training phase, meanwhile ensuring accurate and precise pose estimation and capturing discriminative features across different pathological gait patterns compared to other methods.

Skeleton‐based action recognition algorithms have been widely applied to human action recognition. Graph convolutional networks (GCNs) generalize convolutional neural networks (CNNs) to non‐Euclidean graphs and achieve significant performance in skeleton‐based action recognition. However, existing GCN‐based models have several issues, such as the topology of the graph is defined based on the natural skeleton of the human body, which is fixed during training, and it may not be applied to different layers of the GCN model and diverse datasets. Besides, the higher‐order information of the joint data, for example, skeleton and dynamic information is not fully utilised. This work proposes a novel multi‐stream adaptive spatial‐temporal attention GCN model that overcomes the aforementioned issues. The method designs a learnable topology graph to adaptively adjust the connection relationship and strength, which is updated with training along with other network parameters. Simultaneously, the adaptive connection parameters are utilised to optimise the connection of the natural skeleton graph and the adaptive topology graph. The spatial‐temporal attention module is embedded in each graph convolution layer to ensure that the network focuses on the more critical joints and frames. A multi‐stream framework is built to integrate multiple inputs, which further improves the performance of the network. The final network achieves state‐of‐the‐art performance on both the NTU‐RGBD and Kinetics‐Skeleton action recognition datasets. The simulation results prove that the proposed method reveals better results than existing methods in all perspectives and that shows the superiority of the proposed method.

The general silhouette‐based gait recognition methods usually rely on binary human silhouette, which is easily affected by external factors, making it unsuitable for situations while wearing heavy clothes or carrying objects, etc. In this study, a new skeleton‐based gait recognition model is proposed. The model first extracts the spatial and temporal features of gait using the space and time relationship between body joints, and second, it eliminates redundant features by decomposing the feature map, to achieve a better recognition accuracy in the presence of external factors. Through abundant experiments on two common datasets, CASIA‐B and OUMVLP‐Pose, the proposed model has been proved to have higher recognition accuracy and remarkable robustness.

Skeleton creation is an important phase in the character animation pipeline. However, handcrafting skeleton takes extensive labor time and domain knowledge. Automatic skeletonization provides a solution. However, most of the current approaches are far from real-time and lack the flexibility to control the skeleton complexity. In this paper, we present an efficient skeletonization method, which can be seamlessly integrated into the sketch-based modeling process in real-time. The method contains three steps: local sub-skeleton extraction; sub-skeleton connection; and global skeleton refinement. Firstly, the local skeleton is extracted from the processed polygon stroke and forms a subpart along with the sub-mesh. Then, local sub-skeletons are connected according to the intersecting relationships and the modeling sequence of subparts. Lastly, a global refinement method is proposed to give users coarse-to-fine control on the connected skeleton. We demonstrate the effectiveness of our method on a variety of examples created by both novices and professionals., Shape Modeling International 2021

Gait recognition aims to identify a human through a walking sequence. It is a challenging task in computer vision since monocular camera loses most of the 3D information. Previous works described gait features with the contours of shape or the global geometrical characters of skeleton. So little work is researched on the local patterns of gait skeleton. In this paper, to resist the dress changes and speed changes, a Local Graphical Skeleton Descriptor (LGSD) is proposed to describe both the inner and intra local graphical patterns of a human gait skeleton. The gait features from the same or different identities are paired up and a Pairwise Similarity Network (PSN) is proposed to maximize the similarity of True matched pairs and minimize the similarity of false matched pairs. The contributions of our method are: 1) LGSD is proposed to describe human gait by computing four novel local geometrical patterns of skeleton sequences, which makes use of the intuitive cognition of gait based on the prior knowledge of mankind. 2) PSN is implemented by a two-stream CNN structure to encode the features and train the gait model, which fused two popular gait recognition strategies. 3) The robustness of our method to dress changes and speed changes is proved on the public datasets. We have also achieved some state-of-the-art results on these datasets. The proposed method is examined on three public gait datasets which have RGB or infrared frames for evaluation: the CASIA-B dataset, the NLPR gait database, and the CASIA-C dataset. The performers in these datasets are walking under different views, speeds or dresses. The results are further compared with previous approaches to confirm the effectiveness and the advantages of our method.

One of key challenges of skeleton-based action recognition (SAR) tasks is the complex nature of human motion patterns. Variations such as performers and viewpoints may impose negative effects to the action recognition accuracy. In this work, we propose the Multi-Localized Sensitive Autoencoder-Attention-LSTM (Multi-LiSAAL) for SAR. The Localized Stochastic Sensitive Autoencoder (LiSSA) encodes both spatial and temporal information, and extracts meaningful features from different parts (four limbs and a trunk) from the skeleton. The LiSSA is trained by minimizing the localized generalization error to enhance the robustness of autoencoders via reducing its sensitivity with respect to small variations in inputs. We apply an attention mechanism to assign different weights to different skeleton parts and focus more on informative sections. Then, a backbone classifier network takes weighted features as inputs to differentiates actions. Experimental results on five public benchmarking datasets show that the Multi-LiSAAL outperforms state-of-the-art methods.

Skeleton-based action recognition has attracted significant attention and obtained widespread applications due to the robustness of 3D skeleton data. One of the key challenges is how to extract discriminative and robust spatio-temporal features from sparse skeleton data to describe actions and improve recognition accuracy. To address this issue, this paper combines convolutions with attention mechanisms and proposes a deep network for skeleton-based action recognition, termed as local-and-global attention network (LAGA-Net). First, we encode skeleton sequences into joint feature evolution maps to compactly describe the spatial and temporal characteristics of skeleton sequences. Then, a motion guided channel attention module (MGCAM) is proposed to model the interdependencies between feature channels by calculating temporal frame-level motion and enhance motion-salient features in a channel-wise way. Further, a spatio-temporal attention module (STAM) is proposed to model spatio-temporal context-aware collaboration at sequence level and extract spatio-temporal attention features that involve long-range dependencies. Together, MGCAM and STAM are combined to form LAGA-Net, which extracts discriminative features integrating both local and global representations of skeleton sequences. Moreover, a two-stream architecture is proposed to learn complementary features from joint and bone aspects. We conduct extensive experiments to verify the effectiveness and superiority of our proposed method over state-of-the-art approaches on several benchmarks (e.g., NTU RGB+D, Northwestern-UCLA, UTD-MHAD and NTU RGB+D 120).

Recent studies have made great progress on skeleton-based action recognition. However, most of them are developed with relatively clean skeletons without the presence of intensive noise. We argue that the models learned from relatively clean data are not well generalizable to handle noisy skeletons commonly appeared in the real world. In this paper, we address the challenge of recognizing human actions from noisy skeletons, which is seldom explored by previous methods. Beyond exploring the new problem, we further take a new perspective to address it, \textit{i.e.}, noise adaptation, which gets rid of explicit skeleton noise modeling and reliance on skeleton ground truths. Specifically, we develop regression-based and generation-based adaptation models according to whether pairs of noisy skeletons are available. The regression-based model aims to learn noise-suppressed intrinsic feature representations by mapping pairs of noisy skeletons into a noise-robust space. When only unpaired skeletons are accessible, the generation-based model aims to adapt the features from noisy skeletons to a low-noise space by adversarial learning. To verify our proposed model and facilitate research on noisy skeletons, we collect a new dataset Noisy Skeleton Dataset (NSD), the skeletons of which are with much noise and more similar to daily-life data than previous datasets. Extensive experiments are conducted on the NSD, VV-RGBD and N-UCLA datasets, and results consistently show the outstanding performance of our proposed model.

In this work, we propose a dual-stream structured graph convolution network ( DS-SGCN ) to solve the skeleton-based action recognition problem. The spatio-temporal coordinates and appearance contexts of the skeletal joints are jointly integrated into the graph convolution learning process on both the video and skeleton modalities. To effectively represent the skeletal graph of discrete joints, we create a structured graph convolution module specifically designed to encode partitioned body parts along with their dynamic interactions in the spatio-temporal sequence. In more detail, we build a set of structured intra-part graphs, each of which can be adopted to represent a distinctive body part (e.g., left arm, right leg, head). The inter-part graph is then constructed to model the dynamic interactions across different body parts; here each node corresponds to an intra-part graph built above, while an edge between two nodes is used to express these internal relationships of human movement. We implement the graph convolution learning on both intra- and inter-part graphs in order to obtain the inherent characteristics and dynamic interactions, respectively, of human action. After integrating the intra- and inter-levels of spatial context/coordinate cues, a convolution filtering process is conducted on time slices to capture these temporal dynamics of human motion. Finally, we fuse two streams of graph convolution responses in order to predict the category information of human action in an end-to-end fashion. Comprehensive experiments on five single/multi-modal benchmark datasets (including NTU RGB+D 60, NTU RGB+D 120, MSR-Daily 3D, N-UCLA, and HDM05) demonstrate that the proposed DS-SGCN framework achieves encouraging performance on the skeleton-based action recognition task.

In human-human interaction, body language is one of the most important emotional expressions. However, each emotion category contains abundant emotional body gestures, and basic emotions used in most researches are difficult to describe complex and diverse emotional states. It is costly to collect sufficient samples of all emotional expressions, and new emotions or new body gestures that are not included in the training set may appear during testing. To address the above problems, we design a novel mechanism that treats each emotion category as a collection of multiple body gesture categories to make better use of gesture information for emotion recognition. A Generalized Zero-Shot Learning (GZSL) framework is introduced to recognize both seen and unseen body gesture categories with the help of semantic information, and emotion predictions are further provided based on the relationship between gestures and emotions. This framework consists of two branches. The first branch is a Hierarchical Prototype Network (HPN) which learns the prototypes of body gestures and uses them to calculate the emotion attentive prototypes. This branch aims to obtain predictions on samples of the seen gesture categories. The second branch is a Semantic Auto-Encoder (SAE) which utilizes semantic representations to predict samples of unseen gesture categories. Thresholds are further trained to determine which branch result will be used during testing, and the emotion labels are finally obtained from these results. Comprehensive experiments are conducted on an emotion recognition dataset which contains skeleton data of multiple body gestures, and the performance of our framework is superior to both the traditional emotion classifier and state-of-the-art zero-shot learning methods.

A novel method for skeleton-based action recognition by fusing multi-level spatial features and multi-level temporal features is proposed in this article. Recently, Graph Convolutional Network (GCN) for skeleton-based action recognition has attracted the eyes of many researchers and has a great performance in the field of action recognition. But most of them focus on changing architecture of single-stream network and only use simple methods like average fusion to fuse different forms of skeleton data. In this article, we shift the focus to the problem that insufficient interactions between the different forms of features for that networks are unable to fully capture efficient information from skeleton data. To tackle this problem, we propose a multi-stream network called Symmetrical Enhanced Fusion Network (SEFN). The network is composed of a spatial stream, a temporal stream and a fusion stream. The spatial stream extracts spatial features from skeleton data by GCN. The temporal stream is able to extract temporal features from skeleton data with the help of the embedded Motion Sequence Calculation Algorithm. The fusion stream provides an early fusion method and extra fusion information for the whole network. It gathers multi-level features from two feature extractions and fuses them with the Multi-perspective Attention Fusion Module (MPAFM) we propose. The MPAFM enables different forms of data to enhance each other and can strengthen feature extractions. In the final, we generalize the skeleton data from joint data to bone data and evaluate our network in three large-scale benchmarks: NTU-RGBD, NTU-RGBD 120 and Kinetics-Skeleton. Experiment results demonstrate that our method achieves competitive performance.

Human activity recognition (HAR) has become a hot research topic, due to its wide application prospect. Fusion-based methods can be used to complement single sensing modality methods. This paper presents the simultaneous utilization of skeleton data and inertial signals—which are captured at the same time using a Kinect depth camera and ten wearable inertial sensors —within a fusion framework, in order to achieve more robust human action recognition, compared to situations where each sensing modality is used individually. Skeleton data captured by the Kinect depth camera are transformed into a weighted front-view skeleton motion map (WF-SMM), a weighted multi-view skeleton motion map (WM-SMM), and a 3D weighted skeleton motion map (3DW-SMM), which are then fed as inputs into a convolutional neural network. Meanwhile, the inertial data are transformed into 2D inertial images, then fed into a 2D dilated convolutional neural network. Two types of fusion are considered: decision-level fusion and feature-level fusion. Experiments were conducted using the publicly available Changzhou University Multimodal Human Action Data set (CZU-MHAD), in which simultaneous skeleton sequence and inertial signals were captured for a total of 22 actions. The results obtained indicate that both the decision- and feature-level fusion approaches generate higher recognition accuracies, compared to the approaches where each sensing modality is used individually. The highest accuracy (of 98.90%) was obtained with the decision-level fusion approach using 3DW-SMM. In addition, some experiments are conducted on the continuous action streams generated based on the CZU-MHAD with different score threshold. And the highest f1 score 82.05 % is obtained with the threshold of 0.4.

Methods based on Graph Convolutional Networks (GCN) for skeleton-based action recognition have achieved great success due to their ability to exploit graph structural information from skeleton data. Recently, the bone information has attracted considerable attention as an effective modality which complements the more conventional joint information for action recognition. However, most existing GCN-based methods extract the joint and bone features with two separate GCN networks, ignoring the dependencies between them. In this paper, a novel GCN model is proposed to exploit the information across joints, bones and their relationship collaboratively on a single undirected graph instead of two separate networks. We call the proposed model Vertex-Edge Graph Convolutional Network (VE-GCN) since it conducts the graph convolution operation on the sampling area containing the designated vertexes from joints and edges from bones, respectively. In addition to conducting the Vertex-Edge graph convolution based on the physical connections of the skeleton, we further apply the Vertex-Edge graph convolution to the non-physical joint-joint and joint-bone connections to capture the distal dependencies, and then the convolution results on the non-physical connections are incorporated into the VE-GCN. Moreover, the Conditional Random Field (CRF) is adopted as the loss function to achieve the task of action recognition. Experimental results on four challenging benchmarks (NTU RGB+D, NTU RGB+D 120, N-UCLA, SYSU) show that the proposed model achieves state-of-the-art performance.

Person re-identification (Re-id) is one of the important tools of video surveillance systems, which aims to recognize an individual across the multiple disjoint sensors of a camera network. Despite the recent advances on RGB camera-based person re-identification methods under normal lighting conditions, Re-id researchers fail to take advantages of modern RGB-D sensor-based additional information (e.g. depth and skeleton information). When traditional RGB-based cameras fail to capture the video under poor illumination conditions, RGB-D sensor-based additional information can be advantageous to tackle these constraints. This work takes depth images and skeleton joint points as additional information along with RGB appearance cues and proposes a person re-identification method. We combine 4-channel RGB-D image features with skeleton information using score-level fusion strategy in dissimilarity space to increase re-identification accuracy. Moreover, our propose method overcomes the illumination problem because we use illumination invariant depth image and skeleton information. We carried out rigorous experiments on two publicly available RGBD-ID re-identification datasets and proved the use of combined features of 4-channel RGB-D images and skeleton information boost up the rank 1 recognition accuracy.

In this paper, we proposed a multi-view system for 3D human skeleton tracking based on multi-cue fusion. Multiple Kinect version 2 cameras are applied to build up a low-cost system. Though Kinect cameras can detect 3D skeleton from their depth sensors, some challenges of skeleton extraction still exist, such as left–right confusion and severe self-occlusion. Moreover, human skeleton tracking systems often have difficulty in dealing with lost tracking. These challenges make robust 3D skeleton tracking nontrivial. To address these challenges in a unified framework, we first correct the skeleton's left–right ambiguity by referring to the human joints extracted by OpenPose. Unlike Kinect, and OpenPose extracts target joints by learning-based image analysis to differentiate a person's front side and backside. With help from 2D images, we can correct the left–right skeleton confusion. On the other hand, we find that self-occlusion severely degrades Kinect joint detection owing to incorrect joint depth estimation. To alleviate the problem, we reconstruct a reference 3D skeleton by back-projecting the corresponding 2D OpenPose joints from multiple cameras. The reconstructed joints are less sensitive to occlusion and can be served as 3D anchors for skeleton fusion. Finally, we introduce inter-joint constraints into our probabilistic skeleton tracking framework to trace all joints simultaneously. Unlike conventional methods that treat each joint individually, neighboring joints are utilized to position each other. In this way, when joints are missing due to occlusion, the inter-joint constraints can ensure the skeleton consistency and preserve the length between neighboring joints. In the end, we evaluate our method with five challenging actions by building a real-time demo system. It shows that the system can track skeletons stably without error propagation and vibration. The experimental results also reveal that the average localization error is smaller than that of conventional methods.

Gait features have been widely applied in human identification. The commonly-used representations for gait recognition can be roughly classified into two categories: model-free features and model-based features. However, due to the view variances and clothes changes, model-free features are sensitive to the appearance changes. For model-based features, there is great difficulty in extracting the underlying models from gait sequences. Based on the confidence maps and the part affinity fields produced by a two-branch multi-stage CNN network, a new model-based representation, Skeleton Gait Energy Image (SGEI), has been proposed in this paper. Another contribution is that a hybrid representation has been produced, which uses SGEI to remedy the deficiency of model-free features, Gait Energy Image (GEI) for instance. The experimental performances indicate that our proposed methods are more robust to the cloth changes, and contribute to increasing the robustness of gait recognition in the unconstrained environments with view variances and clothes changes.

We propose a new algorithm for curve skeleton computation which differs from previous algorithms by being based on the notion of local separators. The main benefits of this approach are that it is able to capture relatively fine details and that it works robustly on a range of shape representations. Specifically, our method works on shape representations that can be construed as a spatially embedded graphs. Such representations include meshes, volumetric shapes, and graphs computed from point clouds. We describe a simple pipeline where geometric data is initially converted to a graph, optionally simplified, local separators are computed and selected, and finally a skeleton is constructed. We test our pipeline on polygonal meshes, volumetric shapes, and point clouds. Finally, we compare our results to other methods for skeletonization according to performance and quality., Comment: preprint, 25 pages, 17 figures

Three-dimensional human body curve-skeleton is widely used in pose estimation, skeleton animation and other fields. This paper proposes an improved $$\ell _1$$ median model that can extract three-dimensional human body curve-skeleton. The model includes three-dimensional human body reconstruction from multi-view images, interpolation curve-skeleton extraction, $$\ell _1$$ median skeleton completion, and continuous frame curve-skeleton optimization. Through the completion and optimization processes, the curve-skeleton we extract is smoother and more complete compared with previous methods. We conduct experiments on multi-view human body image dataset collected from light field acquisition system. Both quantitative and qualitative results demonstrate the effectiveness of our model.

Graph convolutional networks (GCN) have received more and more attention in skeleton-based action recognition. Many existing GCN models pay more attention to spatial information and ignore temporal information, but the completion of actions must be accompanied by changes in temporal information. Besides, the channel, spatial, and temporal dimensions often contain redundant information. In this paper, we design a temporal graph convolutional network (FTGCN) module which can concentrate more temporal information and properly balance them for each action. In order to better integrate channel, spatial and temporal information, we propose a unified attention model of the channel, spatial and temporal (CSTA). A basic block containing these two novelties is called FTC-GCN. Extensive experiments on two large-scale datasets, compared with 17 methods on NTU-RGB+D and 8 methods on Kinetics-Skeleton, show that for skeleton-based human action recognition, our method achieves the best performance.

Online action recognition is an important task for human-centered intelligent services. However, it remains a highly challenging problem due to the high varieties and uncertainties of spatial and temporal scales of human actions. In this paper, the following core ideas are proposed to deal with the online action recognition problem. First, we combine spatial and temporal skeleton features to represent human actions, which include not only geometrical features, but also multiscale motion features, such that both spatial and temporal information of the actions are covered. We use an efficient one-dimensional convolutional neural network to fuse spatial and temporal features and train them for action recognition. Second, we propose a group sampling method to combine the previous action frames and current action frames, which are based on the hypothesis that the neighboring frames are largely redundant, and the sampling mechanism ensures that the long-term contextual information is also considered. Third, the skeletons from multiview cameras are fused in a distributed manner, which can improve the human pose accuracy in the case of occlusions. Finally, we propose a Restful style based client-server service architecture to deploy the proposed online action recognition module on the remote server as a public service, such that camera networks for online action recognition can benefit from this architecture due to the limited onboard computational resources. We evaluated our model on the data sets of JHMDB and UT-Kinect, which achieved highly promising accuracy levels of 80.1% and 96.9%, respectively. Our online experiments show that our memory group sampling mechanism is far superior to the traditional sliding window.

The dual stream-based human action recognition model offers the advantage of high recognition accuracy, but the algorithm is less robust in case of lighting changes. The human skeleton has a strong ability to express human behavior and actions; however, the scene information is ignored. Drawing on the idea of the dual-stream model, this paper proposes a human skeleton and scene image-based dual-stream model for human action recognition. The motion features are extracted through the spatio-temporal graph convolution of the human skeleton, and a scene recognition model is proposed based on the sparse frame sampling of video and video-level consensus strategy to process the scene video and gather the visual scene information. The proposed model exploits the advantages of skeleton information in motion expression and the superiority of the image in scene presentation. The scene information and spatio-temporal graph convolution-based human skeleton limbs are fused complementarily to achieve human action recognition. Compared to the conventional optical flow-based dual-stream action recognition method, this model is verified by experimenting under unstable light conditions, and the performance of human action recognition is robust and promising.

One of the most recent challenging tasks in computer vision is Human Activity Recognition (HAR), which aims to analyze and detect the human actions for the benefit of many fields such as video surveillance, behavior analysis and healthcare. Several works in the literature are based on the extraction and analysis of human skeletons in the aim of actions recognition. This paper introduces a new HAR approach based on the extraction of human skeletons from videos. Three features extraction techniques are proposed in this work. They used the extracted skeletons from the videos frames in order to construct a single image that summarizes the activity in that video. The first technique, called dynamic skeleton, is founded on the concept of dynamic images introduced in the literature, while the second one, called skeleton superposition, is based on the superposition of the extracted human skeletons in the same image. The third contribution is called body articulations and it uses only the body joints instead of the whole skeleton in order to recognize the ongoing activity. The obtained images from these three techniques are analyzed and classified using a classification system based on transfer learning principle by fine-tuning three well-known pre-trained CNNs (MobileNet, ResNet-50, VGG16). The designed system is validated and tested on two famous datasets for human activity recognition, which are RGBD-HuDact and KTH datasets. The obtained results are outstanding and proved that the implemented system outperforms the state-of-the-art approaches.

Recently, skeleton-based action recognition has modeled the human skeleton as a graph convolution network (GCN), and has achieved remarkable results. However, most of the methods convolute directly on the whole graph, neglecting that the human skeleton is made up of multiple body parts, which cannot accomplish the task well. We recognize that the physical property of bones (i.e., length and direction) can provide identifiable information which helps effectively to build the multi-level network structure. As the existing methods treat the channel domain and the spatial domain with equal importance, many computing resources are wasted on neglectable features. In our paper, we modify the Convolution Block Attention Module (CBAM) and apply it to the adaptive network. By capturing the implicit weighted information in the channel domain and spatial domain, the network can focus more attention on the key channels and nodes. A new two-stream adaptive-attentional subgraph convolution network (2s-AASGCN) is proposed to extract features in the spatio-temporal domain. We validate 2s-AASGCN on two skeleton datasets, i.e., NTU-RGB+D60 and NTU-RGB+D120. Our model achieves excellent results on these two datasets.

Map generalization is a process of hierarchically reorganizing features whereby the global shape of the original datasets can be transferred in different scales. We propose a stroke and centrality-...

The term melodic template or skeleton refers to a basic melody which is subject to variation during a music performance. In many oral music tradition, these templates are implicitly passed throughout generations without ever being formalized in a score. In this work, we introduce a new geometric optimization problem, the spanning tube problem, to approximate a melodic template for a set of labeled performance transcriptions corresponding to an specific style in oral music traditions. Given a set of $n$ piecewise linear functions, we solve the problem of finding a continuous function, $f^*$, and a minimum value, $\varepsilon^*$, such that, the vertical segment of length $2\varepsilon^*$ centered at $(x,f^*(x))$ intersects at least $p$ functions ($p\leq n$). The method explored here also provide a novel tool for quantitatively assess the amount of melodic variation which occurs across performances.

Process discovery is an important area in the field of process mining. To help advance this area, a process discovery contest (PDC) has been set up, which allows us to compare different approaches. At the moment of writing, there have been three instances of the PDC: in 2016, in 2017, and in 2019. This paper introduces the winning contribution to the PDC 2019, called the Log Skeleton Visualizer. This visualizer uses a novel type of process models called log skeletons. In contrast with many workflow net-based discovery techniques, these log skeletons do not rely on the directly follows relation. As a result, log skeletons offer circumstantial information on the event log at hand rather than only sequential information. Using this visualizer, we were able to classify 898 out of 900 traces correctly for the PDC 2019 and to win this contest.

Single sensing modality is widely adopted for human activity recognition (HAR) for decades and it has made a significant stride. However, it often suffers from challenges such as noises, obstacles, or dropped signals, which might negatively impact on the recognition performance. In this paper, we propose a multi-sensing modality framework for human fine-grained activity recognition and abnormal behaviour detection by combining skeleton and acceleration data at feature level (so-called feature-level fusion). Firstly, deep temporal convolutional networks (TCN), consisting of the dilated causal convolution components, are utilized for feature learning and handling temporal properties. The feature map learnt and represented with convolutional layers in TCN is fed into two fully connected layers for the prediction. Secondly, we conduct an empirical experiment to verify our proposed method. Experimental results have shown that the proposed method could achieve 83% F1-score and surpassed several single modality models as well as early and late fusion methods on the Continuous Multimodal Multi-view Dataset of Human Fall Dataset (CMDFALL), comprised of 20 fine-grained normal and abnormal activities collected from 50 subjects. Moreover, our proposed architecture achieves 96.98% accuracy on the UTD-MHAD dataset, which has 8 subjects and 27 activities. These results indicate the effectiveness of our proposed method for the classification of human fine-grained normal and abnormal activities as well as the potential for HAR-based situated service applications.

In skeleton-based action recognition, graph convolutional networks (GCNs), which model human body skeletons using graphical components such as nodes and connections, have recently achieved remarkable performance. While the current state-of-the-art methods for skeleton-based action recognition usually assume that completely observed skeletons will be provided, it is problematic to realize this assumption in real-world scenarios since the captured skeletons may be incomplete or noisy. In this work, we propose a skeleton-based action recognition method that is robust to noise interference for the given skeleton features. The key insight of our approach is to train a model by maximizing the mutual information between normal and noisy skeletons using predictive coding in the latent space. We conducted comprehensive skeleton-based action recognition experiments with defective skeletons using the NTU-RGB+D and Kinetics-Skeleton datasets. The experimental results demonstrate that when the skeleton samples are noisy, our approach achieves outstanding performances compared with the existing state-of-the-art methods.

With the increasing availability of human motion data captured in the form of 2D or 3D skeleton sequences, more complex motion recordings need to be processed. In this paper, we focus on similarity-based indexing and efficient retrieval of motion episodes — medium-sized skeleton sequences that consist of multiple semantic actions and correspond to some logical motion unit (e.g. a figure skating performance). As a first step toward efficient retrieval, we apply the motion-word technique to transform spatio-temporal skeleton sequences into compact text-like documents. Based on these documents, we introduce a two-phase retrieval scheme that first finds a set of candidate query results and then re-ranks these candidates with more expensive application-specific methods. We further index the motion-word documents using inverted files, which allows us to retrieve the candidate documents in an efficient and scalable manner. We also propose additional query-reduction techniques that accelerate both the retrieval phases by removing semantically irrelevant parts of the motion query. Experimental evaluation is used to analyze the effects of the individual proposed techniques on the retrieval efficiency and effectiveness.

This paper describes the design, construction, and use of an open-source hardware and software tool intended to help Anatomy and Physiology students test their knowledge of muscle actions and joint movements. Orientation sensors are attached to a model skeleton to turn the skeleton into an interactive, physical model for teaching limb movements. A detailed description of the construction of the tool is provided, as well as the configuration and use of companion software.

With the development of 3D hand pose estimation technologies, skeleton-based dynamic hand gesture recognition has attracted widespread attention. In this paper, we propose a novel framework for skeleton-based dynamic hand gesture recognition. In the spatial perception stream (SP-Stream), we design a compact joints encoding method. It can adaptively select compact joints based on the convex hull of the hand skeleton and encode them into a skeleton image for fully extracting spatial features. Besides, we present a global enhancement module (GEM) to enhance key feature maps. In the temporal perception stream (TP-Stream), we propose a motion perception module (MPM) to enhance the notable movement of hand gestures on X / Y / Z coordinate axes. Experimental results show that the proposed framework performs better than the state-of-the-art methods on two benchmark datasets.

This paper presents a sign language recognition system based on color stream and skeleton points. Several approaches have been established to address sign language recognition problems. However, most of the previous approaches still have poor recognition accuracy. The proposed approach uses Kinect sensor based on color stream and skeleton points from the depth stream to improved recognition accuracy. Techniques within this approach use hand trajectories and hand shapes in combating sign recognition challenges. Therefore, for a particular sign a representative feature vector is extracted, which consists of hand trajectories and hand shapes. A sparse dictionary learning algorithm, Label Consistent K-SVD (LC-KSVD) is applied to obtain a discriminative dictionary. Based on that, the system was further developed to a new classification approach for better results. The proposed system was fairly evaluated based on 21 sign words including one-handed signs and two-handed signs. It was observed that the proposed system gets high recognition accuracy of 98.25%, and obtained an average accuracy of 95.34% for signer independent recognition. Keywords: Sign language, Color stream, Skeleton points, Kinect sensor, Discriminative dictionary.

In this paper, we propose two effective manners of utilizing skeleton data for human action recognition (HAR). The proposed method on one hand takes advantage of the skeleton data thanks to their robustness to human appearance change as well as the real-time performance. On the other hand, it avoids inherent drawbacks of the skeleton data such as noises, incorrect human skeleton estimation due to self-occlusion of human pose. To this end, in terms of feature designing, we propose to extract covariance descriptors from joint velocity and combine them with those of joint position. In terms of 3-D skeleton-based activity representation, we propose two schemes to select the most informative joints. The proposed method is evaluated on two benchmark datasets. On the MSRAction-3D dataset, the proposed method outperformed different hand-designed features-based methods. On the challenging dataset CMDFall, the proposed method significantly improves accuracy when compared with techniques based on recent neuronal networks. Finally, we investigate the robustness of the proposed method via a cross-dataset evaluation.

Asperspiropene A was originally reported to have a unique 1,8-dioxaspiro[4.5]decane skeleton. During the course of our ongoing research for novel marine natural products, we isolated compound 1, which has identical 1D and 2D NMR data to asperspiropene A. Detailed and careful analysis of spectroscopic data led us to revise the structure of asperspiropene A and to determine its absolute configuration.

The skeleton, or medial axis, is an important attribute of 2-D shapes. The disk B-spline curve (DBSC) is a skeleton-based parametric freeform 2-D region representation, which is defined in the B-spline form. The DBSC describes not only a 2-D region, which is suitable for describing heterogeneous materials in the region, but also the center curve (skeleton) of the region explicitly, which is suitable for animation, simulation, and recognition. In addition to being useful for error estimation of the B-spline curve, the DBSC can be used in designing and animating freeform 2-D regions. Despite increasing DBSC applications, its theory and fundamentals have not been thoroughly investigated. In this article, we discuss several fundamental properties and algorithms, such as the de Boor algorithm for DBSCs. We first derive the explicit evaluation and derivatives formulas at arbitrary points of a 2-D region (interior and boundary) represented by a DBSC and then provide heterogeneous object representation. We also introduce modeling and interactive heterogeneous object design methods for a DBSC, which consolidates DBSC theory and supports its further applications.

Human gait recognition in a multicamera environment is a challenging task in biometrics because of the presence of the large pose and variations in illumination among different views. In this work, to address the problem of variations in view, we present a novel stacked autoencoder for learning discriminant view‐invariant gait representations. Our autoencoder can efficiently and progressively translate skeleton joint coordinates from any arbitrary view to a common canonical view without requiring the prior estimation of the view angle or covariate type and without losing temporal information. Then, we construct a discriminative gait feature vector by fusing the encoded features with two other spatiotemporal gait features to feed into the main recurrent neural network. Experimental evaluations of the challenging CASIA A and CASIA B gait datasets demonstrate that the proposed approach outperformed other state‐of‐the‐art methods on single‐view gait recognition. In particular, the proposed method achieved 46.31% and 33.86% average correct class recognition on probe set ProbeBG and ProbeCL, respectively, of the CASIA B dataset while considering the view variation; this is 0.3% and 30.68% higher than previous best‐performing methods. Furthermore, in cross‐view recognition, our method shows better results over other state‐of‐the‐art methods when the view‐angle variation is large than 36°.

Humans regularly interact with their surrounding objects. Such interactions often result in strongly correlated motions between humans and the interacting objects. We thus ask: “Is it possible to infer object properties from skeletal motion alone, even without seeing the interacting object itself?” In this paper, we present a fine-grained action recognition method that learns toinfersuch latent object properties from human interaction motion alone. This inference allows us todisentanglethe motion from the object property andtransferobject properties to a given motion. We collected a large number of videos and 3D skeletal motions of performing actors using an inertial motion capture device. We analyzed similar actions and learned subtle differences between them to reveal latent properties of the interacting objects. In particular, we learned to identify the interacting object, by estimating its weight, or its spillability. Our results clearly demonstrate that motions and interacting objects are highly correlated and that related object latent properties can be inferred from 3D skeleton sequences alone, leading to new synthesis possibilities for motions involving human interaction. Our dataset is available athttp://vcc.szu.edu.cn/research/2020/IT.html.

Nonverbal communication primarily by the way of hand gestures is as old as human evolution. Before languages were developed and texts, which came much after that, hand gestures were the only way of human interaction. And hence in historical artefacts across various civilizations and religions across the world hand gestures play a predominant role, some more than others. While selecting a subject for the application of this research we wanted to zero in on a practice or religion where non-verbal communication is a prevalent part and hence our inclination towards Buddhism. In Buddhism, hand gestures (mudras) are considered as a sacred gesture that represent the different Buddha deities and their significance. The article proposes a system that identify the Buddhist hand gesture (mudras) using computer-aided technology. The system comprises a preprocessing stage, which creates a contour plot of the image to obtain the boundary of the region of interest. The features are extracted by generating a star skeleton from the preprocessed image. The star skeleton calculated by considering the local maxima of the distance signal obtained by joining the centroid with the boundary pixels describes the mudras. Each of these mudras has a different star skeleton. The star skeletons computed from the known sample images are used to build a database for the recognition system. The recognition is achieved by choosing the template with the most similar skeleton retrieved from the database.

Human skeleton extraction is a basic problem in the field of computer vision. With the rapid progress of science and technology, it has become a hot issue in the field of target detection such as pedestrian recognition, behavior monitoring, and pedestrian gesture recognition. In recent years, due to the development of deep neural networks, modeling of human joints in acquired images has made progress in skeleton extraction. However, most models have low modeling accuracy, poor real-time performance, and poor model availability. problem. Aiming at the above-mentioned human target detection problem, this paper uses the deep learning skeleton sequence model gesture recognition method in sports scenes to study, aiming to provide a gesture recognition method with strong noise resistance, good real-time performance and accurate model. This article uses motion video frame images to train the VGG16 network. Using the network to extract skeleton information can strengthen the posture feature expression, and use HOG for feature extraction, and use the Adam algorithm to optimize the network to extract more posture features, thereby improving the posture of the network Recognition accuracy. Then adjust the hyperparameters and network structure of the basic network according to the training results, and obtain the key poses in the sports scene through the final classifier.