Search

• This is a comprehensive review of Transformer for Skeleton-based Action Recognition. • This paper proposes a new taxonomy of transformer-style techniques. • This survey aims to help researchers systematically select promising future directions. Skeleton-based action recognition has rapidly become one of the most popular and essential research topics in computer vision. The task is to analyze the characteristics of human joints and accurately classify their behaviors through deep learning technology. Skeleton provides numerous unique advantages over other data modalities, such as robustness, compactness, noise immunity, etc. In particular, the skeleton modality is extremely lightweight, which is especially beneficial for deep learning research in low-resource environments. Due to the non-European nature of skeleton data, Graph Convolution Network (GCN) has become mainstream in the past few years, leveraging the benefits of processing topological information. However, with the explosive development of transformer methods in natural language processing and computer vision, many works have applied transformer into the field of skeleton action recognition, breaking the accuracy monopoly of GCN. Therefore, we conduct a survey using transformer method for skeleton-based action recognition, forming of a taxonomy on existing works. This paper gives a comprehensive overview of the recent transformer techniques for skeleton action recognition, proposes a taxonomy of transformer-style techniques for action recognition, conducts a detailed study on benchmark datasets, compares the algorithm accuracy of standard methods, and finally discusses the future research directions and trends. To the best of our knowledge, this study is the first to describe skeleton-based action recognition techniques in the style of transformers and to suggest novel recognition taxonomies in a review. We are confident that Transformer-based action recognition technology will become mainstream in the near future, so this survey aims to help researchers systematically learn core tasks, select appropriate datasets, understand current challenges, and select promising future directions. [ABSTRACT FROM AUTHOR]

• We propose a new graph convolution network with subgraph embedding. • It extracts local information with subgraphs partitioned by Girvan Newman algorithm. • Experiments confirm the superiority of predicting mutagenicity in aromatic hydrocarbons. • The accuracy is verified with comparison of nine SOTA deep learning models. • The proposed method prevents about 15 %p of GCN's information dilution. An aromatic hydrocarbon refers to an organic material having a carbon ring such as benzene and a functional group in the carbon ring. As the industry develops, natural pollution becomes harsh, new compounds emerge, and the exposure to aromatic hydrocarbons is continuously increasing. Predicting mutagenicity is one of the crucial issues in reducing the risk because these organisms may have properties that penetrate the DNA of living things to cause mutations. Recently, the accuracy of mutation prediction has improved due to the power of deep learning. However, most conventional methods do not consider the characteristics of molecular aromatic hydrocarbons, which dilutes local information and results in a severe deterioration of the prediction performance. In this paper, we propose a method of exploiting subgraph convolution neural networks that enables the extraction of local information of a graph by partitioning it to maintain the detailed information. For extracting the features of molecules, we use the Girvan Newman algorithm to partition the graph according to the carbon ring and functional group and obtain the embedding vectors of the subgraphs as well as the original graph with graph convolution network (GCN). The embedding vectors are combined to represent the whole graph information and predict mutagenicity. Experiments with MUTAG, NCI1 and NCI109, datasets for predicting mutagenicity of molecules in graph structure, confirm that we successfully segment carbon rings and functional groups from molecular graphs and predict mutations using the partitioned graphs, leading to a 2 %p performance improvement. In addition, the proposed method has prevented about 15 %p of information dilution in GCN, and an analysis of the latent space of graphs reveals that the subgraphs extracted maintain the local information appropriately. [ABSTRACT FROM AUTHOR]

• A center-connected graph convolutional network enhanced with salient image features (SIFE-CGCN) was proposed by integrating the image semantics to improve the recognition performance of similar actions. • A center-connected skeleton topology was developed to enhance the learning capability of the GCN on the potential cooperative dependencies of all joints. • The DTW-based metric was developed to measure the action similarity and build the similar action dataset. The proposed model achieves state-of-the-art performance on three large-scale datasets. Skeleton-based human action recognition has drawn more and more attention due to its easy implementation and stable application in intelligent human-robot interaction. However, most existing studies only used the skeleton data but completely ignored other image semantic information to build action recognition models, which would confuse the recognition of similar actions because of the ambiguity between skeleton data. Here, a center-connected graph convolutional network enhanced with salient image features (SIFE-CGCN) is proposed to address the problem of similar action recognition. First, a center-connected network (CGCN) is constructed to capture the small differences between similar actions through exploring the possible collaboration between all joints. Subsequently, a metric of movement changes is employed to optimally select the salient image from an action video, and then the EfficientNet is used to achieve the action semantic classification of the salient images. Finally, the recognition results of CGCN are strengthened with the classification results of salient images to further improve the recognition accuracy for similar actions. Additionally, a metric is proposed to measure the action similarity with the skeleton data, and then a similar action dataset is built. Extensive experiments on the datasets of similar action and NTU RGB + D 60/120 were conducted to verify the performance of the proposed methods. Experimental results validated the effectiveness of salient image feature enhancement and showed that the proposed SIFE-CGCN achieved the state-of-the-art performance on the similar action and NTU RGB + D 60/120 datasets. [ABSTRACT FROM AUTHOR]

Graph convolutional neural networks (GCNs) with domain-specific feature aggregation capabilities have unique advantages in hyperspectral image (HSI) classification. However, current GCN-based approaches frequently encounter the issue of node characteristics being over-smoothed while aggregating in higher-order domains. Furthermore, GCN linear classifiers focus solely on sample separability and ignore the potential manifold information of graph features, resulting in a failure to fully investigate extracted features. To address these problems, we propose a global-local manifold embedding broad graph convolutional network (GLMBG) for HSI classification. In GLMBG, we designed two modules from feature extraction and classification perspectives: The graph convolutional edge feature fusion extractor (GEFF) and the broad classifier of global-local manifold embedding (BGLME). GEFF is designed to learn graph node and local edge features from HSI through GCN and recursive filtering, combining them in a weighted manner to construct fused graph features. BGLME is designed to replace traditional linear classifiers with broad learning classifiers through manifold regularized embedding, fully utilizing the global and local manifold discriminant information of graph node features. The combination of GEFF and BGLME effectively reduces over-smoothing of graph node features while maximizing the utilization of manifold discriminant information, hence improving model feature discriminative ability. Experimental evaluations of three commonly used hyperspectral datasets show that our method surpasses state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Traffic forecasting is an increasingly important research topic in the field of Intelligent Transportation Systems (ITS). In this field, prediction models based on Graph Convolution Networks (GCN) have become very popular. Most GCN-based models focus on constructing various optimized or dynamic road network graphs to represent the spatio-temporal correlation hidden in traffic data. However, these methods currently only consider the construction of a single improved road network graph and ignore the relationship of these existing optimized road network graphs. Therefore, in this paper, we propose a Contrastive Optimized Graph Convolution Network (COGCN) to connect two kinds of optimized road network graphs and maintain their global–local feature consistency through contrastive learning. The proposed COGCN model is evaluated in detail using four real traffic datasets: two traffic speed datasets and two traffic flow datasets. Experimental results show that COGCN improves forecasting accuracy by at least 2% on the two speed datasets and 9% on the two flow datasets compared to the existing state-of-the-art GCN-based methods. [ABSTRACT FROM AUTHOR]

Accurately predicting traffic flow characteristics is crucial for effective urban transportation management. Emergence of artificial intelligence has led to the surge of deep learning methods for short-term traffic forecast. Notably, Graph Convolutional Neural Networks (GCN) have demonstrated remarkable prediction accuracy by incorporating road network topology into deep neural networks. However, many existing GCN-based models are based on the premise that the graph network is static, which may fail to do justice in replicating the situations in the real World. On one hand, real road networks are dynamic and undergo changes such as road maintenance and traffic control, leading to altered network structures over time. On the other hand, relationships between road sections can fluctuate due to factors like traffic accidents, weather conditions, and other events, which can significantly impact traffic patterns and result in inaccurate predictions if a static network and static relationships between nodes are assumed. To address these challenges, we propose the spatiotemporal dual adaptive graph convolutional network (ST-DAGCN) model for spatiotemporal traffic prediction, which utilizes a dual-adaptive adjacency matrix comprising both a static and a dynamic graph structure learning matrix. The dual-adaptive mechanism can adaptively learn the global features and the local dynamic features of the traffic states by updating the correlations of nodes at each prediction step, while the gated recurrent unit (GRU), which is also a component of the model, extracts the temporal dependencies of traffic data. Through a comprehensive comparison analysis on two real-world traffic datasets, our model has achieved the highest prediction accuracy when compared to other advanced models. [ABSTRACT FROM AUTHOR]

Existing methods that use document-level information for event detection ignore the dependencies between sentences and also have shortcomings in modeling the dependencies among words. In this paper, we propose a novel H ierarchical G raph E nhanced E vent D etection (HGEED) framework to make full use of syntax and document information for the task of event detection. First, a sentence graph is used to model word-to-word dependencies, enriching the local information of words by incorporating syntactic features. Then, a document graph is built to model sentence-to-sentence dependencies, obtaining global semantic representations for word-level prediction. The experiment results on the widely used ACE 2005 and TAC KBP 2015 corpora show that our model can capture local and global information with dependencies and achieve significant improvements as compared to all baselines. [ABSTRACT FROM AUTHOR]

Generally, Graph convolution network (GCN) utilizes the graph convolution operators and the softmax to extract the deep representation and make the prediction, respectively. Although GCN successfully represents the connectivity relationship among the nodes by aggregating the information on the graph, the softmax-based decision layer may result in suboptimal performance in semi-supervised learning with less label support due to ignoring the inner distribution of the graph nodes. Besides, the gradient descent will take thousands of interaction for optimization. To address the referred issues, we propose a novel graph deep model with a non-gradient decision layer for graph mining. Firstly, manifold learning is unified with label local-structure preservation to capture the topological information and make accurate predictions with limited label support. Moreover, it is theoretically proven to have analytical solutions and acts as a non-gradient decision layer in graph convolution networks. Particularly, a joint optimization method is designed for this graph model, which extremely accelerates the convergence of the model. Finally, extensive experiments show that the proposed model has achieved excellent performance compared to the current models. [Display omitted] • Unifying the orthogonal manifold with label local-structure preservation to mine the topological information of the deep embeddings and make more accurate predictions, the novel non-gradient graph decision layer is put forward. • With the assistance of the designed theorems, the non-gradient graph decision layer can be solved with an elegant analytical solution theoretically. • By embedding the analytical solution into the gradient descent, a joint optimization strategy is designed to jointly optimize the graph convolution network and the proposed non-gradient decision layer. [ABSTRACT FROM AUTHOR]

In location-based social networks, calibrating a point-of-interest (POI) recommendation system is as important as its accuracy for improving user satisfaction. POI recommendation calibration is primarily classified as categorical or geographical calibration. Categorical calibration ensures that the recommended items are distributed proportionally among the past interest categories of the target user. When a target user checks 80 Chinese, 10 Japanese, and 10 French restaurants, a recommendation list with a ratio of 8:1:1 for Chinese, Japanese, and French restaurants can be reasonably expected. In addition to categorical calibration, geographical calibration has been proposed to increase user interest in the recommended results. Users have a high probability of revisiting locations in their subareas. Therefore, the POIs recommended in multiple subareas of interest are more suitable than those from one small and frequently visited subarea. However, improving the calibration and accuracy are conflicting tasks. To achieve high calibration while maintaining accuracy, previous studies proposed reranking-based techniques to rerank the candidate list and return POIs with high calibration. However, optimizing the calibration by reranking is independent of the basic-candidate-item generation model, resulting in a suboptimal system. To tackle the problem, we propose a novel sampling-based differentiation technique to merge the task of improving calibration into the GCN model training process and directly generate the final recommendation list. The model is flexible and can be applied to different domains, where a domain can be a subarea or category. In a three-layer GCN, the layer one represents the historical check-ins of the user, whereas layer three includes the candidate POIs from which the target user aggregates information. We trained the model to make the distribution of the POI domains at layer three approximated the distribution at layer one. Experimental results on Philadelphia and Tucson datasets confirmed that the proposed method outperforms all state-of-the-art GCN+ geo-reranking and GCN+ MCF baselines, improving Recall@ 5 from 0.0394 to 0.0412 (4.57%) and Jensen–Shannon measure (JS)@ 5 from 0.5931 to 0.6734 (13.54%) on the Philadelphia dataset and improving Recall@ 5 from 0.0495 to 0.0517 (4.40%) and JS@ 5 from 0.5869 to 0.6598 (12.42%) on the Tucson dataset for categorical calibration. The model was also tested in the geographical domain and a similar trend was observed. [ABSTRACT FROM AUTHOR]

To explicitly exploit the collaborative signals in the user-item interaction graph, a growing number of recent Collaborative Filtering (CF) studies adopt Graph Convolution Network (GCN) as a basis. Though effective, these methods basically treat all neighbors equally, ignoring the fact that neighbors should be target-related. While there are some works that assign different weights to neighbors via using the attention mechanism, they still suffer from two problems. First, the performance of them is limited by the lack of fine-grained details. Second, attention learned solely from interaction data may not reflect the user's opinions accurately. To address these issues, we propose a Review-based Feature-level Information Aggregation (RFIA) graph model that incorporates the review information into the graph propagation to achieve more fine-grained information aggregation. The main idea of RFIA is to assign feature-level attention vectors for the interaction edges to adaptively adjust the contribution ratios of input neighbors across various dimensions, based on rich review information. Specifically, we first extract review features from text by BERT-Whitening. Then, we design non-linear feature extractors separately in two directions to further extract and refine these review features as feature-level attention. Finally, we design a graph contrastive learning module to optimize the learning of extractors under limited user behaviors. Experiments on three publicly available datasets validate the effectiveness and performance superiority of our proposed model. [ABSTRACT FROM AUTHOR]

In location-based social networks (LBSNs), point-of-interest (POI) recommendation systems help users identify unvisited POIs by filtering large amounts of information. Accurate POI recommendations can effectively improve user satisfaction and save time in finding POIs. In recent years, the graph convolution network (GCN) technique, which enhances the representational ability of neural networks by learning the embeddings of users and items, has been widely adopted in recommendation systems to improve accuracy. Combining GCN with various information, such as time and geographical information, can further improve recommendation performance. However, existing GCN-based techniques simply adopt time information by modeling users' check-in sequences, which is insufficient and ignores users' time-based high-order connectivity. Note that time-based high-order connectivity refers to the relationship between indirect neighbors with similar preferences in the same time slot. In this paper, we propose a new time-aware GCN model to extract rich collaborative signals contained in time information. Our work is the first to divide user check-ins into multiple subgraphs, i.e., time slots, based on time information. We further propose an edge propagation module to adjust edge affiliation, where edges represent check-ins, to propagate user's time-based preference to multiple time slots. The propagation module is based on an unsupervised learning algorithm and does not require additional ground-truth labels. Experimental results confirm that our method outperforms state-of-the-art GCN models in all baselines, improving Recall @ 5 from 0.0803 to 0.0874 (8.84%) on the Gowalla dataset and from 0.0360 to 0.0388 (7.78%) on the New York dataset. The proposed subgraph mining technique and novel edge-based propagation module have high scalability and can be applied to other subgraph construction models. [ABSTRACT FROM AUTHOR]