Search

• A pose-position attention strategy is proposed to update the bi-modal visual graph. • A linguistic-embedding-based semantic graph is presented to model label relations. • A semantic-preserving loss is designed to maintain the semantics consistency. • Both visual and semantic information are fused for group activity recognition. Video-based group activities typically contain interactive contexts among diverse visual modalities between multiple persons, and semantic relationships between individual actions. Nevertheless, majority of the existing methods for recognizing group activity either captures the relationships among different persons by utilizing a solely RGB modality or neglect to exploit the label hierarchies between individual actions and the group activity. To tackle these issues, we propose a visual-semantic graph neural network, with pose-position attentive learning (VSGNN-PAL), for group activity recognition. Specifically, we first extract the individual-level appearance and motion representations from RGB and optical-flow inputs, to build a bi-modal visual graph. Two attentive aggregators are further proposed to integrate both the pose and position information to measure the relevance scores between persons, and dynamically refine the representation of each visual node from both modality-specific and cross-modal perspectives. To model a semantic hierarchy from a label space, we construct a semantic graph based on the linguistic embeddings of individual actions and group activity labels. We further employ a bi-directional mapping learning scheme, to integrate the label-relation-aware semantic context into the visual representations. Besides, a global reasoning module is introduced to progressively generate the group-level representations with the scene description maintained. Furthermore, we formulate a semantic-preserving loss, to maintain the consistency between the learned high-level representations and the semantics of the ground-truth labels. Experimental results on three group activity benchmarks demonstrate that the proposed method achieves state-of-the-art performance. [ABSTRACT FROM AUTHOR]

The emergence of wearable devices has opened up new potentials for egocentric activity recognition. Although some methods integrate attention mechanisms into deep neural networks to capture fine-grained human-object interactions in a weak-supervision manner, they either ignore exploiting the temporal consistency or generate attention based on considering appearance cues only. To address these limitations, in this paper, we propose an enhanced attention-tracking method, combined with multi-branch network (EAT-MBNet), for egocentric activity recognition. Specifically, we propose class-aware attention maps (CAAMs) by employing a self-attention-based module to refine the class activation maps (CAMs). Our proposed method can enhance the semantic dependency between the activity categories and the feature maps. To highlight the discriminative features from the regions of interest across frames, we propose a flow-guided attention-tracking (F-AT) module, by simultaneously leveraging historical attention and motion patterns. Furthermore, we propose a cross-modality modeling branch based on an interactive GRU module, which captures the time-synchronized long-term relationships between the appearance and motion branches. Experimental results on four egocentric activity benchmarks demonstrate that the proposed method achieves state-of-the-art performance. [ABSTRACT FROM AUTHOR]

Single image super-resolution (SISR) is a classic ill-posed problem in computer vision. In recent years, deep-learning-based (DL-based) models have achieved promising results with the SISR problem. However, most existing methods suffer from an intrinsic trade-off between distortion and perceptual quality. To satisfy the requirements in different real-world situations, the balance of distortion and visual quality for image super-resolution is a critical issue. In DL-based models, the uses of hybrid loss (i.e., the combination of the distortion loss and the perceptual loss) and network interpolation are two common approaches to balancing the distortion and perceptual quality of super-resolved images. However, these two kinds of methods lack flexibility and hold strict constraints on network architectures. In this paper, we propose an image-fusion interpolation method for image super-resolution, which can balance the distortion and visual quality of super-resolved images, based on the optimal transport theory in the wavelet domain. The advantage of our proposed method is that it can be applied to any pretrained DL-based model, without any requirement from the network architecture and parameters. In addition, our proposed method is parameter-free and can run fast without using a GPU. Compared with existing state-of-the-art SISR methods, experiment results show that our proposed method can achieve a better balance between the distortion and visual quality in super-resolved images. [ABSTRACT FROM AUTHOR]

The emergence of low-cost depth cameras creates potential for RGB-D based human action recognition. However, most of the existing RGB-D based approaches simply concatenate original heterogeneous features without discovering the latent relations among different modalities. We propose a discriminative common structure learning (DCSL) model for human action recognition from RGB-D sequences. Specifically, we extract deep learning-based features and hand-crafted features from multimodal data (skeleton, depth, and RGB). In particular, we propose a deep architecture based on 3-D convolutional neural network to automatically extract deep spatiotemporal features from raw sequences. The proposed DCSL model utilizes a generalized version of collective matrix factorization to learn shared features among different modalities. To perform supervised learning and preserve intermodal similarity, we formulate a graph regularization term by considering both label information and similar geometric structure of multimodal data, which intends to improve the discriminative power of shared features. Moreover, we solve the objective function using an iterative optimization algorithm. Then, an improved collaborative representation classifier is employed to perform computationally efficient action recognition. Experimental results on four action datasets demonstrate the superior performance of the proposed method. [ABSTRACT FROM AUTHOR]

Highlights • Our CMFL model jointly learns shared-specific features and action classifiers. • The proposed RSTPF features extract dynamic local patterns around each human joint. • The CMFL model enables the features to be optimized for classification. • The CMFL performs well even if one or two modalities are missing in the testing stage. • A max-margin framework is introduced to fuse skeleton, depth and RGB data. Abstract The emergence of cost-effective depth sensors opens up a new dimension for RGB-D based human action recognition. In this paper, we propose a collaborative multimodal feature learning (CMFL) model for human action recognition from RGB-D sequences. Specifically, we propose a robust spatio-temporal pyramid feature (RSTPF) to capture dynamic local patterns around each human joint. The proposed CMFL model fuses multimodal data (skeleton, depth and RGB), and learns action classifiers using the fused features. The original low-level feature matrices are factorized to learn shared features and modality-specific features under a supervised fashion. The shared features describe the common structures among the three modalities while the modality-specific features capture intrinsic information of each modality. We formulate shared-specific features mining and action classifiers learning in a unified max-margin framework, and solve the formulation using an iterative optimization algorithm. Experimental results on four action datasets demonstrate the efficacy of the proposed method. [ABSTRACT FROM AUTHOR]

High dynamic range (HDR) imaging is an important task in image processing that aims to generate well-exposed images in scenes with varying illumination. Although existing multi-exposure fusion methods have achieved impressive results, generating high-quality HDR images in dynamic scenes remains difficult. The primary challenges are ghosting artifacts caused by object motion between low dynamic range images and distorted content in underexposure and overexposed regions. In this paper, we propose a deep progressive feature aggregation network for improving HDR imaging quality in dynamic scenes. To address the issues of object motion, our method implicitly samples high-correspondence features and aggregates them in a coarse-to-fine manner for alignment. In addition, our method adopts a densely connected network structure based on the discrete wavelet transform, which aims to decompose the input features into multiple frequency subbands and adaptively restore corrupted contents. Experiments show that our proposed method can achieve state-of-the-art performance under different scenes, compared to other promising HDR imaging methods. Specifically, the HDR images generated by our method contain cleaner and more detailed content, with fewer distortions, leading to better visual quality. [ABSTRACT FROM AUTHOR]

A novel mechanism for skeleton-based action recognition is proposed in this paper by enhancing and fusing diverse skeleton features from distinct levels. Graph convolutional neural networks (GCNs) have been proven to be efficient in skeleton-based action recognition. However, most graph convolutional networks tend to capture and fuse discriminative information from different forms of data in spatial neighborhoods. In that case, the deeper interactions among different forms of data as well as the extraction of information in the temporal and channel dimensions are limited. To tackle the issue, we propose the ternary adaptive graph convolution (TAGC) module to capture spatiotemporal information by graph convolution. A novel skeleton information, called parallax information, is explored from original joints or bones with little computation to further improve the performance of action recognition. In addition, in order to make better use of multiple streams, multi-stream feature fusion (MSFF) is proposed to mine deeper-level hybrid features supplementing the original streams. And a graph-based ternary enhance (GTE) module is proposed to further refine the extracted discriminative features. Finally, the proposed multi-stream ternary enhanced graph convolutional network (MS-TEGCN) achieves the state-of-the-art results through extensive experiments on three challenging datasets for skeleton-based action recognition, NTU-60, NTU-120 and Kinetics-Skeleton. [ABSTRACT FROM AUTHOR]

End-to-end multiple object tracking (MOT) has been very popular recently in the field of computer vision. However, when performing the inter-frame data association, the majority of algorithms ignore the diversity among trajectories and treat them uniformly. In the joint-detection-and-tracking (JDT) paradigm, there exists an optimization conflict between detection and re-identification due to their different optimization directions. In this paper, we propose a dynamic trajectory quantification (DTQ) strategy for MOT with feature rearrangement. Initially, the DTQ strategy quantifies each trajectory to show its quality, and the score is renewed every frame. To adapt to different situations, we dynamically adopt different updating mechanisms. Additionally, we provide a channel-enhanced feature rearrangement module to alleviate the optimization conflict between subtasks in the JDT paradigm, which obtains more convincing detection results for the DTQ strategy. We evaluate our proposed tracker on several benchmarks, i.e., MOT15, MOT16, MOT17, and MOT20. Extensive experimental results demonstrate that our method achieves competitive results while still maintaining a sizable real-time tracking speed. [ABSTRACT FROM AUTHOR]

The stable continuation of trajectories among different targets has always been the key to the tracking performance of multi-object tracking (MOT) tasks. If features of the target are aggregated and classified simply, the discriminant features of the target will be ignored. This will affect the robustness of the trajectory generated by the model. Meanwhile, many popular models are keen to execute detection and feature extraction tasks in parallel. But these two tasks will conflict with each other when optimized respectively. Therefore, we propose our tracker MOTFR to solve the above problems. In this paper, we propose a Locally Shared Information Decoupling Module (LSIDM) to reduce task optimization conflicts while ensuring the necessary information sharing. Meanwhile, a feature recoding module for deep extraction of identity discriminative features is proposed, which is called the Feature Purification Module (FPM). By combining LSIDM and FPM modules, the model utilizes the discriminative appearance features to guide the optimization of detection and further improves the performance of our model. To solve the problem of targets disappearing due to various abnormal occlusion, a Short-term Trajectory Online Complement Strategy (STOCS) is proposed to realize the trajectories continuation of these targets in the tracking stage. Through sufficient experiments, we demonstrate the superior performance of our MOTFR, which guarantees high-quality detection while achieving the stability of the target trajectory. [ABSTRACT FROM AUTHOR]

Unsupervised person re-identification (ReID) is a task that aims to retrieve pedestrians across different cameras from unlabeled data. Existing methods rely on clustering to generate pseudo-labels, but they are inevitably noisy. Although pseudo-label refinement approaches have been presented, the essentiality of patch contours is ignored. The tendency analysis of retrieval between global and patch features has not been well investigated. In this paper, we propose a Patch-based Tendency Camera Multi-Constraint Learning (PTCML) model for unsupervised person ReID. First, to explore the tendentious retrieval of global and patch features, we design a Ranking Tendency Similarity (RTS) score by gauging the distribution discrepancy of distance changes. Second, based on RTS score, we propose a Tendency-based Mutual Complementation (TMC) loss to improve the quality of global and patch pseudo-labels. Third, to resist camera variations, we propose an Adaptive Camera Multi-Constraint (ACM) loss to optimize recognition results with camera distribution constraint and instance constraint simultaneously. Finally, numerous experiments on Market-1501 and MSMT17 demonstrate that our method can significantly surpass the state-of-the-art performance. • We design RTS score to mine the complementary tendency of increased distance between global and patch features. • Based on RTS score, TMC loss is proposed to refine the pseudo-labels. • We propose ACM loss to apply camera distribution and instance constraints simultaneously. [ABSTRACT FROM AUTHOR]

Unsupervised domain adaptation (UDA) person re-identification (ReID) faces enormous challenges due to the severe shift between the source and target domains, as well as the dramatic variations within the target domain. In this paper, to address these issues, we propose a multi-loss gap minimization learning (MGML) approach for UDA person ReID. Firstly, we introduce the part model to learn discriminative patch features and design a Patch-based Part Ignoring (PPI) loss to select reliable instances for the efficient learning of the part model. Then, given the gap that typically occurs because of the inter-domain shift and intra-domain variations, a Gap-based Minimum Camera Discrepancy (G-MCD) loss is proposed. Specifically, in terms of the inter-domain, we propose to leverage the tracklet and camera information to label each distance vector, and accordingly align pair-wise distance distributions to bridge the inter-domain gap. As for the intra-domain, to alleviate the biased search, we propose to perform the mined neighborhood for intra-camera and inter-camera separately to optimize matching results by exploring neighborhood relations more deeply. Finally, experimental results on three challenging datasets demonstrate that applying our method to unlabeled target domain outperforms current UDA methods for person ReID. [ABSTRACT FROM AUTHOR]

Most existing multi-object tracking models only focus on the spatial information generated by image-level input while ignoring the necessary temporal information. The temporal motion information between consecutive frames can effectively reflect the target's motion status, which is essential to improving the performance in dealing with occlusion and motion blur of the model. We propose MISTracker to realize the motion information supplement to the original tracking model. Specifically, we divide multi-scale feature maps into two categories from the perspective of space and channel information. Meanwhile, the spatial-level frame differences processing (SFDP) module and the channel-level frame differences processing (CFDP) module are proposed to deal with these differences between continuous frames, respectively. The SFDP processes the differences from the perspective of spatial information and supplements the motion information through the perception of pixel-level information changes in the feature maps. The CFDP processes the differences from the perspective of channels and enhances the information of motion-sensitive channels through the overall pixel differences of different channels. Eventually, temporal and motion information are complementary to each other after upsampling fusion. The whole process is realized by simple convolution, which reduces the computational force as much as possible and enhances the tracking performance of the model. [ABSTRACT FROM AUTHOR]

• The ADJST realizes users' anomalous electricity consumption behaviors detection. • The MS-GCN achieves extracting spatial features and preventing feature coupling. • The MS-TCN module reduces the loss of time sequence information in long sequences. • Enhance the model's stability and generalization by joint spatio-temporal learning. The use of electric energy is an integral part of people's daily life. Anomaly detection of electricity consumption data, as a classification problem, has always been a hot research topic of scholars. Anomaly detection can not only reduce energy waste, but also prevent small problems from becoming overwhelming problems. At present, most anomaly detection algorithms mainly focus on the time series information of electricity consumption data, while ignoring the spatial feature of electricity consumption data. To fill this research gap, the paper proposes an Anomaly Detection based on Joint Spatio-Temporal learning (ADJST) method for building electricity consumption. First, a Multi-Scale Graph Convolutional Network (MS-GCN) is proposed to learn the spatial features of building electricity consumption data. Specifically, two types of graphs are constructed to extract short-term correlation features and long-term regularity features of building electricity consumption data. Second, a Multi-Scale Temporal Convolutional Network (MS-TCN) is proposed to learn the temporal features of building electricity consumption data. Adopt a multi-scale vanilla convolution structure to extract multi-scale time series information from building electricity consumption data. Third, the combination of temporal features and spatial features detects anomalous electricity consumption of marked users. Final, taken the user electricity consumption data collected by the State Grid Corporation's smart meter as examples, compared with a variety of classical anomaly detection algorithms, the results of F1-score and AUC of the proposed method are 0.935 and 0.977 respectively, which proves the superiority of the method. The model shows good stability in dealing with extreme imbalance of data, and is proved to be generalized by experiments and can be transferred to other datasets. [ABSTRACT FROM AUTHOR]

The skeleton-based action recognition task has recently drawn much attention in computer vision, and graph convolutional networks (GCNs) have shown great advantages in this task. However, in shallow layers of GCNs, the sharing of convolution kernels in regular graph convolution (RGC) may result in the underutilization of some convolution kernels due to the particularity of the skeleton data. To capture richer locally discriminative action features, we propose the exclusive kernel graph convolution (EKGC) for the shallow layers of GCNs, which assigns the vertexes exclusive convolution kernels for graph convolution. At the same time, following the mixed graph convolution strategy, RGC is retained in the deep layers of the network, and the mixed graph convolutional network (MGCN) is proposed to utilize EKGC and RGC reasonably in all layers of the network. In addition, as a new form of skeleton information in GCN-based methods, angle information is explored in our work. Compared with joint data and bone data, angle data have better invariance against rotation and scaling, and they expand the relationships between skeletons and enrich the data representation of action. Finally, our final model, the multi-stream MGCN, is validated on two challenging datasets, achieving the state-of-the-art performance on the NTU-RGBD dataset and making competitive performance on a kinetics-skeleton dataset. [ABSTRACT FROM AUTHOR]

The two-stream convolutional network has been proved to be one milestone in the study of video-based action recognition. Lots of recent works modify internal structure of two-stream convolutional network directly and put top-level features into a 2D/3D convolution fusion module or a simpler one. However, these fusion methods cannot fully utilize features and the way fusing only top-level features lacks rich vital details. To tackle these issues, a novel network called Diverse Features Fusion Network (DFFN) is proposed. The fusion stream of DFFN contains two types of uniquely designed modules, the diverse compact bilinear fusion (DCBF) module and the channel-spatial attention (CSA) module, to distill and refine diverse compact spatiotemporal features. The DCBF modules use the diverse compact bilinear algorithm to fuse features extracted from multiple layers of the base network that are called diverse features in this paper. Further, the CSA module leverages channel attention and multi-size spatial attention to boost key information as well as restraining the noise of fusion features. We evaluate our three-stream network DFFN on three public challenging video action benchmarks: UCF101, HMDB51 and Something-Something V1. Experiment results indicate that our method achieves state-of-the-art performance. • Diverse compact bilinear fusion modules fuse spatial features and temporal features. • Diverse compact bilinear fusion modules fuse diverse features from different layers. • The multi-size spatial-channel attention module can adjust the attention of network. • The proposed network achieves state-of-the-art performance. [ABSTRACT FROM AUTHOR]

As a challenging task of video classification, action recognition has become a significant topic of computer vision community. The most popular methods based on two-stream architecture up to now are still simply fusing the prediction scores of each stream. In that case, the complementary characteristics of two streams cannot be fully utilized and the effect of shallower features is often overlooked. In addition, the equal treatment to features may weaken the role of the feature contributing significantly to the classification. Accordingly, a novel network called Multiple Depth-levels Features Fusion Enhanced Network (MDFFEN) is proposed. It improves on two aspects of two-stream architecture. In terms of the two-stream interaction mechanism, multiple depth-levels features fusion (MDFF) is formed to aggregate spatial–temporal features extracted from several sub-modules of original two streams by spatial–temporal features fusion (STFF). And with respect to further refining the spatiotemporal features, we propose a group-wise spatial-channel enhance (GSCE) module to highlight the meaningful regions and expressive channels automatically by priority assignment. The competitive results are achieved after we validate MDFFEN on three public challenging action recognition datasets, HDMB51, UCF101 and ChaLearn LAP IsoGD. • The proposed Two-Stream Features Fusion extracts diverse hybrid features. • The fusion of three types of features learns more discriminative information. • The Group-wise Spatial-Channel Enhance mechanism reduces interference information. • The proposed network achieves competitive performance. [ABSTRACT FROM AUTHOR]