Search

Visual Question Answering (VQA) aims to appropriately answer a text question by understanding the image content. Attention‐based VQA models mine the implicit relationships between objects according to the feature similarity, which neglects the explicit relationships between objects, for example, the relative position. Most Visual Scene Graph‐based VQA models exploit the relative positions or visual relationships between objects to construct the visual scene graph, while they suffer from the semantic insufficiency of visual edge relations. Besides, the scene graph of text modality is often ignored in these works. In this article, a novel Dual Scene Graph Enhancement Module (DSGEM) is proposed that exploits the relevant external knowledge to simultaneously construct two interpretable scene graph structures of image and text modalities, which makes the reasoning process more logical and precise. Specifically, the authors respectively build the visual and textual scene graphs with the help of commonsense knowledge and syntactic structure, which explicitly endows the specific semantics to each edge relation. Then, two scene graph enhancement modules are proposed to propagate the involved external and structural knowledge to explicitly guide the feature interaction between objects (nodes). Finally, the authors embed such two scene graph enhancement modules to existing VQA models to introduce the explicit relation reasoning ability. Experimental results on both VQA V2 and OK‐VQA datasets show that the proposed DSGEM is effective and compatible to various VQA architectures. [ABSTRACT FROM AUTHOR]

To learn the embedding representation of graph structure data corrupted by noise and outliers, existing graph structure learning networks usually follow the two-step paradigm, i.e., constructing a "good" graph structure and achieving the message passing for signals supported on the learned graph. However, the data corrupted by noise may make the learned graph structure unreliable. In this paper, we propose an adaptive graph convolutional clustering network that alternatively adjusts the graph structure and node representation layer-by-layer with back-propagation. Specifically, we design a Graph Structure Learning layer before each Graph Convolutional layer to learn the sparse graph structure from the node representations, where the graph structure is implicitly determined by the solution to the optimal self-expression problem. This is one of the first works that uses an optimization process as a Graph Network layer, which is obviously different from the function operation in traditional deep learning layers. An efficient iterative optimization algorithm is given to solve the optimal self-expression problem in the Graph Structure Learning layer. Experimental results show that the proposed method can effectively defend the negative effects of inaccurate graph structures. The code is available at https://github.com/HeXiax/SSGNN. [ABSTRACT FROM AUTHOR]

Accurate and timely destination prediction of subway passengers is of great significance in improving urban residents' travel efficiency, alleviating urban traffic pressure, and recommending the proper location-based service. Although some individual travel destination prediction methods have been proposed, the prediction performance is poor due to the large difference in travel locations of different individuals, the difficulty of evaluating the individual travel intention, the sparsity of individual travel trajectory data, and other problems. To solve these problems, this paper proposes a knowledge graph-based enhanced Transformer method (KG-Trans) for the metro individual travel destination prediction task (MITD-Pre), which contains three main modules: (1) the knowledge graph (KG) module constructs a multilayer individual travel KG from top to bottom, which accurately describes the travel individuals and their travel intentions. By analyzing the association relationship between nodes in the KG, the relationship between travel individuals can be naturally established. The learned similar travel regularity can solve the problem of sparse travel trajectories of some individuals. (2) The enhanced Transformer module extracts the dynamic and hierarchical features from the long-term sequential travel trajectory data. (3) The classifier module introduces the cross-entropy loss to constrain the uniqueness of the predicted subway travel station. The experimental results show that the proposed method obtains a higher destination prediction accuracy than the previous individual travel destination prediction methods. [ABSTRACT FROM AUTHOR]

Subspace data representation has recently become a common practice in many computer vision tasks. Low-rank representation (LRR) is one of the most successful models for clustering vectorial data according to their subspace structures. This paper explores the possibility of extending LRR for subspace data on Grassmann manifold. Rather than directly embedding the Grassmann manifold into the symmetric matrix space, an extrinsic view is taken to build the self-representation in the local area of the tangent space at each Grassmannian point, resulting in a localized LRR method on Grassmann manifold. A novel algorithm for solving the proposed model is investigated and implemented. The performance of the new clustering algorithm is assessed through experiments on several real-world data sets including MNIST handwritten digits, ballet video clips, SKIG action clips, and DynTex++ data set and highway traffic video clips. The experimental results show that the new method outperforms a number of state-of-the-art clustering methods. [ABSTRACT FROM AUTHOR]

In multicamera video surveillance, it is challenging to represent videos from different cameras properly and fuse them efficiently for specific applications such as human activity recognition and clustering. In this paper, a novel representation for multicamera video data, namely, the product Grassmann manifold (PGM), is proposed to model video sequences as points on the Grassmann manifold and integrate them as a whole in the product manifold form. In addition, with a new geometry metric on the product manifold, the conventional low rank representation (LRR) model is extended onto PGM and the new LRR model can be used for clustering nonlinear data, such as multicamera video data. To evaluate the proposed method, a number of clustering experiments are conducted on several multicamera video data sets of human activity, including the Dongzhimen Transport Hub Crowd action data set, the ACT 42 Human Action data set, and the SKIG action data set. The experiment results show that the proposed method outperforms many state-of-the-art clustering methods. [ABSTRACT FROM PUBLISHER]

This study proposes a novel multi‐view soft block diagonal representation framework for clustering complete and incomplete multi‐view data. First, given that the multi‐view self‐representation model offers better performance in exploring the intrinsic structure of multi‐view data, it can be nicely adopted to individually construct a graph for each view. Second, since an ideal block diagonal graph is beneficial for clustering, a 'soft' block diagonal affinity matrix is constructed by fusing multiple previous graphs. The soft diagonal block regulariser encourages a matrix to approximately have (not exactly) K diagonal blocks, where K is the number of clusters. This strategy adds robustness to noise and outliers. Third, to handle incomplete multi‐view data, multiple indicator matrices are utilised, which can mark the position of missing elements of each view. Finally, the alternative direction of multipliers algorithm is employed to optimise the proposed model, and the corresponding algorithm complexity and convergence are also analysed. Extensive experimental results on several real‐world datasets achieve the best performance among the state‐of‐the‐art complete and incomplete clustering methods, which proves the effectiveness of the proposed methods. [ABSTRACT FROM AUTHOR]

With the popularity of cameras and sensors, massive data are captured from various view angles or modalities, which provide abundant complementary information and also bring great challenges for traditional clustering methods. In this article, we propose a novel Adaptive K-Multiple-Means for multi-view clustering method (AKM3C). Unlike traditional multi-view K-means methods by grouping samples into $C$ clusters each with a cluster center in every view, the proposed AKM3C employs $M (M>C)$ sub-cluster centers in each view to reveal the sub-cluster structure in the multi-view data thus enhances the clustering performance. Additionally, to distinguish the importance of different views, instead of using empirical weights, AKM3C exploits the multi-view combination weights strategy to assign a weight to each view automatically and thus fuses the complementary information of different views properly to get an optimally shared bipartite graph, on which the Laplacian rank constraint is executed and the final clusters are obtained by directly partitioning. An efficient optimization algorithm proposed with complexity and convergence analysis is used to solve the proposed AKM3C method. The extensive experimental results on eight public datasets show that the proposed AKM3C performs better than state-of-the-art multi-view clustering methods. The code can be downloaded at https://drive.google.com/file/d/1CQ0royrYxKFJdNLnbBQSbDrohtfH71di/view?usp=sharing. [ABSTRACT FROM AUTHOR]

Although object detection algorithms based on deep learning have been widely used in many scenarios, they face challenges under some degraded conditions, such as low-light. A conventional solution is that image enhancement approaches are used as a separate pre-processing module to improve the quality of degraded image. However, this two-step approach makes it difficult to unify the goals of enhancement and detection, that is, low-light enhancement operations are not always helpful for subsequent object detection. Recently, some works try to integrate enhancement and detection in an end-to-end network, but still suffer from complex network structure, training convergence problem and demanding reference images. To address above problems, a plug-and-play image enhancement model is proposed in this paper, namely, low-light image enhancement (LLIE) model, which can be easily embedded into some off-the-shelf object detection methods in an end-to-end manner. LLIE is composed of a parameter estimation module and image processing module. The former learns to regress lighting enhancement parameters according to the feedback of detection network, and the latter enhances degraded image adaptively to promote subsequent detection model under low-light condition. Extensive object detection experiments on several low-light image data sets show that the performance of detector is significantly improved when LLIE is integrated. [ABSTRACT FROM AUTHOR]

The low-rank representation (LRR) method has attracted widespread attention due to its excellent performance in pattern recognition and machine learning. LRR-based variants have been proposed to solve the three existing problems in LRR: (1) the projection matrix is permanently fixed when dimensionality reduction techniques are adopted; (2) LRR fails to capture the local geometric structure; and (3) the solution deviates from the real low-rank solution. To address these problems, this paper proposes a low-rank representation with projection distance regularization (PDRLRR) via manifold optimization for clustering. In detail, we first introduce a low-dimensional projection matrix and a projection distance regularization term to fit the projected data automatically and capture the local structure of the data, respectively. Consequently, the projection matrix and representation matrix are obtained jointly. Then, we obtain a more accurate low-rank solution by minimizing the Schatten- p norm instead of the nuclear norm. Next, the projection matrix is optimized through a generalized Stiefel manifold. Extensive experiments demonstrate that our proposed method outperforms the state-of-the-art methods. • This paper proposes a novel PDRLRR model that can simultaneously address the three common problems in LRR. • Data dimensionality reduction technology is integrated into LRR, reducing the data dimensions while learning representation matrices. • For extracting the complete information, the projection distance regularization term is introduced to capture the global and local structure of the data. • The Schatten- p norm instead of the nuclear norm is employed to solve the rank minimization problem of the representation matrix, which can more accurately approximate the real low-rank solution. [ABSTRACT FROM AUTHOR]

Nowadays, face detection and head pose estimation have a lot of application such as face recognition, aiding in gaze estimation and modeling attention. For these two tasks, it is usually to design two different models. However, the head pose estimation model often depends on the region of interest (ROI) detected in advance, which means that a serial face detector is needed. Even the lightest face detector will slow down the whole forward inference time and cannot achieve real-time performance when detecting the head pose of multiple people. We can see that both face detection and head pose estimation need face features, so a shared face feature map can be used between them. In this paper, a multi-task learning model is proposed that can solve both problems simultaneously. We directly detect the location of the center point of the bounding box of face; at this location, we calculate the size of the bounding box of face and the head attitude. We evaluate our model's performance on the AFLW. The proposed model has great competitiveness with the multi-stage face attribute analysis model, and our model can achieve real-time performance. [ABSTRACT FROM AUTHOR]

Visual attention has been successfully applied in image captioning to selectively incorporate the most relevant areas to the language generation procedure. However, the attention in current image captioning methods is only guided by the hidden state of language model, e.g. LSTM (Long-Short Term Memory), indirectly and implicitly, and thus the attended areas are weakly relevant at different time steps. Besides the spatial relationship of attention areas, the temporal relationship in attention is crucial for image captioning according to the attention transmission mechanism of human vision. In this paper, we propose a new spatio-temporal memory attention (STMA) model to learn the spatio-temporal relationship in attention for image captioning. The STMA introduces the memory mechanism to the attention model through a tailored LSTM, where the new cell is used to memorize and propagate the attention information, and the output gate is used to generate attention weights. The attention in STMA transmits with memory adaptively and dependently, which builds strong temporal connections of attentions and learns the spatio-temporal relationship of attended areas simultaneously. Besides, the proposed STMA is flexible to combine with attention-based image captioning frameworks. Experiments on MS COCO dataset demonstrate the superiority of the proposed STMA model in exploring the spatio-temporal relationship in attention and improving the current attention-based image captioning. [ABSTRACT FROM AUTHOR]

With the rapid popularity of multi-camera networks, one human action is usually captured by multiple cameras located at different angles simultaneously. Multi-camera videos contain the distinct perspectives of one action, therefore multiple views can overcome the impacts of illumination and occlusion. In this paper, we propose a novel multi-camera video clustering model, named Shareability-Exclusivity Representation on Product Grassmann Manifolds (PGM-SER), to address two key issues in traditional multi-view clustering methods (MVC): (1) Most MVC methods directly construct a shared similarity matrix by fusing multi-view data or their corresponding similarity matrices, which ignores the exclusive information in each view; (2) Most MVC methods are designed for multi-view vectorial data, which cannot handle the nonlinear manifold structure hidden in multi-camera videos. The proposed PGM-SER firstly adopts Product Grassmann Manifolds to represent multi-camera videos, then simultaneously learn their shared and exclusive information in global structures to achieve multi-camera video clustering. We provide an effective optimization algorithm to solve PGM-SER and present the corresponding convergence analysis. Finally, PGM-SER is tested on three multi-camera human action video datasets and obtain satisfied experimental results. [ABSTRACT FROM AUTHOR]