Search

Due to the rich spatio-temporal visual content and complex multimodal relations, Video Question Answering (VideoQA) has become a challenging task and attracted increasing attention. Current methods usually leverage visual attention, linguistic attention, or self-attention to uncover latent correlations between video content and question semantics. Although these methods exploit interactive information between different modalities to improve comprehension ability, inter- and intra-modality correlations cannot be effectively integrated in a uniform model. To address this problem, we propose a novel VideoQA model called Cross-Attentional Spatio-Temporal Semantic Graph Networks (CASSG). Specifically, a multi-head multi-hop attention module with diversity and progressivity is first proposed to explore fine-grained interactions between different modalities in a crossing manner. Then, heterogeneous graphs are constructed from the cross-attended video frames, clips, and question words, in which the multi-stream spatio-temporal semantic graphs are designed to synchronously reasoning inter- and intra-modality correlations. Last, the global and local information fusion method is proposed to coalesce the local reasoning vector learned from multi-stream spatio-temporal semantic graphs and the global vector learned from another branch to infer the answer. Experimental results on three public VideoQA datasets confirm the effectiveness and superiority of our model compared with state-of-the-art methods.

In recent years, deep neural networks have achieved excellent performance in many fields of artificial intelligence. The requirements for the interpretability and robustness of neural networks are also increasing. In this paper, we propose to understand the functional mechanism of neural networks by extracting critical subnetworks. Specifically, we denote the critical subnetworks as a group of important channels across layers such that if they were suppressed to zeros, the final test performance would deteriorate severely. This novel perspective can not only reveal the layerwise semantic behavior within the model but also present more accurate visual explanations appearing in the data through attribution methods. Moreover, we propose two adversarial example detection methods based on the properties of sample-specific and class-specific subnetworks, which provides the possibility for increasing the model robustness.

In this paper, we propose a nonlocal total variation (NLTV)-based variational model for polarimetric synthetic aperture radar (PolSAR) data speckle reduction. This model, named WisNLTV, is obtained based on the Wishart fidelity term and the NLTV regularization defined for the complex-valued fourth-order tensor data. Since the proposed model is non-convex, an equivalent bi-convex model is obtained using the property of conjugate functions. Then, an efficient iteration algorithm is developed to solve the equivalent bi-convex model, based on the alternating minimization and the forward-backward operator splitting technique. The proposed iteration algorithm is proved to be convergent under certain conditions theoretically and numerically. Experimental results on both synthetic and real PolSAR data demonstrate that the proposed method can effectively reduce speckle noise and, meanwhile, better preserve the details and the repetitive structures such as textures and edges, and the polarimetric scattering characteristics, compared with the other methods.

In this paper, we propose a variational model for polarimetric synthetic aperture radar (PolSAR) data speckle reduction, which is based on the complex Wishart distribution of the covariance or coherency matrix and multichannel total variation (TV) regularization defined for complex-valued matrices. By assuming the TV regularization to be a prior and taking the statistical distribution of the covariance matrix in each resolution element into account, the variational model for PolSAR covariance data speckle suppression, named WisTV-C, is derived from the maximum a posteriori estimate. A similar variational model for PolSAR coherency data speckle reduction, named WisTV-T, is also obtained. As far as we know, this is the first variational model for the whole PolSAR covariance or coherency matrix data despeckling. Since the model is nonconvex, a convex relaxation iterative algorithm is designed to solve the variational problem, based on the variable splitting and alternating minimization techniques. Experimental results on both simulated and real PolSAR data demonstrate that the proposed approach notably removes speckles in the extended uniform areas and, meanwhile, better preserves the spatial resolution, the details such as edges and point scatterers, and the polarimetric scattering characteristics, compared with other methods.