Search

Digital video is widely used to record people's daily lives and share people's moods, but few researchers have conducted research on the consistency of emotional expression between short videos and music. In order to be able to match the appropriate background music to the short video image autonomously and efficiently, the paper analyzed the emotional connection between the two from the audio-visual synesthesia. First, emotional semantics was used as a bridge to connect video data and music data, and a video-music synesthesia data set based on semantic words was constructed. Then, an attention mechanism was incorporated to better extract key features in video images. In the extraction of music features, an improved lenet5 network was used, and the optimal network parameters were determined through experiments. Finally, the two types of features were fused and the mutual retrieval between video and music was performed. In order to compare the performance of different models, different CNN models were calculated in the processing of video images, including VGG16, VGG19, AlexNet and GoogleNet, and the attention mechanism was added to each network for calculation to compare its retrieval accuracy. In the processing of music data, different CNN algorithms were also used for comparative experiments, and networks with different layers were used to determine the optimal results. The experimental results show that the audiovisual synesthesia retrieval model based on emotion can effectively measure the emotional similarity between video images and music, and the method of the paper can produce a good match between them. The research method of the paper is the exploration of computer synesthetic intelligence, which can stimulate the creative inspiration of image and music creative designers. While enhancing the emotional experience of digital products, it also improves the efficiency and quality of development. [ABSTRACT FROM AUTHOR]

Feature pyramid network (FPN) is widely used in object detection in order to divide and conquer objects of different scales and to fuse high and low-level features, and it has achieved encouraging achievements in multi-scale object processing. However, due to the mismatch between receptive fields at different stages, the direct fusion of the two features from different receptive fields may be unable to achieve satisfactory results. Moreover, simple lateral connections in FPN may lead to loss of spatial relationships and details. To alleviate these problems, in this paper we propose a field-matching attention network (FMANet) for object detection. Particularly, we first propose a receptive field dilated module (RFDM), which is used to normalize receptive fields between features at different stages to the same scale. Furthermore, to capture the spatial informations and details, we build a dual attention module (DAM) by employing the spatial attention and channel attention. Utilizing both spatial and channel attention mechanisms simultaneously improves performance while maintaining speed. Finally, experimental results reveal that our proposed FMANet with DSPDarkNet-53 as backbone achieves a competitive detection performance. [ABSTRACT FROM AUTHOR]

Convolutional neural networks have been widely used in image semantic segmentation. However, continuous downsampling operations in convolutional neural networks (such as pooling or convolution with step size) reduce the initial image resolution and lose the spatial details of the image, resulting in blurred image segmentation results. To alleviate this problem, in this paper we propose a multi-stage context refinement network (MCRNet) for semantic segmentation. Specifically, we first construct a Lowest-resolution Chain Context Aggregation (LCCA) module to encode rich semantic information. For obtaining more spatial detail information, we further build a High-resolution Context Attention Refinement (HCAR) module consisting of context feature extraction and context feature refinement. Finally, MCRNet fuses the context information generated by LCCA and HCAR for pixel prediction. Experimental results on three challenging semantic segmentation datasets, namely PASCAL VOC2012, ADE20K and Cityscapes, reveals that our proposed MCRNet is effective. [ABSTRACT FROM AUTHOR]

Zero-Shot Learning (ZSL) aims to recognise unseen object classes which are not observed during the training phase. Most of the existing methods on ZSL focus on learning a compatibility function between the image representation and class semantic information. Few others concentrate on learning image representation by combining local and global features. However, the existing approaches still fail to address the bias issue towards the seen classes. This paper proposes implicit and explicit attention mechanisms to address the existing bias problem in generalised ZSL models. We formulate the implicit attention mechanism with a self-supervised image angle rotation task, which focuses on specific image features aiding in solving the task. On the other hand, the explicit attention mechanism is composed via the consideration of a multi-headed self-attention mechanism in the Vision Transformer model, which learns to attend important image locations and map global image features to semantic space during the training stage. We have conducted comprehensive experiments on three popular benchmarks: AWA2, CUB and SUN, where the effectiveness of our proposed attention mechanisms is shown in both discriminative and generative settings. Our extensive experiments show that our method has achieved state-of-the-art performance obtaining the highest harmonic mean on all three datasets, which is very encouraging to consider the ViT-based attention mechanisms for ZSL tasks in the future. [ABSTRACT FROM AUTHOR]

• A new video anomaly detection model via spatio-temporal relationships among objects. • An attention module makes the model focus on the spatio-temporal relationships. • A dynamic pattern generator is designed to memorize spatio-temporal relationships. • Extensive experiments show the effectiveness and advantage of the proposed method. Video anomaly detection is to automatically identify predefined anomalous contents (e.g. abnormal objects, behaviors and scenes) in videos. The performance of video anomaly detection can be effectively improved by making the model focus more on the anomalous objects in videos. However, such existing approaches usually rely on pre-trained models, which not only require additional auxiliary information but also face the challenge of anomaly diversity in the real world. In this paper, we propose a new video anomaly detection method based on spatio-temporal relationships among objects. Concretely, we use a fully convolutional encoder-decoder network with symmetric skip connections as the backbone network, which can effectively extract features from the object regions at different scales. In the encoding stage, an attention mechanism is used to enhance the model's understanding of the spatio-temporal relationships among various types of objects in the video. In the decoding stage, a dynamic pattern generator is designed to memorize the inter-object spatio-temporal relationships, which thus enhances the reconstructions of normal samples while making the reconstructions of abnormal samples more difficult. We conduct extensive experiments on three widely used video anomaly detection datasets CUHK Avenue , ShanghaiTech Campus and UCSD Ped2 , and the experimental results show that our proposed method can significantly improve the performance, and achieves state-of-the-art overall performance (considering both effectiveness and efficiency). In particular, our method achieves a state-of-the-art AUC of 98.4% on the UCSD Ped2 dataset that consists of various anomalies in real-world scenarios. [ABSTRACT FROM AUTHOR]

Entity relation extraction aims to identify the semantic relation between entity pairs in a sentence and is an important technical support for downstream tasks such as question answering systems and semantic searching. Existing relation extraction models mainly rely on neural networks to extract the semantic information of sentences, ignoring the critical role of important phrase information in relation extraction. For this problem, this paper proposes a relation extraction model based on BERT gated multi-window attention network (BERT-GMAN). The model first uses BERT to extract the semantic representation features of the sentence and its constraint information. Secondly, it constructs the key phrases extraction network to obtain multi-granularity phrase information and uses element-wise max pooling to select key phrases features. Thirdly, it adopts classification feature perception network to further filter and globally perceive key phrase feature to form the overall features of relation classification. Finally, it combines with Softmax classifier to perform relation extraction. The experimental results on the Semeval-2010 Task 8 dataset show that the performance of the model in this paper is further improved compared with the existing methods, and the F1-score reaches 90.25%. [ABSTRACT FROM AUTHOR]

A challenging problem that arises in few-shot intent detection is the complexity of multiple intention (multi-label) detection. The prototypical network uses the mean value of support instances as label prototype, which cannot eliminate the interference among features of multiple labels, making the learned label prototypes deviate from the real label features. Meanwhile, regardless of the correlation with the label prototype, all the feature dimensions in the query instance are treated equally, which reduces the accuracy of similarity measurement between the query instance and the label prototype. In this paper, we propose a hybrid calculation of correlation for few-shot multi-label intent detection method (HCC-FSML) to overcome the problem. This method proposes an instance-level attention mechanism to focus on the instance representation with high correlation between support instances and positive labels, so as to improve the consistency between the label prototype and the real label features; In the similarity measurement, the feature-level attention mechanism is introduced to focus on the feature dimensions of query instance with high correlation with positive label prototype, so as to improve the accuracy of similarity measurement. The experiment shows that the proposed method achieves new state-of-the-art results in intent detection by focusing on typical features of instances and reducing multiple interference among labels. [ABSTRACT FROM AUTHOR]

As a key technology for scene understanding, real-time semantic segmentation is an important topic in the field of computer vision in recent years. However, some lightweight networks designed for real-time semantic segmentation only have limited receptive field, so they cannot effectively perceive multi-scale objects in images. In addition, although the fast downsampling feature extraction network reduces the amount of computation, it has the problem of loss of detailed information, resulting in poor prediction accuracy. In this paper, we propose an efficient lightweight semantic segmentation network called BFMNet to address these issues. First, we use a lightweight bilateral structure to encode the semantic and detailed information from images respectively and introduce feature interactions during the encoding process. Furthermore, we design a novel Multi-Scale Context Aggregation Module (MSCAM) to help the network perceive the information of multi-scale objects, which is crucial for semantic segmentation. Finally, we introduce a new fusion module (AEFM) that uses attention perception to facilitate bilateral feature fusion. Our network achieves competitive results on three popular semantic segmentation benchmarks: Cityscapes, CamVid and COCO-Stuff. Specifically, on a single 2080Ti GPU, our network yields 77.7 % mIoU at 63.7 FPS on Cityscapes test set with the input resolution of 768 × 1536. Considering the speed-accuracy trade-off, we also report the results with 1024 × 2048 input resolution: 78.9 % mIoU at 31.4 FPS. On the Camvid test set, our network achieves 75.6 % mIoU at 95.8 FPS, while on the COCO-Stuff validation set, our network achieves 31.2 % mIoU. [ABSTRACT FROM AUTHOR]

Feature pyramid network (FPN) is constructed from a bottom-up pathway and a top-down pathway. The method involves multi-scale features, so it can obtain rich contextual information from lower scales and high resolution from the largest scale. Additionally, different receptive fields are effective to capture both thin and large objects in image scenes. All feature maps concatenate together to predict the targets. However, the average pooling method yields the problem of combining the best predictions with poorer ones. In this paper, we proposed a dual prediction to leverage the useful characteristics of each FPN feature map. A low scale prediction attains good precision for large objects. The other one suitably segments narrow objects. Finally, a multi-scale fusion is deployed with an attention part. The attention module finds pixels of a low scale having high probabilities of wrong labels, and then requires the supplements from a high scale. A multi-scale fusion allows the network to learn across the different scales of predictions. We have achieved good Results 77.9% mIoU at 62 FPS on Cityscapes and 44.1% mIoU on Mapillary Vistas. [ABSTRACT FROM AUTHOR]

Gated recurrent unit (GRU) is a variant of the recurrent neural network (RNN). It has been widely used in many applications, such as handwriting recognition and natural language processing. However, GRU can only memorize the sequential information, but lacks the capability of adaptively paying attention to important parts in the sequences. In this paper, we propose a novel RNN model, called recurrent attention unit (RAU), which can seamlessly integrate the attention mechanism into the interior of the GRU cell by adding an attention gate. The attention gate enhances the ability of RAU to remember long-term information and pay attention to important parts in the sequential data. Extensive experiments on adding problem, image classification, sentiment classification and language modeling show that RAU consistently outperforms GRU and other related models. [ABSTRACT FROM AUTHOR]

Pooling has been the core ingredient of modern convolutional neural networks (CNNs). Although classic pooling methods are simple and effective, it will inevitably lead to the problem that some features that make a great contribution to classification may be ignored. To solve this issue, this paper presents a novel cascade wavelet transform module, which makes full use of different frequency components and can be seamlessly integrated into the existing CNNs by replacing the existing pooling operation. In our method, wavelet transforms are performed in both spatial and channel domain. In spatial domain, using 2D discrete wavelet transform, we design a spatial pooling layer with attention mechanism by integrating low-frequency and high-frequency information. In channel domain, based on 1D discrete wavelet transform, a channel pooling layer with the attention mechanism is proposed for the final feature reconstruction. We call the proposed cascade wavelet transform based CNNs CasDWTNets. Compared to the traditional CNNs, experiments demonstrate that CasDWTNets obtain outstanding consistency and accuracy in image classification. Code will be made available. [ABSTRACT FROM AUTHOR]

Multimodal classification of social media is used to classify data from different modalities into different categories, which is essential for understanding user behavior on the web. In this paper, we focus on classifying image-text pairs, specifically user-generated content on social media. Recently, the transformer network, a kind of self-attention network, has been widely studied in the disciplines of visual computing and language processing. In the attention mechanism, positive correlation is considered. However, multimedia content posted on social media is diverse. Images and text are not always consistent, and contrary information is also helpful for representation. Therefore, it is equally important to detect conflicts based on negative or inverse attention. Inspired by the attention mechanism, we propose a novel model, namely Crossmodal Bipolar Attention Network (CBAN). Different from existing positive dot-product and additive attention mechanisms, we propose a bipolar attention mechanism, which fuses visual and textual information through their direct and inverse semantic relationships to classify multimodal data. We conducted experiments on multiple multimodal classification data sets, for performing sentiment analysis, sarcasm detection, crisis categorization and hate-speech detection. Experimental results show that our proposed CBAN consistently outperforms state-of-the-art methods in all classification tasks. [ABSTRACT FROM AUTHOR]

• A new 2D object detection method based on CenterNet is proposed. • An improved 2D object detection method based on RetinaNet is developed. • A novel 3D object detection method based on frustum-level point cloud is proposed. Object detection is a hot research topic in the communities of computer vision and intelligent transportation system. In this paper, we propose a novel framework for multi-level object detection by mutli-sensor perception from road scenes. Firstly, two types of 2D object detection methods are proposed based on multi-level feature processing. Among them, an improved 2D object detection method is designed based on CenterNet, which introduces a centripetal offset module and a deformable convolution module to improve the corner matching accuracy and solve the problem of missed detection. Moreover, an improved 2D object detection method based on RetinaNet is designed by the optimization of the sub-network of ResNet. The coordinate attention mechanism and bidirectional feature fusion mechanism are incorporated. Finally, we propose a 3D object detection method based on an improved PointNet network. The point cloud data are projected into a frustum according to the 2D detection result, then the frustum is segmented according to the linearly increased steps. The experimental results on the KITTI dataset and TSD-max dataset well demonstrate the effectiveness of the proposed framework. [ABSTRACT FROM AUTHOR]

The complex and varied urban road conditions have always been a difficult and a pivotal component in the study of driverless technology especially at intersections. In China, it is necessary for driverless cars to understand the gestures of traffic police. To identify the traffic police gesture at the intersection, the key step is to detect the traffic police at the intersection. At present, the research on traffic police detection is still in its infancy, there exists common problems such as slow detection speed and other real time problems in this method, and there is not a standardized traffic police data set ether. For the real time problems, this paper introduces the attention mechanism, and proposes a new real-time detection method of traffic police based on attention mechanism. The method proposed in this paper has strong robustness and can quickly complete the target detection task. For the data set problem, this paper analyzes and discloses similar data sets published through research. In the meanwhile, this paper collects 24,530 video data on the actual road, and it extracts and saves 12,000 pictures containing traffic police at frame rate of 30FPS as traffic police detection training and validated data set. [ABSTRACT FROM AUTHOR]

Subtle facial expression recognition (SFER) aims to classify facial expressions with very low intensity into corresponding human emotions. Subtle facial expression can be regarded as a special kind of facial expression, whose facial muscle movements are more difficult to capture. In the last decade, various methods have been developed for common facial expression recognition (FER). However, most of them failed to automatically find the most discriminative parts of facial expression and the correlation of muscle movements when human makes facial expression, which makes them unsuitable for SFER. To better solve SFER problem, an attention mechanism based model focusing on salient local regions and their correlations is proposed in this paper. The proposed method: 1) utilizes multiple attention blocks to attend to distinct discriminative regions and extract corresponding local features automatically, 2) a correlation attention module is integrated in the model to extract global correlation feature over the salient regions, and finally 3) fuses the correlation feature and local features in an efficient way for the final facial expression classification. By this way, the useful but subtle local information can be utilized in more detail, and the correlation of different local regions is also extracted. Extensive experiment on the LSEMSW and CK+ datasets shows that the method proposed in this paper achieves superior results, which demonstrates its effectiveness. [ABSTRACT FROM AUTHOR]

Recently, the attention mechanism became the key issue for image super-resolution (SR) because it has the ability to extract different features from the image according to the used attention type. Despite the great success of attention-based methods, the conflict among features of different attention types can affect the SR performance. In this paper, we propose an efficient single image SR model called balanced spatial feature distillation and pyramid attention (BSPAN). The idea of BSPAN is based on the trade-off among the extracted features of different attention types. Also, we propose a balanced spatial feature distillation block (BSFDB) as the backbone of BSPAN so that the network can effectively advantage from the different attention features. Two different attention types, namely spatial attention residual feature distillation (SARFD) and classical attention (CA) are considered in the BSFDB to achieve a leveling between them based on the contents of the low-resolution feature map using the balancing attention block. The BSFDB block is designed to improve the SR performance and has lightweight parameters and low computation complexity. Moreover, to further improve the SR performance, the pyramid attention is introduced in the middle of the BSPAN network for extracting long-range features at a variety of locations and scales. Evaluation based on five benchmark datasets, we concluded that the balancing between features of a variety of attention types can effectively improve the SR performance. So, the proposed BSPAN model achieves significant enhancements in comparison with the state-of-the-art and superior visual quality and reconstruction accuracy. [ABSTRACT FROM AUTHOR]

Sentence pair modeling is a fundamental yet challenging issue for feature mining in natural language processing (NLP) tasks. Recently, most works have generated feature and sentence representation based on the interactive attention mechanism. However, these models have two limitations: (1) they only consider global information through attention coefficient weighting, which makes insufficient utilization of critical features; (2) they only conduct internal training by fine-tuning network parameters, in which attention results are poorly explained. In this paper, inspired by human reasoning, we propose a Commonality Aggregated approach (CA) to enhance the lightweight interaction model by considering phrase features and contextual words. Specifically, we first fuse positional encoding and employ supervised training to extract critical phrase information from the text as the commonality of sentence pairs. Then, we deploy transfer learning and utilize interaction network to combine crucial phrase features, core word features, and positional encoding to enhance sentence pair modeling. Compared with the original network, extensive experiments on multiple benchmark datasets demonstrate the effectiveness of the proposed commonality aggregated method with stronger competitiveness. Further visual analysisanalysies validated the more explicit interpretability of attention, and extended experimental results indicate the excellent generalization of our approach. [ABSTRACT FROM AUTHOR]

• The first lightweight framework LDA-GAN is proposed for unpaired image-toimage translation with fewer parameters and less memory usage. • An improved domain-attention module (DAM) makes the generator learn to focus on the most relevant regions between two domains. • A novel separable-residual block (SRB) is introduced to retain spatial information and correct the details of the translation images. • Extensive experimental results prove the effectiveness of our proposed method on various datasets and other challenging style transfer tasks. Recently, image-to-image translation has attracted the interest of researchers, which purpose is to learn a mapping between two image domains. However, image translation will become an intrinsically ill-posed problem when given unpaired training data, that is, there are infinite mappings between two domains. Existing methods usually fail to learn a relatively accurate mapping, leading to poor quality of generated results. We believe that if the framework can focus more on the translation of important object regions instead of irrelevant information, such as background, then the difficulty of mapping learning will be reduced. In this paper, we propose a lightweight domain-attention generative adversarial network (LDA-GAN) for unpaired image-to-image translation, which has fewer parameters and lower memory usage. An improved domain-attention module (DAM) is introduced to establish a long-range dependency between two domains. Thus, the generator can focus more on the relevant regions to generate more realistic images. Furthermore, a novel separable-residual block (SRB) is designed to retain depth and spatial information during the translation with a lower computational cost. Extensive experiments show the effectiveness of our model on various image translation tasks according to qualitative and quantitative evaluation. [ABSTRACT FROM AUTHOR]

Colorization is an effective technology to improve the imaging quality of thermal infrared sensors, which is of great importance to environmental perception systems. Recently, colorization for thermal infrared images has realized obvious improvement with the development of deep learning. However, the performance of current designs is not adequate to achieve satisfied colorization quality. This is mainly because that thermal infrared image colorization requires the abilities of both color prediction and grayscale detail recovery, which poses severe challenges for feature extraction due to the poor quality of thermal infrared images. Furthermore, most of the current networks are based on generative adversarial network, suffering the risks of training instability and mode collapse. In this paper, we propose a high-quality colorization network for thermal infrared images. By leveraging hierarchical architecture, attention mechanism, and proposing a composite loss function, our network not only significantly improves the quality of colorized thermal infrared images but also achieves a speed of 32 fps on NVIDIA 2080Ti GPU platform. In evaluation experiments, PSNR and SSIM can be increased by at least 0.87 dB and 0.0277 compared with previous methods on publicly available datasets. Experimental results demonstrate that the proposed network can produce state-of-the-art colorized thermal infrared images both in objective metrics and subjective quality. [ABSTRACT FROM AUTHOR]

Recently, deep learning technology has achieved remarkable progress in compressed video quality enhancement. However, the existing methods fail to consider the fact that the regions with different compression losses contain varied effective information. To address this issue, this paper proposes a Compression Loss-based Spatial-Temporal Attention (CLSTA) module that predicts the compression loss-based attention of each pixel when restoring target frames. Based on this, the pixels with lower compression losses contribute more to restoring target frames. Meanwhile, the compression-domain information in the compressed video stream is exploited to overcome the difficulty in inferring compression loss-based attention directly from the pixel-domain. With a slight modification, the CLSTA module can be easily integrated into the existing methods for end-to-end training. A series of experiments have been conducted in this study to validate the effectiveness and generality of the CLSTA module for compressed video quality enhancement. By adding the CLSTA module to the state-of-the-art method STDF, the proposed method achieves an average PSNR improvement of 0.96 dB and a BD-rate reduction of 26.85% under the Low-Delay P configuration, which is respectively 0.06 dB and 1.83% better than the baseline STDF. In the best case, the proposed method obtains 0.19 dB higher PSNR than STDF for a single frame. [ABSTRACT FROM AUTHOR]

Pedestrian attribute recognition is a challenging task because of appearance variations, illumination variations, etc., in pedestrian images. We observe that two typical relations, i.e., relations of regions and relations of attributes, are beneficial to accomplish this task. In this paper, we explore the potential of Transformer on pedestrian attribute recognition task for the first time, and propose a Transformer framework called Dual-Relations Transformer (DRFormer). Vision Transformer (ViT) is adopted as a feature extractor for its nature of modeling long-range relations of regions. Furthermore, an Attribute Relation Module (ARM) is designed with Transformer encoder to capture relations of attributes. In ARM, we encode spatial information and semantic information of attributes into vector embedding representations. Being equipped with spatial information, DRFormer is capable of localizing attribute-related regions. Semantic information enables DRFormer to learn underlying semantic relations among attributes. Extensive experiments are conducted on three popular datasets, including PETA, PA-100K, and RAP, and demonstrate the superiority of our proposed DRFormer over state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Existing spectral super-resolution (SSR) methods have achieved satisfactory performance by designing complicated deep convolution neural networks (DCNNs) to extract spectral and spatial features. However, these methods ignore the fact that the significance of spatial and spectral information in each hyperspectral image (HSI) is different, and most of them directly fuse two kinds of information with concatenation and convolution operation, which resulting in generating redundant information and have negative effects on reconstruction. To address such inadequacies, this paper proposes a novel adaptive spatial–spectral modulation network (ASSM-Net). Specifically, we propose a new adaptive feature fusion module (AFFM) to replace traditional convolutional fusion schemes. Through explicitly measuring the weights of spatial information and spectral features of HSI, AFFM can select dominant features for each pixel to modulate spatial and spectral information. Additionally, we develop a pixel-weighted aware attention (PAA) mechanism to enhance the feature interdependences for recovering finer structure information. Finally, a large number of quantitative and qualitative experiments reveal that the proposed network achieves competitive reconstruction results on three benchmark datasets (NTIRE 2022, CAVE and Harvard). [ABSTRACT FROM AUTHOR]

Temporal knowledge graphs (TKGs) are still far from completeness although they have benefited many artificial intelligence applications successfully. To alleviate the incompleteness of TKGs, temporal knowledge graph completion (TKGC) aims to perform link prediction and reason new time-sensitive facts from existing facts. Among the various TKGC methods, temporal knowledge graph embedding (TKGE) methods represent entities, relations, and timestamps in low-dimensional spaces, which are known for their high reasoning performance and robust scalability. Some previous researches on TKGE aim to endow the embedding quadruple with the translational characteristic by modeling the global relationship in the embedding quadruple. However, the weights of same dimensional entries of the embedding quadruple are fixed and unique, which limits the ability to capture diverse global relationships. Besides, most TKGE models view timestamps as a kind of general feature and overlook their interrelations and correlations with the queries. Moreover, the distinct characteristics of an entity as the subject entity and the object entity of facts have not been fully leveraged. In this paper, we propose a query-aware embedding model based on the convolutional neural network (CNN) named ConvTKG to perform TKGE, thus tackling the TKGC task. We design a new temporal information encoder based on the Gated Recurrent Unit (GRU) and attention mechanism to learn the query-aware representation of temporal information. More importantly, we utilize a CNN-based decoder where multiple one-dimensional convolution kernels are operated on the matrix composed of entity embedding, relation embedding, and temporal representation to capture global relationships among them. In addition, we assign two independent vectors for each entity and take advantage of the inverse relation to allow them to be learned dependently. Experiments show that ConvTKG achieves better link prediction performance than previous state-of-the-art baseline methods for TKGC on three benchmark datasets: ICEWS14, ICEWS05-15, and GDELT. • We propose the first weight-adaptive TKGE model with translational characteristic. • We introduce a new query-aware timestamp encoding method for TKGC. • Two dependent embedding vectors for each entity capture its distinct semantics. • Experiments demonstrate the effectiveness of ConvTKG on three TKGC benchmarks. [ABSTRACT FROM AUTHOR]

Few-shot semantic segmentation is a newly developing and challenging computer vision task which aims to predict pixel-wise segmentation on the novel categories where only a few annotated samples are supplied. Because of the scarcity of the annotated novel class samples, the main obstacle of this issue is the diversity of objects in the support set and query set. This paper proposes a novel network aiming to bridge the gap by exploring the correlation between the support feature and the query feature. Specifically, a task-aware adaptive attention module(TAAM) is introduced to extract the task-specific information from the current input and integrates it into the feature representations both in channel dimension and spatial dimension for adaptive reinforcement. Besides, an additional prediction refinement module(RPM) is attached to further optimize the predictions to present more details of objects. Furthermore, through a non-parameter aggregation operation, the proposed network is easy to generalize to k-shot segmentation without developing specific architectures. Extensive experiments on three benchmarks demonstrate that our method exceeds previous state-of-the-arts with a sizable margin, verifying the effectiveness of the proposed method. [ABSTRACT FROM AUTHOR]

In recent years, computed tomography (CT) has been widely used in various clinical diagnosis. Given potential health risks bring by the X-ray radiation, the major objective of the current research is to achieve high-quality CT imaging while reducing X-ray radiation. However, most existing studies on low-dose CT image super-resolution reconstruction do not focus on the interaction between the denoising task and the super-resolution task. In this paper, we propose a dual-channel joint learning framework to accurately reconstruct high-resolution CT images from low-resolution CT images. Unlike the previous cascaded models which directly combine the denoising network and the super-resolution network, our method can process the denoising reconstruction and the super-resolution reconstruction in parallel. Additionally, we design a filter gate module that can filter features from the denoising branch and highlight important features which can benefit the super-resolution task. We evaluate the performance of our method in medical image enhancement by testing on the 2016 Low-Dose CT Grand Challenge dataset and the piglet dataset. The experimental results show that the proposed network is superior to other state-of-the-art methods in terms of both peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM). We also demonstrate that our method can better remove noise and recover details. Furthermore, the method achieves competitive results not only for super-resolution reconstruction of low-dose CT, but also for super-resolution reconstruction of sparse-view CT. [ABSTRACT FROM AUTHOR]

• A frequency-aware cross-modality attention network (FCMNet) is proposed for RGB-D SOD. • The frequency-aware cross-modality attention (FACMA) module and the weighted cross-modality fusion (WCMF) module are presented. • We evaluate the proposed methods with 17 state-of-the-art approaches on eight widely used datasets. RGB-D saliency detection aims to comprehensively use RGB images and depth maps to detect object saliency. This field still faces two challenges: 1) how to extract representative multimodal features and 2) how to effectively fuse them. Most of the previous methods in this field equally treat RGB and depth information as two modalities, while not considering the difference in the frequency domain of the two modalities, and may lose some complementary information. In this paper, we introduce the frequency channel attention mechanism into the fusion process. First, we design a frequency-aware cross-modality attention (FACMA) module to interweave adequate channel features and select representative features. In the FACMA module, we also propose a spatial frequency channel attention (SFCA) module to introduce more complementary information in different channels. Second, we develop a weighted cross-modality fusion (WCMF) module to adaptively fuse multimodality features by learning the content-dependent weight maps. Comprehensive experiments on several benchmark datasets demonstrate that the proposed framework outperforms seventeen state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Accurate traffic prediction is critical for enhancing the performance of intelligent transportation systems. The key challenge to this task is how to properly model the complex dynamics of traffic while respecting and exploiting both spatial and temporal heterogeneity in data. This paper proposes a novel framework called Meta Graph Transformer (MGT) to address this problem. The MGT framework is a generalization of the original transformer, which is used to model vector sequences in natural language processing. Specifically, MGT has an encoder-decoder architecture. The encoder is responsible for encoding historical traffic data into intermediate representations, while the decoder predicts future traffic states autoregressively. The main building blocks of MGT are three types of attention layers named Temporal Self-Attention (TSA), Spatial Self-Attention (SSA), and Temporal Encoder-Decoder Attention (TEDA), respectively. They all have a multi-head structure. TSAs and SSAs are employed by both the encoder and decoder to capture temporal and spatial correlations. TEDAs are employed by the decoder, allowing every position in the decoder to attend all positions in the input sequence temporally. By leveraging multiple graphs, SSA can conduct sparse spatial attention with various inductive biases. To facilitate the model's awareness of temporal and spatial conditions, Spatial–Temporal Embeddings (STEs) are learned from external attributes, which are composed of temporal attributes (e.g. sequential order, time of day) and spatial attributes (e.g. Laplacian eigenmaps). These embeddings are then utilized by all the attention layers via meta-learning, hence endowing these layers with Spatial–Temporal Heterogeneity-Aware (STHA) properties. Experiments on three real-world traffic datasets demonstrate the superiority of our model over several state-of-the-art methods. Our code and data are available at (http://github.com/lonicera-yx/MGT). [ABSTRACT FROM AUTHOR]

Crowd counting that aims to estimate the crowd density has recently made significant progress but remains an unsolved problem due to several challenges. In this paper, we propose an Attentive Encoder-Decoder Network (AEDNet) to overcome the notorious scale-variation problem in crowd counting. Our major contributions can be summarized in three aspects. First, we design an Attentive Feature Refinement (AFR) block in the encoder to adaptively extract multi-scale features. AFR compares the spatial information in different scales through the attention mechanism and then adaptively assign importance weights to each point, which highlights the distinctive roles in multi-scale feature extraction. Second, we develop a Separable Non-local Fusion (SNF) block in the decoder with the self-attention mechanism to aggregate multi-scale features from different layers, which not only achieves the sufficient feature fusion by capturing long-range dependencies, but also vastly reduces the computation cost compared to the original non-local operation. Third, we propose a Regional MSE (R-MSE) loss to tackle the pixel-isolation problems in regular MSE loss. To demonstrate the effectiveness of the proposed AEDNet, we conduct extensive experiments on four widely-used crowd counting datasets, and our AEDNet consistently achieves the state-of-the-art performance. [ABSTRACT FROM AUTHOR]

[Display omitted] • A steering angle prediction method is proposed for autonomous ego-vehicles, based on an end-to-end deep learning algorithm. • A temporal and spatial attention block is designed to better capture long- and short-term memory and visual information. • The experimental results performed on the comma2k19 dataset are better than the baselines. • An attention mechanism is visualized to explain the trigger mechanism of steering behavior prediction. In recent years, autonomous vehicles have attracted many researchers from academia and industry. Their efforts include object detection and tracking in the fields of computer vision, probabilistic-fusion and decision-making algorithms, etc. But most current methods that directly map from image pixels to steering behavior are not able to generate accurate results, especially under scenarios such as light and weather change drastically; this is largely due to the fact that these methods ignore the temporal relationship between frames. In this paper, we propose a novel end-to-end deep learning framework based on temporal and spatial attention mechanism, which aims to solve the problem of inaccurate vehicle steering angle prediction in complex environments and the difficulty of model interpretation. First, we use video sequence and historical steering angle sequence as inputs to the model, instead of just using a single frame as input. Second, we design a temporal attention mechanism to capture the long- and short-term memory in input visual information, and a spatial attention mechanism to capture key objects in the image and obtain their position information. This is achieved by inserting carefully-designed SE-Net, ConvLSTM and CNN layers into the appropriate layers of the network framework. Finally, we demonstrate the feasibility of the proposed model with public Comma2k19 , with comparison to current advanced methods. Experimental results show that compared with state-of-the-art methods, the average absolute error (MAE) values of our model on the training set and testing set are reduced by 10.2% and 6.3%, respectively, and has more accurate steering prediction performance. In addition, we explain the trigger mechanism of steering behavior prediction by visualizing the spatial attention map and temporal attention score on Comma2k19 and Udacity datasets, which further demonstrates that the proposed model can learn human-like driving behavior. [ABSTRACT FROM AUTHOR]

Community question answering (CQA) is a popular platform where users can ask questions or solve the questions proposed by other users. The expert recommendation aims at providing high-quality answers for the newly proposed questions in time, which is the key to a successful CQA. Questions in CQA usually consist of two parts, a subject which describes the main point, and a body which gives the details of the question. In previous studies, researchers usually ignore the differences between the subject and the body and concatenate them as a whole. In this paper, we propose a multi-hop interactive attention based classification network (MIACN) to recommend experts for newly proposed questions. In our model, the subject and the body are seen as two separate parts. A multi-hop attention is used to capture the multiple latent interactions among the two parts. Then, a high-level representation of the question is generated from the interactions. Experiment results on two real-world datasets demonstrate the effectiveness of our model. [ABSTRACT FROM AUTHOR]

Visual dialog is a task that two agents complete a multi-round conversation based on an image, a caption, and dialog histories. Despite the recent progress, existing methods still undergo degradation on the condition of complex scenarios. Handling these scenarios depends on logical reasoning that requires commonsense priors. In this paper, we propose a novel visual dialog pipeline named Structured Knowledge-Aware Network (SKANet), consisting of an Image Knowledge-Aware Module and a Caption Knowledge-Aware Module. Specifically, the Image and Caption Knowledge-Aware Modules construct commonsense knowledge graphs from ConceptNet. We apply SKANet to two sub-tasks: the conventional visual dialog and a goal-oriented visual dialog named 'image guessing'. For the conventional visual dialog, the SKANet is combined with an additional Multi-Modality Fusion Module, which is designed to explore the visual content and the textual context about the dialog history. For the goal-oriented visual dialog, we directly apply the Image and Caption Knowledge-Aware Modules to two agents, respectively. Experimental results on VisDial v0.9 and VisDial v1.0 datasets show that our proposed method effectively outperforms comparative methods on both sub-tasks. [ABSTRACT FROM AUTHOR]

• Based on the latest research progress of GAN, this paper proposes ESA-CGAN, for image data simulation application. • (1) The attention mechanism and object saliency model are integrated to learn the global information and object characteristics of the image. • (2) A cooperative learning mechanism between generators is designed to combine self-constructed goal generation model together with several pre-trained generation models thus forming a generation model group which can learn from each other and retain global similarity while having local differences. In order to address such negative issues as GAN's mediocre image quality, high-demand for training samples and computation resources, this paper proposes the collaborative Generative Adversarial Network with Visual perception and memory reasoning (ESA-CGAN). This not only makes use of the vision self-attention mechanism and objects salience model to analyze the global information and detailed features of objects, but also designs cross-correlation self-attention module so as to make a balance between the computation efficiency and computational on the one hand and the statistical efficiency and the ability to simulate remote dependencies on the other hand. Based on convolutional long-term and short-term memory network, this paper optimizes the attention feature map so as to highlight the features of objects themselves and improve their generative abilities. Meanwhile, a cooperative learning mechanism between generators is designed, which combines self-constructed generation model and pre-training generation model to form a generation model group. It not only improves effectively the generation ability of the model, improve the computing efficiency, but also restrains the collapse of the model from another angle. In fact, the model proposed here has completed numerical experiments on multiple common standard datasets and self-configuring datasets, and has made comparisons with several mainstream generation antagonistic network models in terms of the performance of image data augmentation. The experimental results demonstrate that the model has excellent simulation ability to enable itself to effectively realize the augmentation of data, thus making it highly applicable in the future. [ABSTRACT FROM AUTHOR]

As a dense prediction problem, semantic segmentation consumes extensive memory and computational resources. However, the application of semantic segmentation requires the model to perform real-time analyses in portable devices, thus it is crucial to seek a trade-off between segmentation accuracy and inference speed. In this paper, we propose a lightweight semantic segmentation method based on attention mechanism to address this problem. First, we use novel Efficient Bottleneck Residual (EBR) Module and Efficient Asymmetric Bottleneck Residual (EABR) Module to extract both local and contextual information,which adopt a well-designed combination of depth-wise convolution, dilated convolution and factorized convolution, with channel shuffle to boost information interaction. Second, we introduce attention mechanism into skip connection between the encoder and decoder to promote reasonable fusion of high-level and low-level features, which furtherly enhance the accuracy. With only 1.9 M parameters, our model obtains 74.8% mIoU and 64 FPS running speed on Cityscapes dataset and 71.2% mIoU and 79 FPS running speed on Camvid dataset. Experiments demonstrate that our model achieves competitive results in terms of segmentation accuracy and running speed while controlling parameters. [ABSTRACT FROM AUTHOR]

In this paper, we concentrate on the text-to-image synthesis task that aims at automatically producing perceptually realistic pictures from text descriptions. Recently, several single-stage methods have been proposed to deal with the problems of a more complicated multi-stage modular architecture. However, they often suffer from the lack-of-diversity issue, yielding similar outputs given a single textual sequence. To this end, we present an efficient and effective single-stage framework (DiverGAN) to generate diverse, plausible and semantically consistent images according to a natural-language description. DiverGAN adopts two novel word-level attention modules, i.e., a channel-attention module (CAM) and a pixel-attention module (PAM), which model the importance of each word in the given sentence while allowing the network to assign larger weights to the significant channels and pixels semantically aligning with the salient words. After that, Conditional Adaptive Instance-Layer Normalization (CAdaILN) is introduced to enable the linguistic cues from the sentence embedding to flexibly manipulate the amount of change in shape and texture, further improving visual-semantic representation and helping stabilize the training. Also, a dual-residual structure is developed to preserve more original visual features while allowing for deeper networks, resulting in faster convergence speed and more vivid details. Furthermore, we propose to plug a fully-connected layer into the pipeline to address the lack-of-diversity problem, since we observe that a dense layer will remarkably enhance the generative capability of the network, balancing the trade-off between a low-dimensional random latent code contributing to variants and modulation modules that use high-dimensional and textual contexts to strength feature maps. Inserting a linear layer after the second residual block achieves the best variety and quality. Both qualitative and quantitative results on benchmark data sets demonstrate the superiority of our DiverGAN for realizing diversity, without harming quality and semantic consistency. [ABSTRACT FROM AUTHOR]

Named Entity Recognition(NER), one of the most fundamental problems in natural language processing, seeks to identify the boundaries and types of entities with specific meanings in natural language text. As an important international language, Chinese has uniqueness in many aspects, and Chinese NER (CNER) is receiving increasing attention. In this paper, we give a comprehensive survey of recent advances in CNER. We first introduce some preliminary knowledge, including the common datasets, tag schemes, evaluation metrics and difficulties of CNER. Then, we separately describe recent advances in traditional research and deep learning research of CNER, in which the CNER with deep learning is our focus. We summarize related works in a basic three-layer architecture, including character representation, context encoder, and context encoder and tag decoder. Meanwhile, the attention mechanism and adversarial-transfer learning methods based on this architecture are introduced. Finally, we present the future research trends and challenges of CNER. [ABSTRACT FROM AUTHOR]

The voluminous quantity of data accessible on social media platforms offers insight into the sentiment disposition of individual users, where multi-modal sarcasm detection is often confounding. Existing sarcasm detection methods use different information fusion methods to combine information from different modalities but ignore hidden information within modalities and inconsistent information between modalities. Discovering the implicit information within the modalities and strengthening the information interaction between modalities is still an important challenge. In this paper, we propose a Multi-Channel Enhanced Fusion (MCEF) model for cross-modal sarcasm detection to maximize the information extraction between different modalities. Specifically, text extracted from images acts as a new modality in the front-end fusion models to augment the utilization of image semantic information. Then, we propose a novel bipolar semantic attention mechanism to uncover the inconsistencies among different modal features. Furthermore, a decision-level fusion strategy from a new perspective is devised based on four models to achieve multi-channel fusion, each with a distinct focus, to leverage their advantages and mitigate the limitations. Extensive experiments demonstrate that our model surpasses current state-of-the-art models in multi-modal sarcasm detection. [ABSTRACT FROM AUTHOR]

Object-level features from Faster R-CNN and attention mechanism have been used extensively in image captioning based on Encoder-Decoder frameworks. However, most existing methods feed the average pooling of object features as the global representation to the captioning model and recalculate the attention weights of object regions when generating a new word without considering the visual persistence like humans. In this paper, we respectively build Visual Persistence modules in encoder and decoder: The visual persistence module in encoder seeks the core object features to replace the image global representation; the visual persistence module in decoder evaluates the correlation between previous attention results and current attention results, and fuses them as the final attended feature to generate a new word. The experimental results on MSCOCO validate the effectiveness and competitiveness of our Visual Persistence Model (VPNet). Remarkably, VPNet also achieves competitive scores in most metrics on MSCOCO online test server compared to the existing state-of-the-art methods. [ABSTRACT FROM AUTHOR]

The technology for Visual Odometry (VO) that estimates the position and orientation of the moving object through analyzing the image sequences captured by on-board cameras, has been well investigated with the rising interest in autonomous driving. This paper studies monocular VO from the perspective of Deep Learning (DL). Unlike most current learning-based methods, our approach, called DeepAVO, is established on the intuition that features contribute discriminately to different motion patterns. Specifically, we present a novel four-branch network to learn the rotation and translation by leveraging Convolutional Neural Networks (CNNs) to focus on different quadrants of optical flow input. To enhance the ability of feature selection, we further introduce an effective channel-spatial attention mechanism to force each branch to explicitly distill related information for specific Frame to Frame (F2F) motion estimation. Experiments on various datasets involving outdoor driving and indoor walking scenarios show that the proposed DeepAVO outperforms the state-of-the-art monocular methods by a large margin, demonstrating competitive performance to the stereo VO algorithm and verifying promising potential for generalization. [ABSTRACT FROM AUTHOR]

As the main research content of artificial intelligence, the artificial neural network has been widely concerned because of its excellent performance in the fields such as computer vision and natural language processing since it was proposed in the 1940s. The neuron model of the traditional neural network was proposed by McCulloch and Pitts in 1943 (MP neurons), But MP neurons is too simple to representing biological neurons. Based on this, this paper studies the attention mechanism and proposes vectorized neuron and its activation function. Firstly, we propose vectorized neurons, then use the attention mechanism to dynamically generate connection weights between vectorized neurons. Nextly, we construct a new type of neural network with vectorized neurons, which we called neural functional group (NFG). Finally, we tested the proposed neural functional group model on two tasks: image classifcation and few-shot learning. The vectorized neuron can be conditionally activated through its activation function. Besides, the vectorized neuron has the potential of representing complex biological neurons, which is difficult for MP neuron. The experimental results show that it can achieve higher accuracy with fewer parameters than convolutional neural networks (CNN) and capsule networks in image classication task; it also competitive to CNN based feature extractor in few-shot learning task. [ABSTRACT FROM AUTHOR]

Online multi-object tracking is a process of extending multi-object trajectories with only past information. In this process, tracking drift, missing detection, and occlusion among objects are common problems. To address the problem of tracking drift, in this paper, a detector–tracker integration framework is proposed and the framework includes the linear regression model (LRM) that can detect and track objects simultaneously. The two kinds of results are combined using LRM to suppress tracking drift. Moreover, to overcoming missing detection and the occlusion, a structural similarity calculation based on the attention mechanism is proposed. The attention mechanism is utilized to learn the discriminative features of each object, and a structural similarity calculation are used to improve the ability to extend the objects' trajectories. Based on these, we design a multi-object tracking strategy, enabling the trajectory's initialization, extension, and termination. Finally, experiments and analysis are executed on MOT16 and MOT17 benchmarks datasets, and the proposed method obtains multi-object tracking accuracy of 60%. [ABSTRACT FROM AUTHOR]

Visible-infrared person re-identification (VI-ReID) aims at matching people across images from RGB and infrared modality. Existing methods tend to utilize the readily available models to extract features, ignoring mining spatial and channel information of features. In this paper, we propose an attentive modality-consistent network (AMC-Net) for VI-ReID. Firstly, to avoid the damage to discriminability of features caused by overfitting of local regions in the network, a context-aware attention block (CAB) is designed to mine the spatial information of the whole person region by magnifying the perception scope of convolution layers. Secondly, the attentive channel aggregation block (ACB) is adopted to mine the channel information with richer semantic cues by considering local cross-channel interactive information. Thirdly, we propose a modality-consistent regularizer to narrow down the discrepancy of high order features between heterogeneous images. Extensive experiments on two datasets indicate that the proposed method outperforms the state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Convolutional neural networks (CNNs) have been achieving remarkable results in medical image segmentation. However, for accurate segmentation of subcortical brain structure in MR images, it is still a challenge due to the ambiguous boundaries, the complex structures, and the various shapes, which limits their clinical application. In this paper, we focus on utilizing multi-scale image contexts and attention mechanisms to improve networks' ability to learn discriminative feature representation for accurate segmentation and present a novel FCNN architecture called multis-scale guided squeeze-and-excitation network (MSGSE-Net). In particular, we first propose the multi-scale guided squeeze-and-excitation (MSGSE) attention module which can progressively and selectively aggregate discriminative features. In contrast to existing attention modules, the MSGSE module performs an adaptive recalibration that features at different locations of the feature map are recalibrated under the guidance of multi-scale contexts. Then multi-scale spatial attention supervision is adopted to enhance the intra-class homogeneity and inter-class distinction of the attention weights. Moreover, we propose a novel entropy-weighted Dice loss (EDL) to force the network to focus on the ambiguous voxels around the boundaries of subcortical structures. We evaluate the proposed method on two challenging benchmark datasets (the IBSR dataset and the MALC dataset). The experimental results show that our model consistently yields better segmentation performance than several state-of-the-art methods and improves the segmentation Dice score by 1.6% at most compared with baseline method U-Net. Our code is available at https://github.com/neulxlx/MSGSE-Net. [ABSTRACT FROM AUTHOR]

For the multi-label text classification problems with many classes, many existing multi-label classification algorithms become infeasible or suffer an unaffordable cost. Some researches hence perform the Label Space Dimension Reduction(LSDR) to solve this problem, but a number of methods ignore the sequence information of texts and the label correlation in the original label space, and treat the label as a meaningless multi-hot vector. In this paper, we put forward a multi-label text classification algorithm LELC(joint learning from L abel E mbedding and L abel C orrelation) based on the multi-layer attention and label correlation to solve the issue of multi-label text classification with a large number of class labels. Specifically, we firstly extract features through Bidirectional Gated Recurrent Unit Network(Bi-GRU), multi-layer attention and linear layers. Bi-GRU will capture the content information and sequence information of the text at the same time, and the attention mechanism can help us select the valid features related to labels. Then, we use matrix factorization to perform LSDR, and consider label correlation of the original label space in this process, which allows us to implicitly encode the latent space and simplify the model learning. Finally, Deep Canonical Correlation Analysis(CCA) technology is exploited to couple features and the latent space in an end-to-end pattern, so that these two can influence each other to learn the mapping of feature space to latent space. Experiments on 11 real-world datasets show the comparability between our proposed model and the state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Thyroid cancer is the most common form of endocrine malignancy, and the informative pathological images are critical for thyroid cancer risk stratification, prognosis and treatment guidance. Deep learning methods have achieved promising results on pathology image classification benchmarks. However, due to the complexity of thyroid carcinoma pathological images and the lack of labeled data, there are few researches study on the auto-classification of thyroid cancer. Inspired by the diagnostic process of pathologists, this paper proposes an active classification method for papillary thyroid carcinoma (PTC) pathological images classification to divide thyroid pathological images into PTC and normal thyroid pathological images. We employ the attention mechanism to combine pathological images with different magnification factors, which imitates the diagnosis procedures of thyroid cancer under the microscope. At the same time, we utilize the uncertainty and representative information provided by the convolutional neural network to determine the most valuable samples for annotation that can reduce the labeling cost. Besides, a pathological image dataset named VIP-TCHis is conducted from thyroid tissues of 55 real cases. The experimental results show that our method can obtain good performance of PTC recognition on the VIP-TCHis dataset. [ABSTRACT FROM AUTHOR]

Person image generation becomes a challenging problem due to the content ambiguity and style inconsistency. In this paper, we propose a novel Attention-based Injection Network (AIN) to address this issue. Instead of directly learning the relationship between the source and target image, we decompose the process into two accessible modules, namely Semantic-guided Attention Network (SAN) and Pose-guided Attention Network (PAN). SAN is proposed to capture the semantic information which can embed the human attributes into the latent space via the semantic layout. PAN enables a natural re-coupling of the pose and appearance, which can selectively integrate features to complete the human pose transformation. Additionally, a semantic layout loss is proposed to focus on the semantic content similarity between the source and generated images. Compared with other methods, our networks can enforce the local textures and styles consistency between the source and generated image. Experiments show that superior both qualitative and quantitative results are obtained on Market-1501 and DeepFashion datasets. On the basis of AIN, our network can further achieve the data augmentation for person re-identification (Re-ID) with dramatically improving the person Re-ID accuracy. [ABSTRACT FROM AUTHOR]

For the multi-label text classification problems with many classes, many existing multi-label classification algorithms become infeasible or suffer an unaffordable cost. Some researches hence perform the Label Space Dimension Reduction(LSDR) to solve this problem, but a number of methods ignore the sequence information of texts and the label correlation in the original label space, and treat the label as a meaningless multi-hot vector. In this paper, we put forward a multi-label text classification algorithm LELC(joint learning from L abel E mbedding and L abel C orrelation) based on the multi-layer attention and label correlation to solve the issue of multi-label text classification with a large number of class labels. Specifically, we firstly extract features through Bidirectional Gated Recurrent Unit Network(Bi-GRU), multi-layer attention and linear layers. Bi-GRU will capture the content information and sequence information of the text at the same time, and the attention mechanism can help us select the valid features related to labels. Then, we use matrix factorization to perform LSDR, and consider label correlation of the original label space in this process, which allows us to implicitly encode the latent space and simplify the model learning. Finally, Deep Canonical Correlation Analysis(CCA) technology is exploited to couple features and the latent space in an end-to-end pattern, so that these two can influence each other to learn the mapping of feature space to latent space. Experiments on 11 real-world datasets show the comparability between our proposed model and the state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Person image generation becomes a challenging problem due to the content ambiguity and style inconsistency. In this paper, we propose a novel Attention-based Injection Network (AIN) to address this issue. Instead of directly learning the relationship between the source and target image, we decompose the process into two accessible modules, namely Semantic-guided Attention Network (SAN) and Pose-guided Attention Network (PAN). SAN is proposed to capture the semantic information which can embed the human attributes into the latent space via the semantic layout. PAN enables a natural re-coupling of the pose and appearance, which can selectively integrate features to complete the human pose transformation. Additionally, a semantic layout loss is proposed to focus on the semantic content similarity between the source and generated images. Compared with other methods, our networks can enforce the local textures and styles consistency between the source and generated image. Experiments show that superior both qualitative and quantitative results are obtained on Market-1501 and DeepFashion datasets. On the basis of AIN, our network can further achieve the data augmentation for person re-identification (Re-ID) with dramatically improving the person Re-ID accuracy. [ABSTRACT FROM AUTHOR]

Thyroid cancer is the most common form of endocrine malignancy, and the informative pathological images are critical for thyroid cancer risk stratification, prognosis and treatment guidance. Deep learning methods have achieved promising results on pathology image classification benchmarks. However, due to the complexity of thyroid carcinoma pathological images and the lack of labeled data, there are few researches study on the auto-classification of thyroid cancer. Inspired by the diagnostic process of pathologists, this paper proposes an active classification method for papillary thyroid carcinoma (PTC) pathological images classification to divide thyroid pathological images into PTC and normal thyroid pathological images. We employ the attention mechanism to combine pathological images with different magnification factors, which imitates the diagnosis procedures of thyroid cancer under the microscope. At the same time, we utilize the uncertainty and representative information provided by the convolutional neural network to determine the most valuable samples for annotation that can reduce the labeling cost. Besides, a pathological image dataset named VIP-TCHis is conducted from thyroid tissues of 55 real cases. The experimental results show that our method can obtain good performance of PTC recognition on the VIP-TCHis dataset. [ABSTRACT FROM AUTHOR]

In recent years, deep learning techniques have significantly improved the performance of single image super-resolution (SISR). However, this improvement is often achieved at the cost of introducing a large amount of parameters, which limits the real-world applications for SISR. In this paper, we propose a lightweight SISR network called Multi-scale Channel Attention Network for Image Super-Resolution (MCSN). Our contributions are threefold. First of all, the multi-scale feature fusion block (MSFFB) can extract multi-scale features by filters with different receptive fields. Secondly, the channel shuffle attention mechanism (CSAM) encourages the flow of the information across feature channels and enhances the ability of feature selection. Thirdly, the global feature fusion connection (GFFC) can effectively improve feature utilization. Extensive experiments demonstrate that the parameter amount of our method is reduced by 3/4 compared with the current state-of-the-art MSRN method, while both the subjective visual effect and objective quality of the reconstructed high-resolution images are significantly better. [ABSTRACT FROM AUTHOR]

In text classification models based on deep learning, feature extraction and feature aggregation are two key steps. As one of the basic feature extraction methods, CNN has certain limitations due to its inability to effectively extract temporal features from text data. Using max-pooling can significantly reduce the amount of calculation while performing feature aggregation, but it will have an adverse effect on the classification results due to the loss of some text features. In this paper, in response to the above two issues, a Bidirectional Gated Temporal Convolutional Attention(BG-TCA) model is proposed. In the feature extraction stage, the BG-TCA model uses the bidirectional TCN to extract the bidirectional temporal features in text data, and a gating mechanism similar to the LSTM is added between the convolution layers. In the feature aggregation stage, the BG-TCA model uses the attention mechanism to replace the max-pooling method, which makes it possible to distinguish the importance of different features while retaining the text features to the maximum. Finally, experimental results on five benchmark datasets show that the classification accuracy of the BG-TCA model has been greatly improved compared to basic models, and is better than several other state-of-the-art models. [ABSTRACT FROM AUTHOR]