Search

Domain adaptation, which transfers the knowledge from label-rich source domain to unlabeled target domains, is a challenging task in machine learning. The prior domain adaptation methods focus on pairwise adaptation assumption with a single source and a single target domain, while little work concerns the scenario of one source domain and multiple target domains. Applying pairwise adaptation methods to this setting may be suboptimal, as they fail to consider the semantic association among multiple target domains. In this work we propose a deep semantic information propagation approach in the novel context of multiple unlabeled target domains and one labeled source domain. Our model aims to learn a unified subspace common for all domains with a heterogeneous graph attention network, where the transductive ability of the graph attention network can conduct semantic propagation of the related samples among multiple domains. In particular, the attention mechanism is applied to optimize the relationships of multiple domain samples for better semantic transfer. Then, the pseudo labels of the target domains predicted by the graph attention network are utilized to learn domain-invariant representations by aligning labeled source centroid and pseudo-labeled target centroid. We test our approach on four challenging public datasets, and it outperforms several popular domain adaptation methods.

Facial expression recognition (FER) is crucial for social communication. However, current studies present limitations when addressing facial expression difference due to demographic variation, e.g., race, gender, and age, etc. In this study, we first propose a deeply-supervised attention network (DSAN) to recognize human emotions based on facial images automatically. Based on DSAN, a two-stage training scheme is designed, taking full advantage of the race/gender/age-related information. In our DSAN framework, multi-scale features are leveraged to capture more discriminative information from the deep layers to the shallow layers. Furthermore, we adopt the attention block to highlight the essential local facial characteristics; it performs well when it is incorporated into the deeply-supervised framework. Finally, we combine the complementary characteristics of multiple convolutional layers in deeply-supervised manner and ensemble the intermediate predicted scores. Our experimental results have shown that our proposed framework can (i) effectively integrate demographic information in improving the performance of a variety of FER tasks, (ii) learn informative feature representations with a visual explanation by capturing the regions of interests (ROI), (iii) achieve superior performance for both the posed and the spontaneous FER databases, each containing pictures of human facial expressions varied in gender, age and race.

Many CNN-based segmentation methods have been applied in lane marking detection recently and gain excellent success for a strong ability in modeling semantic information. Although the accuracy of lane line prediction is getting better and better, lane markings' localization ability is relatively weak, especially when the lane marking point is remote. Traditional lane detection methods usually utilize highly specialized handcrafted features and carefully designed postprocessing to detect the lanes. However, these methods are based on strong assumptions and, thus, are prone to scalability. In this work, we propose a novel multitask method that: 1) integrates the ability to model semantic information of CNN and the strong localization ability provided by handcrafted features and 2) predicts the position of vanishing line. A novel lane fitting method based on vanishing line prediction is also proposed for sharp curves and nonflat road in this article. By integrating segmentation, specialized handcrafted features, and fitting, the accuracy of location and the convergence speed of networks are improved. Extensive experimental results on four-lane marking detection data sets show that our method achieves state-of-the-art performance.

In recent years, deep learning techniques have significantly improved the performance of single image super-resolution (SISR). However, this improvement is often achieved at the cost of introducing a large amount of parameters, which limits the real-world applications for SISR. In this paper, we propose a lightweight SISR network called Multi-scale Channel Attention Network for Image Super-Resolution (MCSN). Our contributions are threefold. First of all, the multi-scale feature fusion block (MSFFB) can extract multi-scale features by filters with different receptive fields. Secondly, the channel shuffle attention mechanism (CSAM) encourages the flow of the information across feature channels and enhances the ability of feature selection. Thirdly, the global feature fusion connection (GFFC) can effectively improve feature utilization. Extensive experiments demonstrate that the parameter amount of our method is reduced by 3/4 compared with the current state-of-the-art MSRN method, while both the subjective visual effect and objective quality of the reconstructed high-resolution images are significantly better.

Person re-identification (Re-ID) is of important capability for artificial intelligence and human–computer interaction. The main challenge of person Re-ID lies in limited data to precisely capture a wide range of appearance variations over multiple viewpoints. Furthermore, compared to the Re-ID between the single person, the person groups contain information about the relationship between pedestrians that can potentially help identify certain identities. The Re-ID combining groups and individuals remain to be a promising task under rare study. In this paper, we propose a harmonious attention network for person re-identification, in which we jointly consider the complementarity between person groups and individuals. Concretely, first we propose a two-stream attentive network (TSAN) to respectively learn the information from the person groups and individuals. TSAN consists of a spatial–temporal fusion network for the group Re-ID, as well as a deep network for the traditionally individual person Re-ID. To jointly consider the contributions of the groups and individuals, then we propose a novel re-ranking algorithm (GIRK) based on the learned features to associate the group and individual information. We also propose a new group Re-ID dataset DukeGroupVid to evaluate the performance of our approach. Comprehensive experimental results on the proposed dataset and other Re-ID datasets demonstrate the effectiveness of our model.

In this paper, we propose an attention-guided multi-scale fusion network (named as AMS-Net) for crowd counting in dense scenarios. The overall model is mainly comprised by the density and the attention networks. The density network is able to provide a coarse prediction of the crowd distribution (density map), while the attention network helps to distinguish crowded regions from backgrounds. The output of the attention network serves as a mask of the coarse density map. The number of persons in the scene is finally estimated by applying integration on the refined density map. In order to deal with persons of varied resolutions, we introduce a multi-scale fusion strategy which is built upon dilated convolution. Experiments are carried out on the standard benchmark datasets, covering varied over-crowded scenarios. Experimental results demonstrate the effectiveness of the proposed approach.

Detecting vehicle at night is critical to both assistant driving systems and autonomous driving systems. In this paper, we propose a deep network scheme assisted by light information with good generalization to detect vehicle at night. Our approach is divided into two branches, the object stream and the pixel stream. The object stream generates a batch of bounding boxes, and the pixel stream utilizes the vehicle light information to calibrate the bounding boxes of the object stream. In the object stream, we propose a new structure, Direction Attention Pooling (DAP), to improve the accuracy of the prior boxes. DAP leads into attention mechanism. The feature maps obtained from backbone network is divided into two branches. One branch obtains direction perception information through IRNN layer, and the other branch learns attention weights. The weights are multiplied with the direction perception features in an element-wise manner. In the pixel stream, we propose a corner localization algorithm based on Bayes to get more accurate corners with the vehicle light pixels. The locations of the corners are considered as a discrete random variable. When the mask of the object is known, solving the probability distribution of the corner of the object is the next step. The corners with the highest probability is the correct corner. On the nighttime vehicle detection datasets CHUK and SYSU, our method achieves the accuracy of 97.2% and 96.86%, which outperforms other state-of-the-art methods by at least 0.31% and 0.34%.

Speech emotion recognition is an important but difficult task in human–computer interaction systems. One of the main challenges in speech emotion recognition is how to extract effective emotion features from a long utterance. To address this issue, we propose a novel spatiotemporal and frequential cascaded attention network with large-margin learning in this paper. Spatiotemporal attention selectively locates the targeted emotional regions from a long speech spectrogram. In these targeted regions, frequential attention captures the emotional features by frequency distribution. The cascaded attention assists the neural network to gradually extract effective emotion features from the long spectrogram. During training, large-margin learning is applied to improve intra-class compactness and enlarge inter-class distances. Experiments on four public datasets demonstrate that our proposed model achieves a promising performance in speech emotion recognition.

To integrate the complete degradation information of machinery, deep learning-based prognostics approaches usually use monitoring data acquired by different sensors as the inputs of networks. These approaches, however, lack an explicit learning mechanism to effectively identify the distinctions of different sensor data and highlight the important degradation information, thereby affecting the accuracy of deep prognostics networks and limiting their generalization. To overcome the aforementioned weaknesses, a new deep prognostics framework named multiscale convolutional attention network (MSCAN) is proposed in this article for predicting the remaining useful life (RUL) of machinery. In the proposed MSCAN, self-attention modules are first constructed to effectively fuse the input multisensor data. Then, a multiscale learning strategy is developed to automatically learn representations from different temporal scales. Finally, the learned high-level representations are fed into dynamic dense layers to perform regression analysis and RUL estimation. The proposed MSCAN is evaluated using multisensor monitoring data from life testing of milling cutters, and also compared with some state-of-the-art prognostics approaches. Experimental results demonstrate the effectiveness and superiority of the proposed MSCAN in fusing multisensor information and improving RUL prediction accuracy.

Cross-domain recommendation can alleviate the data sparsity problem in the target domain and has become a promising research area. Recently, various models have been proposed to provide recommendation across domains. Some models successfully embedded adversarial learning to reduce domain discrepancies between the source and target domains; however, these models only focuse on extracting the domain-shared features among multiple domains. In this paper, we devise a novel framework that considers both domain-shared and domain-specific knowledge across domains, namely, DAAN . Specifically, we tightly couple matrix factorization-based collaborative filtering with deep adversarial domain adaptation via an attention network . In this framework, we first learn the domain-specific representations for each user and each item from the source and target user-item interaction matrices . Then, we capture the domain-shared features between two domains with common user (or item) embeddings in a domain-adversarial paradigm. Additionally, an attention network is used to adjust the degree of importance between domain-shared and domain-specific knowledge. Extensive experiments on six tasks demonstrate that our method outperforms various baselines in terms of two ranking metrics. To the best of our knowledge, our proposed approach is the first deep model that considers both domain-shared and domain-specific knowledge across domains within a domain-adversarial training paradigm for cross-domain recommendation.

Social networks are critical sources for event detection thanks to the characteristics of publicity and dissemination. Unfortunately, the randomness and semantic sparsity of the social network text bring significant challenges to the event detection task. In addition to text, time is another vital element in reflecting events since events are often followed for a while. Therefore, in this article, we propose a novel method named Multi-View Graph Attention Network (MVGAN) for event detection in social networks. It enriches event semantics through both neighbor aggregation and multi-view fusion in a heterogeneous social event graph. Specifically, we first construct a heterogeneous graph by adding the hashtag to associate the isolated short texts and describe events comprehensively. Then, we learn view-specific representations of events through graph convolutional networks from the perspectives of text semantics and time distribution, respectively. Finally, we design a hashtag-based multi-view graph attention mechanism to capture the intrinsic interaction across different views and integrate the feature representations to discover events. Extensive experiments on public benchmark datasets demonstrate that MVGAN performs favorably against many state-of-the-art social network event detection algorithms. It also proves that more meaningful signals can contribute to improving the event detection effect in social networks, such as published time and hashtags.

PurposeCourse recommendations are important for improving learner satisfaction and reducing dropout rates on massive open online course (MOOC) platforms. This study aims to propose an interpretable method of analyzing students' learning behaviors and recommending MOOCs by integrating multiple data sources.Design/methodology/approachThe study proposes a deep learning method of recommending MOOCs to students based on a multi-attention mechanism comprising learning records attention, word-level review attention, sentence-level review attention and course description attention. The proposed model is validated using real-world data consisting of the learning records of 6,628 students for 1,789 courses and 65,155 reviews.FindingsThe main contribution of this study is its exploration of multiple unstructured information using the proposed multi-attention network model. It provides an interpretable strategy for analyzing students' learning behaviors and conducting personalized MOOC recommendations.Practical implicationsThe findings suggest that MOOC platforms must fully utilize the information implied in course reviews to extract personalized learning preferences.Originality/valueThis study is the first attempt to recommend MOOCs by exploring students' preferences in course reviews. The proposed multi-attention mechanism improves the interpretability of MOOC recommendations.

With the development of the wide-area monitoring system (WAMS), power system operators are capable of providing an accurate and fast estimation of time-varying load parameters. This study proposes a spatial-temporal deep network-based new attention concept to capture the dynamic and static patterns of electrical load consumption through modeling complicated and non-stationary interdependencies between time sequences. The designed deep attention-based network benefits from long short-term memory (LSTM) based component to learning temporal features in time and frequency-domains as encoder-decoder based recurrent neural network. Furthermore, to inherently learn spatial features, a convolutional neural network (CNN) based attention mechanism is developed. Besides, this paper develops a loss function based on a pseudo-Huber concept to enhance the robustness of the proposed network in noisy conditions as well as improve the training performance. The simulation results on IEEE 68-bus demonstrates the effectiveness and superiority of the proposed network through comparison with several previously presented and state-of-the-art methods.

The analysis for social networks, such as the socially connected Internet of Things, has shown a deep influence of intelligent information processing technology on industrial systems for Smart Cities. The goal of social media representation learning is to learn dense, low-dimensional, and continuous representations for multimodal data within social networks, facilitating many real-world applications. Since social media images are usually accompanied by rich metadata (e.g., textual descriptions, tags, groups, and submitted users), simply modeling the image is not effective to learn the comprehensive information from social media images. In this work, we treat the image and its textual description as multimodal content, and transform other metainformation into the links between contents (such as two images marked by the same tag or submitted by the same user). Based on the multimodal content and social links, we propose a Deep Attentive Multimodal Graph Embedding model named DAMGE for more effective social image representation learning. We introduce both small- and large-scale datasets to conduct extensive experiments, of which the results confirm the superiority of the proposal on the tasks of social image classification and link prediction.

Single image deraining is an important problem in many computer vision tasks since rain streaks can severely hamper and degrade the visibility of images. Exisiting methods either focus on extracting rain streaks and ignore the background recovery, or the network structure is extremely complex and the number of parameters is quite large. Although some methods mention background restoration work, they generally ignore effective contextual information and result in unsatisfactory results. In this paper, we propose a novel network single image Deraining via detail-guided Efficient Channel Attention Network (DECAN) to remove rain streaks from rainy images. Specifically, we introduce two sub-networks with a comprehensive loss function that synergize to remove rain streaks and recover the background of the derained image. For completing rain streaks removal, we construct a rain streaks removal network with detail-guided efficient-channel-attention module to identify effective low-level features. For background recovery, we present a specialized background repair network consisting of well-designed blocks, named background details recovery network, to repair the background with effective contextual information for eliminating image degradations. Experiments on four synthetic datasets and some real-world rainy image sets show visual and numerical improvements of proposed method over the state-of-the-arts considerably.

The difficulty of cross-domain text sentiment classification is that the data distributions in the source domain and the target domain are inconsistent. This paper proposes an attention network based on feature sequences (ANFS) for cross-domain sentiment classification, which focuses on important semantic features by using the attention mechanism. Particularly, ANFS uses a three-layer convolutional neural network (CNN) to perform deep feature extraction on the text, and then uses a bidirectional long short-term memory (BiLSTM) to capture the long-term dependency relationship among the text feature sequences. We first transfer the ANFS model trained on the source domain to the target domain and share the parameters of the convolutional layer; then we use a small amount of labeled target domain data to fine-tune the model of the BiLSTM layer and the attention layer. The experimental results on cross-domain sentiment analysis tasks demonstrate that ANFS can significantly outperform the state-of-the-art methods for cross-domain sentiment classification problems.

In the environment perception stage of autonomous driving, vehicles need to track its surrounding objects quickly and accurately to avoid dangerous behaviors. Therefore, visual object tracking has important practical application value in autonomous driving system. However, the performance of most hierarchical convolutional feature trackers are limited by ignoring the complex environment of autonomous driving. In this paper, a novel Siamese Attention Network to explore the rich spatial and channel information of objects was proposed. Because of the lack of important information between the channel and the spatial position, the tracking performance is reduced by the challenges of illumination change and deformation. The spatial attention block and channel attention block focus on the importance of different spatial positions and channels, respectively. The effective fusion of the two makes our tracker achieve the state-of-the art performance of 0.300 in the EAO criterion of 2017, which exceeds the baseline by 5.7%.

Video-based action recognition has become a challenging task in computer vision and attracted extensive attention from the academic community. Most existing methods for action recognition treat all spatial or temporal input features equally, thus ignoring the difference of contribution provided by different features. To address this problem, we propose a spatial-temporal channel-wise attention network (STCAN) that is able to effectively learn discriminative features of human actions by adaptively recalibrating channel-wise feature responses. Specifically, the STCAN is constructed on a two-stream structure and we design a channel-wise attention unit (CAU) module. Two-stream network can effectively extract spatial and temporal information. Using the CAU module, the interdependencies between channels can be modelled to further generate a weight distribution for selectively enhancing informative features. The network performance of STCAN has been evaluated on two typical action recognition datasets, namely UCF101 and HMDB51, and comparable experiments have been performed to demonstrate the effectiveness of the proposed STCAN.

Product reviews can provide rich information about the opinions users have of products. However, it is nontrivial to effectively infer user preference and item characteristics from reviews due to the complicated semantic understanding. Existing methods usually learn features for users and items from reviews in single static fashions and cannot fully capture user preference and item features. In this article, we propose a neural review-based recommendation approach that aims to learn comprehensive representations of users/items under a three-tier attention framework. We design a review encoder to learn review features from words via a word-level attention, an aspect encoder to learn aspect features via a review-level attention, and a user/item encoder to learn the final representations of users/items via an aspect-level attention. In word- and review-level attentions, we adopt the context-aware mechanism to indicate importance of words and reviews dynamically instead of static attention weights. In addition, the attentions in the word and review levels are of multiple paradigms to learn multiple features effectively, which could indicate the diversity of user/item features. Furthermore, we propose a personalized aspect-level attention module in user/item encoder to learn the final comprehensive features. Extensive experiments are conducted and the results in rating prediction validate the effectiveness of our method.

Due to the influence of color, boundaries, and shapes of melanoma, the segmentation of the lesion area is still a challenging problem. In this paper, we propose a method to connect two attention modules (channel attention and spatial attention) serially and embed them into the skip connection of the encoder–decoder network. Different from previous work on image segmentation through attention mechanism, we have made a different combination of the existing popular spatial attention module and channel attention module. The experimental results show that the sequence combinations with channel attention in front and spatial attention in the rear are more likely to aggregate global and local information as well as information between channels than other combinations. Our method achieved an average Jaccard Index of 0.7692 on the ISIC2017 dataset. At the same time, we also compared with some advanced methods of image segmentation, the experimental results show our proposed method has a competitive performance.

We present a structural attention network (SAN) for graph modeling, which is a novel approach to learn node representations based on graph attention networks (GATs), with the introduction of two improvements specially designed for graph-structured data. The transition matrix was used to differentiate the structures between the nodes. The output features of nodes in the graph are represented as the concatenation of multi-order features to differentiate the structures among multiple orders. This novel neural network is based on a graph attention network, which makes the model pay attention to the topology of the graph. Using various experiments on citation networks and a protein-protein interaction dataset, we demonstrate the benefits of structural information in graph attention mechanisms.

Reading irregular text of arbitrary shape or low quality in natural scene images is a challenging task. Existing irregular scene text recognition methods mainly focus on irregular text with arbitrary shape, but rarely focus on irregular text of low quality. In this work, we propose a simple but effective method for recognizing irregular texts with arbitrary shape and low quality simultaneously. The proposed Multi-Branch guided Attention Network (MBAN) makes mutual guidance among multi-branch data in training, so as to learn invariant semantic representation between regular text images and the corresponding irregular images. Compared with the standard attention framework for text recognition, MBAN can significantly improve the performance of irregular text recognition while preserving similar performance for regular text recognition. In addition, regarding the attention drift problem encountered in standard attention network, MBAN can significantly improve the accuracy of alignment factors at each time step. We verify the effectiveness of MBAN in irregular text recognition and attention drift problem through extensive experiments. The performance of MBAN is shown to be comparable on regular datasets and superior on some irregular datasets with state-of-the-art methods.

The potential impact of social influence prediction has become a hot topic in the current graph data mining area. This paper proposes a deep learning framework named Multi-view Influence prediction network (MvInf) which combines multi-view learning and graph attention neural network together to address the problem of social influence prediction. MvInf takes different attribute features of users as the input of graph attention network and uses the complementarity and consistency between different views to enhance learning performance and thus to better predict user behavior. Experiments performed on four standard datasets (Open Academic Graph, Twitter, Weibo, and Digg) demonstrate that the proposed MvInf model can obtain better performance than previous single view-based approach.

The multistep-ahead prediction of chlorophyll provides an effective means for early warning of red tide. However, since multistep-ahead forecasting presents challenges, such as vague interactive relationships among ocean factors, long-term dependence modeling, and accumulative errors, existing methods mostly concentrate on the current time or one-step-ahead forecasting. In this paper, a hierarchical multistep-ahead forecasting model spatial-temporal attention network(STAN), which integrates the spatial context extractor network(SCE-net), long short-term memory network(LSTM), and the temporal attention mechanism, is proposed for the prediction of chlorophyll. In STAN, the input layer utilizes SCE-net to excavate relationships among ocean factors and generate high-level semantic via embedding factors into a continuous low-dimensional space. The middle layer applies LSTM to build the long-term dependencies of corresponding semantic representations. The output layer uses another LSTM with temporal attention to reduce accumulative errors and maintain temporal continuity. The attention can assign different weights to the middle layer’s hidden state and generate a context vector. Then the context vector and the final predicted value are considered as the current input for better forecasting. The buoy observation data of the Xiamen coastal area monitored in 2009–2011 is used to verify the efficiency of STAN. Experimental results prove that STAN outperforms the state-of-the-art methods of multistep-ahead prediction. When using 7 observation steps to forecast 15 steps, the MAPE of STAN is 0.3209, and the MAE is 0.1 lower than the values of the baselines approaches.

In recently years, some useful unsupervised video object segmentation methods that emphasize the common information in videos have been proposed. Despite the effectiveness of these methods, they ignore the information from the shallow layers of the network and thus fail to segment the details of the objects. To address this problem, we propose a multi-attention network for unsupervised video object segmentation (MANet). Recent studies show that the deep layers of networks are sensitive to high-level semantic information but messy details, while it is opposite for shallow layers. From this insight, a multi-attention module is designed by taking into account the information from the shallow layers in addition to that from the deep layers. This module can distinguish the primary object and segment the details of the object effectively by enhancing the common information between video frames while combing the features from the shallow layers and the deep layers. Experimental results on the DAVIS-2016 and SegTrack v2 datasets show that our network outperforms the state-of-the-art methods.

The goal of aspect-level sentiment analysis is to identify the sentiment polarity of a specific opinion target expressed; it is a fine-grained sentiment analysis task. Most of the existing works study how to better use the target information to model the sentence without using the interactive information between the sentence and target. In this article, we argue that the prediction of aspect-level sentiment polarity depends on both context and target. First, we propose a new model based on LSTM and the attention mechanism to predict the sentiment of each target in the review, the matrix-interactive attention network (M-IAN) that models target and context, respectively. M-IAN use an attention matrix to learn the interactive attention of context and target and generates the final representations of target and context. Then we introduce two gate networks based on M-IAN to build a deep interactive memory network to capture multiple interactions of target and context. The deep interactive memory network can excellently formulate specific memory for different targets, which is helpful in sentiment analysis. The experimental results of Restaurant and Laptop datasets of SemEval 2014 validate the effectiveness of our model.

Video segmentation automatically segments a target object throughout a video and has recently achieved good progress due to the development of deep convolutional neural networks (DCNNs). However, how to simultaneously capture long-range dependencies in multiple spaces remains an important issue in video segmentation. In this paper, we propose a novel triple attention network (TriANet) that simultaneously exploits temporal, spatial, and channel context knowledge by using the self-attention mechanism to enhance the discriminant ability of feature representations. We verify our method on the Shining3D dental, DAVIS16, and DAVIS17 datasets, and the results show our method to be competitive when compared with other state-of-the-art video segmentation methods.

Detecting multiple license plate numbers in crowded street scenes is challenging and requires the attention of researchers. In contrast to existing methods that focus on images that are not crowded with vehicles, in this work we aim at situations that are common and complex, for example, in city environments where numerous vehicles of different types like cars, trucks, motorbike etc. may present in a single image. In such cases, one can expect large variations in license plates in terms of quality, backgrounds, and various forms of occlusion. To address these challenges, we explore Adaptive Progressive Scale Expansion based Graph Attention Network (APSEGAT). This approach extracts local information which represents the license plates irrespective of vehicle types and numbers because it works at the pixel level in a progressive way, and identifies the dominant information in the image. This may include other parts of vehicles, drivers and pedestrians, and various other background objects. To overcome this problem, we integrate concepts of graph attention networks with progressive scale expansion networks. For evaluating the proposed method, we use our own dataset, named as AMLPR, which contains images captured in different crowded street scenes in different time span, and the benchmark dataset namely, UFPR-ALPR, which provides images of a single vehicle, and another benchmark dataset called, UCSD, which contains images of cars with different orientations. Experimental results on these datasets show that the method outperforms existing methods and is effective in detecting license plate numbers in crowded street scenes.

Massive Open Online Courses, offering millions of high-quality courses from prestigious universities and prominent experts, are picking up momentum in popularity. Although users enrolling on MOOCs have free access to abundant knowledge, they may easily get overwhelmed by information overload. Therefore, there is a need of recommending technology as a fundamental and well-accepted effective solution. However, differing from many other online recommendations, recommending courses to users on MOOCs faces two challenges. First, users’ knowledge background differs, so does their purpose of learning. Second, online courses are not independent but intertwined with prerequisite relations. Therefore, it is necessary to take these two challenges into account when designing a recommending method. To tackle this issue, in this article, we first propose two algorithms for extracting concept-level and course-level prerequisite relations. We then present the recommending method GuessUNeed based on neural attention network and course prerequisite relation embeddings. The experimental results on real-world datasets demonstrate the superiority of the proposed GuessUNeed method.

This paper conducts a systematic study on the role of spatial and temporal attention mechanism in the video salient object detection (VSOD) task. We present a two-stage spatial-temporal attention network, named STA-Net, which makes two major contributions. In the first stage, we devise a Multi-Scale-Spatial-Attention (MSSA) module to reduce calculation cost on non-salient regions while exploiting multi-scale saliency information. Such a sliced attention method offers an individual way to efficiently exploit the high-level features of the network with an enlarged receptive field. The second stage is to propose a Pyramid-Saliency-Shift-Aware (PSSA) module, which puts emphasis on the importance of dynamic object information since it offers a valid shift cue to confirm salient object and capture temporal information. Such a temporal detection module is able to encourage precise salient region detection. Exhaustive experiments show that the proposed STA-Net is effective for video salient object detection task, and achieves compelling performance in comparison with state-of-the-art.

Recognition of human emotion from facial expression is affected by distortions of pictorial quality and facial pose, which is often ignored by traditional video emotion recognition methods. On the other hand, context information can also provide different degrees of extra clues, which can further improve the recognition accuracy. In this paper, we first build a video dataset with seven categories of human emotion, named human emotion in the video (HEIV). With the HEIV dataset, we trained a context-aware attention network (CAAN) to recognize human emotion. The network consists of two subnetworks to process both face and context information. Features from facial expression and context clues are fused to represent the emotion of video frames, which will be then passed through an attention network and generate emotion scores. Then, the emotion features of all frames will be aggregated according to their emotional score. Experimental results show that our proposed method is effective on HEIV dataset.

Aspect-based sentiment analysis can predict the sentiment polarity of specific aspect terms in the text. Compared to general sentiment analysis, it extracts more useful information and analyzes the sentiment more accurately in the comment text. Many previous approaches use long short-term memory networks with attention mechanisms to directly learn aspect-specific representations and model comment text. However, these methods always ignore the importance of the aspect terms position and interactive information between the aspect terms and other words. To address these issues, we propose an improved model based on convolutional neural networks. First, a novel relative position encode layer can integrate the relative position information of specific aspect terms validly in a text. Second, by using the aspect attention mechanism, the semantic relationship between aspect terms and words in the text is fully considered. To verify the effectiveness of the proposed models, we conduct a large number of experiments and comparisons on seven public datasets. The experimental results show that this model outperforms to other state-of-the-art methods.

Forecasting the crowd flows of some densely populated areas (functional areas) at urban level is vital for the city and traffic management and much efforts have been put into this field. However, considering the complex dependencies of crowd flows prediction, the current models generally neglected the multi-scale correlation of spatial view and weakened the influence of external features on crowd flows data. In this paper, we proposed a deep learning framework, called MV-RANet, to deal with them and further improve the prediction performance. Firstly, a double-branch residual attention network was utilized to model the spatial correlation between fine-grained regions and coarse-grained functional areas separately with different branch. And furthermore, some human mobility pattern information were incorporated in MV-RANet to capture the correlation between functional areas in a different view. And as for the impact of external factors, we combined them with crowd flows data by multi-channel mechanism, which can make the external factors and spatio-temporal crowd flows data one-to-one mapping, to refine their influence on the crowd flows. We evaluated our model with two different datasets, a large scale Call Detail Records (CDRs) data and Beijing taxicabs’ trajectories data (TaxiBJ) (Zhang et al., 2017). The experimental results demonstrate that the proposed MV-RANet consistently outperforms all the baselines.

Cross-modal person re-identification(Re-id) under infrared light and visible light (RGB-IR) is of great significance for modern video surveillance, especially nighttime surveillance. The existing research results in the single-mode person re-identification field have reached a high level. Cross-model person re-identification, however, is rather challenging for the tremendous cross-modality and intra-modality difference in addition to common issues such as lighting conditions, human posture, camera angle, and etc..The Cross-modal Multi-granularity Attention Network (CMGN) proposed by this paper enables network to learn the common features of different modalities and map them to the same feature space. Major contributions made by this paper includes:1) A new ”butterfly” attention module that can be used for cross-modal tasks is designed to constrain the network attention to common areas of different modes. And in the RegDB and SYSU-MM01 dataset reached the effect of State-of-the-art(SOA). 2) An end-to-end multi-granularity feature fusion network dedicated to processing cross-modal problems is proposed.

Rating prediction has long been a popular research topic in the field of recommendation systems. Latent factor models, particularly matrix factorization (MF) methods, constitute the most prevalent techniques for rating prediction. However, MF-based methods suffer from the problems of data sparsity and lack of explicability. In this paper, we propose a novel model in which ratings and topic-level review information are integrated into a deep neural framework to address these problems. Specifically, we designed a topic alignment operation and a topic attention mechanism to reflect the user’s preferences and an item’s properties in terms of a topic in the reviews. We present a neural prediction layer that extends user and item representations by including both the latent factors from ratings and the textual information from reviews. The results of extensive experiments on four real-world datasets demonstrate that our proposed method consistently outperforms the state-of-the-art recommendation approaches that follow this direction in terms of rating prediction. Furthermore, our model can categorize representative reviews of users/items and group reviews into topics for the users’ reference.

Scene text recognition has become a research hotspot owing to its abundant semantic information and various applications. Recent methods of scene text recognition usually focus on handling shape distortion, attention drift, or background noise, ignoring that text recognition encounters character scale-variation problem. To address this issue, in this paper, we propose a new scale-aware hierarchical attention network (SaHAN) for scene text recognition. Inspired by feature pyramid network, we exploit the inherent pyramidal structure of a deep convolutional network to retain multi-scale features for flexible receptive fields. Then, we construct a hierarchical attention decoder that performs the attention mechanism twice on multi-scale features to collect the most fine-grained information for prediction. The SaHAN is trained in a weak supervision way, requiring only images and corresponding text labels. Extensive experiments on seven benchmarks reveal that SaHAN achieves state-of-the-art performance.

Recently, attention mechanisms have shown a developing tendency toward convolutional neural network (CNN), and some representative attention mechanisms, i.e., channel attention (CA) and spatial attention (SA) have been fully applied to single image super-resolution (SISR) tasks. However, the existing architectures directly apply these attention mechanisms to SISR without much consideration of the nature characteristic, resulting in less strong representational power. In this article, we propose a novel kernel attention module (KAM) for SISR, which enables the network to adjust its receptive field size corresponding to various scales of input by dynamically selecting the appropriate kernel. Based on this, we stack multiple kernel attention modules with group and residual connection to constitute a novel architecture for SISR, which enables our network to learn more distinguishing representations through filtering the information under different receptive fields. Thus, our network is more sensitive to multi-scale features, which enables our single network to deal with multi-scale SR task by predefining the upscaling modules. Besides, other attention mechanisms in super-resolution are also investigated and illustrated in detail in this article. Thanks to the kernel attention mechanism, the extensive benchmark evaluation shows that our method outperforms the other state-of-the-art methods.

Target-dependent sentiment analysis (TDSA) aims to classify the sentiment of a text towards a given target. The major challenge of this task lies in modeling the semantic relatedness between a target and its context sentence. This paper proposes a novel Target-Guided Structured Attention Network (TG-SAN), which captures target-related contexts for TDSA in a fine-to-coarse manner. Given a target and its context sentence, the proposed TG-SAN first identifies multiple semantic segments from the sentence using a target-guided structured attention mechanism. It then fuses the extracted segments based on their relatedness with the target for sentiment classification. We present comprehensive comparative experiments on three benchmarks with three major findings. First, TG-SAN outperforms the state-of-the-art by up to 1.61% and 3.58% in terms of accuracy and Marco-F1, respectively. Second, it shows a strong advantage in determining the sentiment of a target when the context sentence contains multiple semantic segments. Lastly, visualization results show that the attention scores produced by TG-SAN are highly interpretable

Natural language inference (NLI) is the basic task of many applications such as question answering and paraphrase recognition. Existing methods have solved the key issue of how the NLI model can benefit from external knowledge. Inspired by this, we attempt to further explore the following two problems: (1) how to make better use of external knowledge when the total amount of such knowledge is constant and (2) how to bring external knowledge to the NLI model more conveniently in the application scenario. In this paper, we propose a novel joint training framework that consists of a modified graph attention network, called the knowledge graph attention network, and an NLI model. We demonstrate that the proposed method outperforms the existing method which introduces external knowledge, and we improve the performance of multiple NLI models without additional external knowledge.

Aspect-based sentiment classification aims to predict the sentiment polarity of an aspect term in a sentence instead of the sentiment polarity of the entire sentence. Neural networks have been used for this task, and most existing methods have adopted sequence models, which require more training time than other models. When an aspect term comprises several words, most methods involve a coarse-level attention mechanism to model the aspect, and this may result in information loss. In this paper, we propose a multi-attention network (MAN) to address the above problems. The proposed model uses intra- and inter-level attention mechanisms. In the former, the MAN employs a transformer encoder instead of a sequence model to reduce training time. The transformer encoder encodes the input sentence in parallel and preserves long-distance sentiment relations. In the latter, the MAN uses a global and a local attention module to capture differently grained interactive information between aspect and context. The global attention module focuses on the entire relation, whereas the local attention module considers interactions at word level; this was often neglected in previous studies. Experiments demonstrate that the proposed model achieves superior results when compared to the baseline models.

Humor refers to the quality of being amusing. With the development of artificial intelligence, humor recognition is attracting a lot of research attention. Although phonetics and ambiguity have been introduced by previous studies, existing recognition methods still lack suitable feature design for neural networks. In this paper, we illustrate that phonetics structure and ambiguity associated with confusing words need to be learned for their own representations via the neural network. Then, we propose the Phonetics and Ambiguity Comprehension Gated Attention network (PACGA) to learn phonetic structures and semantic representation for humor recognition. The PACGA model can well represent phonetic information and semantic information with ambiguous words, which is of great benefit to humor recognition. Experimental results on two public datasets demonstrate the effectiveness of our model.

Semantic segmentation networks usually utilize special pyramid structure after encoder or combine low-level and high-level feature maps in decoder to capture multi-scale context information, which we term them feature combination and feature connection, respectively. However, both such frameworks are less valid with the fixed geometric structure or unsuitable interim. In this paper, we advocate Dynamic Attention Network (DAN) to solve these problems. First, we design a Deformable Attention Pyramid (DAP) module to perform a self-adjustable descriptor of high-level output, which utilizes deformable function to model geometric transformation. With DAP, semantic information can be captured effectively. Second, we propose a Fusing Attention Interim (FAI) module to guide the back-propagation of long-short range information in each level of decoder. We evaluate DAN on the challenging PASCAL VOC 2012 and Cityscapes segmentation benchmarks and find that it achieves state-of-the-art results without post-processing. Our observation can be concluded that the flexible structure that possesses dynamic attention mechanism is beneficial to learn multi-scale context information.

The normativity of workers’ actions during producing has a great impact on the quality of the products and the safety of the operation process. Previous studies mainly focused on the normativity of each single producing action instead of considering the normativity of continuous producing actions, which is defined as producing action flow (PAF) in this paper, during operation process. For this issue, a normativity judging method based on two-LSTM fusion network (TFN) and normativity-aware attention network (NAN) is proposed. First, TFN is designed to detect and recognize the producing actions based on skeleton sequences of a worker during complete operation process, and PAF data in sequential form are obtained. Then, NAN is built to allocate different levels of attention to each producing action within the sequence of PAF, and by this means, an efficient normativity judging is conducted. The combustor surface cleaning (CSC) process of rocket engine is taken as the experimental case, and the CSC-Action2D dataset is established for evaluation. Experiment results show the high performance of TFN and NAN, demonstrating the effectiveness of the proposed method for PAF normativity judging.

Matching video clips of people across non-overlapping surveillance cameras (video-based person re-identification) is of significant importance in many real-world applications. In this paper, we address the video-based person re-identification by developing a Local and Global Aligned Spatiotemporal Attention (LGASA) network. Our LGASA network consists of five cascaded modules, including 3D convolutional layers, residual block, spatial transformer network (STN), multi-stream recurrent network and multiple-attention module. Specifically, the 3D convolutional layers are used to capture local short-term fast-varying motion information encoded in multiple adjacent original frames. The residual block is used to extract mid-level feature maps. STN is applied to align the mid-level feature maps. The multi-stream recurrent network is designed to exploit the useful local and global long-term temporal dependency from the aligned mid-level feature maps. The multiple-attention module is designed to aggregate feature vectors of the same body part (or global) from different frames within each video into a single vector according to their importance. Experimental results on three video pedestrian datasets verify the effectiveness of the proposed local and global aligned spatiotemporal attention network.

The deep learning model has been used to predict the variation rule of the target series of multivariate time series data. Based on the attention mechanism, the influence information of multiple non-predictive time series on target series in different time stages is processed as the same weight in the previous studies. However, on real-world datasets, multiple non-predictive time series has different influence (such as different mutation information) on target series in different time stages. Therefore, a new multistage attention network is designed to capture the different influence. The model is mainly composed of the influential attention mechanism and temporal attention mechanism. In the influential attention mechanism, the same and different time stage attention mechanisms are used to capture the influence information of different non-predictive time series on the target series over time. In the temporal attention mechanism, the variation law of data can be captured over time. Besides, the prediction performance of proposed model on two different real-world multivariate time series datasets is comprehensively evaluated. The results show that, the prediction performance of the proposed model beat all baseline models and SOTA models. In a word, the multistage attention network model can effectively learn the information of the influence of different non-predictive time series on the target series in different time stages in the historical data.

For most previous attention-based scene text recognition methods, images are transformed into high-level feature vectors that form a feature map with height equal to one. Such vectors may contain unnecessary noise that limits recognition performance. To address this issue, in this paper, we propose the effective parts attention network (EPAN) which can attentively highlight the character region for more precise recognition. EPAN consists of a text image encoder and character effective parts decoder (CEPD), and it is end-to-end trainable. The former separates the high-dimensional feature map into one-dimensional vectors row-by-row, which are connected to a bidirectional long short term memory unit to encode contextual information. Subsequently, the CEPD transforms the vectors using a novel glimpse network at each time step to roughly determine the position of the characters. Then the CEPD uses a refinement network to generate a mask to gradually localize the precise position of important parts of the current character. Experiments were conducted on various benchmarks, including IIIT5K-Words, Street View Text, ICDAR 2003, ICDAR 2013, CUTE80, Street View Text Perspective, and ICDAR 2015, which demonstrated that the proposed EPAN method significantly outperformed or was comparable to existing methods in terms of lexicon-free word accuracy. Additionally, substantial qualitative results further demonstrated the robustness of our method.

Person re-identification (Re-ID) has become a hot topic in both research and industry. We joined in a person Re-ID challenge of the First National Artificial Intelligence Challenge (China, 2019) and found some model designs and training tricks work great or not on a super big private dataset. In this paper, we propose a model that combines the most effective designs, including multi-scale, multi-branch and attention mechanism, and report training tricks that are no less or even more important in improving person Re-ID performance. We analyze four commonly used public datasets: Market1501, DukeMTMC-ReID, CUHK03, and MSMT17, and achieve the state-of-the-art performance. Besides, we analyze and confirm the effectiveness of the designs by ablation studies. We also share strategies that play a key role in the challenge and experience of model designs that do not generalize well on large datasets.

With the availability of a large amount of user-generated online data, discovering users’ sequential behaviour has become an integral part of a Sequential Recommender System (SRS). Combining the recent observed items (i.e., short-term preferences) with prior interacted items (i.e., long-term preferences) has gained increasing attention in recent years. However, the existing methods mostly assume that all the adjacent items in a sequence are highly dependent, which may not be practical in real-world scenarios due to the uncertainty of customers’ shopping behaviours. A user-item interaction sequence may contain some irrelevant items which may in turn lead to false dependencies between items. Moreover, current studies usually assign a static representation to each item when modeling a user’s long-term preferences. Therefore, they cannot differentiate the contributions of the items. Specifically, these two types of users’ preferences have been separately modeled and then linearly combined, which may fail to model complicated user-item interactions. In order to overcome the above mentioned problems, we propose a novel Deep Attention-based Sequential (DAS) model. DAS consists of three different blocks, $(i)$ an embedding block: which embeds users and items into low-dimensional spaces; $(ii)$ an attention block: which aims to discriminatively learn dependencies among items in both users’ long-term and short-term item sets; and $(iii)$ a fully-connected block : which first learns a mixture of users’ preferences representation through a nonlinear way and then combines it with users’ embeddings to have a personalized recommendation. Extensive experiments demonstrate the superiority of our proposed model compared to the state-of-the-art approaches in SRSs.

One key challenge in facial expression recognition (FER) is the extraction of discriminative features from critical facial regions. Because of their promising ability to learn discriminative features, visual attention mechanisms are increasingly used to address pattern recognition problems. This paper presents a novel multiple attention network that simulates humans' coarse-to-fine visual attention to improve expression recognition performance. In the proposed network, a region-aware sub-net (RASnet) learns binary masks for locating expression-related critical regions with coarse-to-fine granularity levels and an expression recognition sub-net (ERSnet) with a multiple attention (MA) block learns comprehensive discriminative features. Embedded in the convolutional layers, the MA block fuses diversified attention using the learned masks from the RASnet. The MA block contains a hybrid attention branch with a series of sub-branches, where each sub-branch provides region-specific attention. To explore the complementary benefits of diversified attention, the MA block also has a weight learning branch that adaptively learns the contributions of the different critical regions. Experiments have been carried out on two publicly available databases, RAF and CK+, and the reported accuracies are 85.69% and 96.28%, respectively. The results indicate that our method achieves competitive or better performance than state-of-the-art methods.

Distinguishing between humans and common animals through a wall is necessary for facilitating successful rescue of survivors and enhancing the confidence of rescuers in post-disaster search and rescue operations. However, few existing solutions are available with only dogs considered in this scenario. This poses an issue in ensuring the recognition accuracy involving different animal species. This work proposed a novel multiscale residual attention network for distinguishing between stationary humans and common animals under a through-wall condition based on ultra-wideband radar, which is yet to be performed by existing research using deep learning. Humans, dogs, cats, rabbits, and no target data are collected and distinguished. The overall architecture of the proposed method differed from conventional deep learning methods as it is constructed by parallel 3 × 3 and 5 × 5 kernels incorporated with the residual attention learning mechanism. The effect of the slow-time dimension on the classification performance is analyzed, thereby producing an optimal input size. The overall F1-score of the proposed network can reach a high value of 0.9064 and the recognition accuracy of human targets can reach 0.983 satisfying the requirements for post-disaster rescue. Then, the effectiveness and advancement of the three components of the overall network architecture are validated by ablation studies. Finally, the proposed method is compared with three state-of-the-art methods. Comparison results indicate that the proposed method achieve a better performance. The network and its results are envisioned to be applied in various practical situations, such as earthquake rescue and intelligent homecare.