Search

In recent years, deep learning techniques have significantly improved the performance of single image super-resolution (SISR). However, this improvement is often achieved at the cost of introducing a large amount of parameters, which limits the real-world applications for SISR. In this paper, we propose a lightweight SISR network called Multi-scale Channel Attention Network for Image Super-Resolution (MCSN). Our contributions are threefold. First of all, the multi-scale feature fusion block (MSFFB) can extract multi-scale features by filters with different receptive fields. Secondly, the channel shuffle attention mechanism (CSAM) encourages the flow of the information across feature channels and enhances the ability of feature selection. Thirdly, the global feature fusion connection (GFFC) can effectively improve feature utilization. Extensive experiments demonstrate that the parameter amount of our method is reduced by 3/4 compared with the current state-of-the-art MSRN method, while both the subjective visual effect and objective quality of the reconstructed high-resolution images are significantly better. [ABSTRACT FROM AUTHOR]

In text classification models based on deep learning, feature extraction and feature aggregation are two key steps. As one of the basic feature extraction methods, CNN has certain limitations due to its inability to effectively extract temporal features from text data. Using max-pooling can significantly reduce the amount of calculation while performing feature aggregation, but it will have an adverse effect on the classification results due to the loss of some text features. In this paper, in response to the above two issues, a Bidirectional Gated Temporal Convolutional Attention(BG-TCA) model is proposed. In the feature extraction stage, the BG-TCA model uses the bidirectional TCN to extract the bidirectional temporal features in text data, and a gating mechanism similar to the LSTM is added between the convolution layers. In the feature aggregation stage, the BG-TCA model uses the attention mechanism to replace the max-pooling method, which makes it possible to distinguish the importance of different features while retaining the text features to the maximum. Finally, experimental results on five benchmark datasets show that the classification accuracy of the BG-TCA model has been greatly improved compared to basic models, and is better than several other state-of-the-art models. [ABSTRACT FROM AUTHOR]

Recently, depth map super-resolution (DSR) has obtained remarkable performance with the development of convolutional neural networks (CNNs). High-resolution (HR) depth map can be inferred from a low-resolution (LR) one with the guidance of its corresponding HR intensity image. However, most of the existing CNNs-based methods unilaterally transfer structures information of guidance image to the input depth map, which ignores the corresponding relations between the depth map and the intensity map. In this paper, we propose a novel dual auto-encoder attention network (DAEANet) for DSR. The proposed DAEANet includes two auto-encoder networks, where guidance auto-encoder network (GAENet) and target auto-encoder network (TAENet) aim to extract feature information from intensity image and depth map. Specifically, all auto-encoder networks are similar and trained simultaneously to ensure structural consistency. Furthermore, to preserve the structure information in the process of training, the attention mechanism is employed to our DAEANet. Extensive experiments on several popular benchmarks show that the proposed DAEANet outperforms existing state-of-the-art algorithms. [ABSTRACT FROM AUTHOR]

With the increasing attention to the energy crisis and global warming, solar generation has become an important way to use clean solar energy and is playing an increasingly important role. Due to the highly-variable patterns of solar generation, the estimation of prediction intervals is receiving more attention, which is conducive to the safe and stable operation of the power system. In order to further improve the performance of prediction intervals of solar generation, this paper proposes a prediction intervals estimation method for solar generation based on gated recurrent unit (GRU) neural networks and kernel density estimation (KDE). GRU, a commonly used recurrent neural networks, is utilized to obtain the deterministic forecast of solar generation. In addition, according to the characteristics of solar generation, attention mechanism is designed on the GRU prediction model to further improve the prediction performance. Then, the KDE method is used to fit the prediction errors of solar generation obtained by the deterministic forecasting method. In order to verify the effectiveness of the proposed method, we have carried out a large number of experiments on freely available datasets. The experimental results show that the proposed method outperforms competing methods and can generate high-quality prediction intervals. [ABSTRACT FROM AUTHOR]

Most existing methods for video person re-identification apply spatial-temporal global average or attention pooling to aggregate frame-level feature and get video-level feature. The obtained video-level feature models only the first-order statistics of the appearance feature from holistic video, resulting in limited representation capability of the feature network. In this paper, we propose a novel Global Statistic Pooling network (GSPnet) which takes full advantage of the second-order information for enhancing modeling capability. Firstly, a novel global statistic pooling module is proposed to summarize both the first- and second-order statistics across frame-level feature, and then transfer them into a compact and robust video-level feature embedding. Secondly, a statistic-based attention block is incorporated into multiple stages of convolutional networks to fully explore the second-order representations from low- to high-level features. To enhance the representation learning ability and further boost re-identification (re-ID) performance, we also propose a multi-level self-attention distillation training scheme, which squeezes the knowledge learned in the deeper portion of the networks into the shallow ones. Extensive experimental results have demonstrated the effectiveness and superiority of our approach on four popular video person re-ID datasets. [ABSTRACT FROM AUTHOR]

Video deblurring is a challenging low-level vision task due to variant blur artifacts caused by factors such as depth variations, high-speed movements and camera shakes. Although significant efforts have been devoted to addressing this task, two challenges of capturing temporal patterns and spatial topologies still remain. In this paper, an attention-based interframe compensation scheme is proposed to address the first challenge. The proposed scheme replaces frames in blurry sequences with newly restored frames, and estimates temporal patterns among the replaced sequence to restore the whole sequence. After each replacement, an attention block is employed to exploit dependencies among restored and blurry frames to capture stable temporal patterns. To tackle the second challenge, we propose an adaptive residual block that dynamically fuses multi-level features via learning location-specific weights. Comprehensive experimental results demonstrate that the proposed method achieves state-of-the-art performance in terms of accuracy, visual effect and model size. [ABSTRACT FROM AUTHOR]

Attention mechanisms can extract salient features in images, which has been proven to be effective for person re-identification. However, focusing on the saliency of an image is not enough. On the one hand, the salient features extracted from the model are not necessarily the features needed, e.g., a similar background may also be mistaken as salient features; on the other hand, various salient features are often more conducive to improving the performance of the model. Based on this, in this paper, a model that has joint weak saliency and attention aware is proposed, which can obtain more complete global features by weakening saliency features. The model then obtains diversified saliency features via attention diversity to improve the performance of the model. Experiments on commonly used datasets prove the effectiveness of the proposed method. [ABSTRACT FROM AUTHOR]

Traffic sign detection has made great progress with the rise of deep learning in recent years. As a result of the complex and changeable traffic environment, detecting small traffic signs in a real-world scene is still a challenging problem. In this paper, a novel group multi-scale attention pyramid network is proposed to address the problem. Specifically, to aggregate the feature at different scales and suppress the messy information in the background, an effective multi-scale attention module is proposed. Furthermore, a feature fusion module, named adaptive pyramid convolution, is further designed, which can drive the network to learn the optimal feature fusion pattern and construct an informative feature pyramid for detecting traffic signs in different sizes. Extensive experimental results on the public traffic sign detection datasets demonstrate the effectiveness and superiority of the proposed method when compared with several state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Attention has arguably become one of the most important concepts in the deep learning field. It is inspired by the biological systems of humans that tend to focus on the distinctive parts when processing large amounts of information. With the development of deep neural networks, attention mechanism has been widely used in diverse application domains. This paper aims to give an overview of the state-of-the-art attention models proposed in recent years. Toward a better general understanding of attention mechanisms, we define a unified model that is suitable for most attention structures. Each step of the attention mechanism implemented in the model is described in detail. Furthermore, we classify existing attention models according to four criteria: the softness of attention, forms of input feature, input representation, and output representation. Besides, we summarize network architectures used in conjunction with the attention mechanism and describe some typical applications of attention mechanism. Finally, we discuss the interpretability that attention brings to deep learning and present its potential future trends. [ABSTRACT FROM AUTHOR]

Optical coherence tomography (OCT) is one of the most commonly used ophthalmic diagnostic techniques. Macular Edema (ME) is the swelling of the macular region in the eye. Segmentation of the fluid region in the retinal layer is an important step in detecting lesions. However, manual segmentation is often a time consuming and subjective process. In this paper, an improved U-Net segmentation method is proposed. In this method, the attention mechanism is introduced to automatically locate the fluid region, which avoids the problem of excessive calculation in multi-stage methods. At the same time, the use of dense skip connections which combines high-level and low-level features makes the segmentation results more precise. The loss function is a joint loss, including weighted binary cross entropy loss, dice loss, and regression loss, where regression loss is used to avoid the problem of merging multiple fluid regions into one. The experimental results show that the proposed method can adapt to the OCT scans acquired by various imaging scanning devices, and this method is more effective than other start-of-the-art fluid segmentation methods. [ABSTRACT FROM AUTHOR]

• A novel framework is proposed to deal with crowd counting in dense scenarios. • We adopt a multi-scale fusion strategy, which is built upon dilated convolution. • An attention mechanism is introduced to concentrate on crowd regions. In this paper, we propose an attention-guided multi-scale fusion network (named as AMS-Net) for crowd counting in dense scenarios. The overall model is mainly comprised by the density and the attention networks. The density network is able to provide a coarse prediction of the crowd distribution (density map), while the attention network helps to distinguish crowded regions from backgrounds. The output of the attention network serves as a mask of the coarse density map. The number of persons in the scene is finally estimated by applying integration on the refined density map. In order to deal with persons of varied resolutions, we introduce a multi-scale fusion strategy which is built upon dilated convolution. Experiments are carried out on the standard benchmark datasets, covering varied over-crowded scenarios. Experimental results demonstrate the effectiveness of the proposed approach. [ABSTRACT FROM AUTHOR]

Multi-object tracking (MOT) has received constant attention from researchers with the development of deep learning and person re-identification (ReID). However, the occlusion caused tracking failure is still far from solved. In this paper, we propose a Hard-Soft Attention Network (HSAN) to improve the ReID performance and get robust appearance features of different targets. The pose information and attention mechanism are combined to distinguish between challenging targets. Besides, the unary and binary costs are constructed to ensure consistency and long-term tracking, which consider not only the appearance-motion affinity of single tracks, but also the interactions between neighboring tracks. For that we cluster the tracks into different groups and choose reliable tracks as anchors to establish the two types of costs. Our HSAN appearance model is evaluated on the Market-1501, DUKE and CUHK03 ReID datasets and the MOT tracking method is conducted on MOTChallenge 15, 16 and 17. The experimental results demonstrate that our method can improve tracking accuracy and reduce fragments. [ABSTRACT FROM AUTHOR]

• We propose an accurate and efficient suggestive annotation strategy based on quality-driven active learning (QDAL) for 3D segmentation in brain MRI datasets. • We derive two quality indicators (the pseudo dice similarity coefficient and the pseudo accuracy) from the candidate segmentation predictions to measure the segmentation quality of the current model to the unlabeled slices. Both two indicators have a positive correlation with the actual segmentation accuracy. • We only use 15–20% sparsely annotated slices for brain extraction and 30–40% sparsely annotated slices for tissue segmentation to train our proposed model, which gains the competitive segmentation results compared with fully supervised training methods. Automatic segmentation of the brain Magnetic Resonance Imaging (MRI) plays a crucial role in many brain MRI processing algorithms, which is effective for the prevention, detection, monitoring, and treatment planning of brain disease. Currently, deep learning algorithms have shown outstanding performance in brain segmentation. Most algorithms train models with fully annotated brain MRI datasets. However, full annotation of 3D brain MRI is laborious and time-consuming. Using sparsely annotated datasets to train models may be one appropriate solution to reduce annotation cost. However, the approach of 3D dense segmentation with sparse annotation has not been fully studied. In this paper, we develop a segmentation framework combined with the quality-driven active learning (QDAL) module for suggestive annotation. In the proposed Active Learning module, attention mechanism and deep supervision mode are used to improve the segmentation accuracy and feedback segmentation quality information. Meanwhile, we observe a high correlation coefficient between the proposed two surrogate metrics and the real segmentation accuracy of per slice in one scan. We validate our framework on two public brain MRI datasets for brain region extraction and brain tissue segmentation. The comparative experiments demonstrate that the QDAL method outperforms the other four popular sampling strategies. The segmentation network with the guidance of the QDAL method only needs 15–20% annotated slices in brain extraction task, and 30–40% annotated slices in tissue segmentation task to achieve competitive results compared with training with full supervision. [ABSTRACT FROM AUTHOR]

In interpersonal communication, the expression is an import way to express one's emotions. In order to make computers understand facial expressions like human beings, a large number of researchers have put a lot of time and energy into it. But for now, most of the work of dynamic sequence facial expression recognition fails to make full use of the combined advantages of shallow features (prior knowledge) and depth features (high-level semantic). Therefore, this paper implements a dynamic sequence facial expression recognition system that integrates shallow features and deep features with the attention mechanism. In order to extract the shallow features, an Attention Shallow Model (ASModel) is proposed by using the relative position of facial landmarks and the texture characteristics of the local area of the face to describe the Action Units of the Facial Action Coding System. And with the advantage of the deep convolutional neural network in expressing high-level features, a Attention Deep Model (ADModel) is also designed to extract deep features on sequence facial images. Finally, the ASModel and the ADModel are integrated to a Multi-attention Shallow and Deep Model (MSDModel) to complete the dynamic sequence facial expression recognition. There are three kinds of attention mechanism introduced, such as Self-Attention (SA), Weight-Attention (WA), and Convolution-Attention (CA). We verify our dynamic expression recognition system on three publicly available databases include CK+, MMI, and Oulu-CASIA and get superior performance than other state-of-art results. [ABSTRACT FROM AUTHOR]

The main objective of this work is to improve the quality and transparency of the medical text classification solutions. Conventional text classification methods provide users with only a restricted mechanism (based on frequency) for selecting features. In this paper, a three-stage hybrid method combining the gated attention-based bi-directional Long Short-Term Memory (ABLSTM) and the regular expression based classifier is proposed for medical text classification tasks. The bi-directional Long Short-Term Memory (LSTM) architecture with an attention layer allows the network to weigh words according to their perceived importance and focus on crucial parts of a sentence. Feature words (or keywords) extracted by ABLSTM model are utilized to guide the regular expression rule construction. Our proposed approach leverages the advantages of both the interpretability of rule-based algorithms and the computational power of deep learning approaches for a production-ready scenario. Experimental results on real-world medical online query data clearly validate the superiority of our system in selecting domain-specific and topic-related features. Results show that the proposed approach achieves an accuracy of 0.89 and an F 1 -score of 0.92 respectively. Furthermore, our experimentation also illustrates the versatility of regular expressions as a user-level tool for focusing on desired patterns and providing interpretable solutions for human modification. [ABSTRACT FROM AUTHOR]

• We propose the residually dense distillated attention networks (RDDAN). • We introduce a dense distillation network (DDN). • We propose the channel attention producing (CAP) module. Deep convolutional neural networks (CNNs) have been playing an increasingly important role in image super-resolution (SR). However, if we just deepen or widen the networks, it could result in the excess of parameters and the increase of training difficulty. In this paper, we propose a residually dense distilled attention network (RDDAN) to address the problems in SR. Residual networks could make full use of the information of previous layers. In RDDAN we propose a connection block group (CBG), which is stacked in the feature extraction module of the network. CBG consists of two parts, dense enhancement network (DEN) and channel attention producing (CAP) module. First, instead of simply stacking residual blocks, DEN utilizes feature distillation with both dense concatenation and skip connection to extract deep and shallow features, which could enhance the representation ability. Second, with attention mechanism, CAP pays attention to the channel-wise association to adjust channel-wise features and restore high-frequency feature information. By evaluating the performance of results based on benchmark methods, our method achieves a more desirable performance than state-of-the-art methods. [ABSTRACT FROM AUTHOR]

The main objective of the recommender system is to determine user behavior or preference and suggest the items of interest to the user. We notice that topic information is one of the significant factors that enables a user to choose the next items. To mimic this behavior, we design a topic diversity discovering (TDD) model to capture user preferences on topics. Unlike the previous studies applied the topic information to recommender systems, which only considered the item topics. In this paper, we not only focus on the item topic information but also consider the user topic information on the recommender system. We develop a TDD model to learn the topic information from the item, and we then utilize the transfer learning to obtain the user topic information. We then integrate the TDD model with the Bayesian personalized ranking (BPR) model to build a novel TDD-BPR recommender system. We experiment on three well-known three datasets MovieLens-1 M, Pinterest, and TMDB dataset, which achieved 71.8%, 91.2%, and 96.1% in the Hit Ratio@10. Empirical evidence showed that TDD-BPR discover the potential topic information to understand the user's preferences and get better recommendation performance. [ABSTRACT FROM AUTHOR]

The performance of existing approaches for dialogue state tracking (DST) is often limited by the deficiency of labeled datasets, and inefficient utilization of data is also a practical yet tough problem of the DST task. In this paper, we aim to tackle these challenges in a self-supervised manner by introducing an auxiliary pre-training task that learns to pick up the correct dialogue response from a group of candidates. Moreover, we propose an attention flow mechanism that is augmented with a soft-threshold function in a dynamic way to better understand the user intent and filter out the redundant information. Extensive experiments on the multi-domain dialogue state tracking dataset MultiWOZ 2.1 demonstrate the effectiveness of our proposed method, and we also show that it is able to adapt to zero/few-shot cases under the proposed self-supervised framework. [ABSTRACT FROM AUTHOR]

Visual affordance detection is an important issue in the field of robotics and computer vision. This paper proposes a novel and practical convolutional neural network architecture that adopts an encoder-decoder architecture for pixel-wise affordance detection. The encoder network comprises two modules: a dilated residual network that is the backbone for feature extraction, and an attention mechanism that is used for modeling long-range, multi-level dependency relations. The decoder network consists of a novel up-sampling layer that maps the low-resolution encoder feature to a high-resolution pixel-wise prediction map. Specifically, integrating an attention mechanism into our network reduces the loss of salient details and improves the feature representation performance of the model. The results of experiments conducted on the University of Maryland dataset (UMD) verify that the proposed network with the attention mechanism and up-sampling layer improved performance compared with classical methods. The proposed method lays the foundation for subsequent research on multi-task learning by physical robots. [ABSTRACT FROM AUTHOR]

Facial image inpainting is a challenging task since the facial images lose much content causing blur and unnaturalness. In this paper, we propose facial image inpainting using attention-based multi-level generative network. We adopt multi-level feature processing to reduce the training and testing time while enhancing the performance by reducing the number of channels of each convolutional layer in the generator. We combine attention with multi-level feature processing to maintain soft correlation with the surrounding content. For network optimization, we utilize two kinds of loss functions: content and texture. Content loss consists of mean absolute error (MAE) and edge-preserving losses, and produces inpainting results close to the ground truth. Texture loss consists of adversarial and perceptual losses to fine-tune the texture synthesis. Besides, we use the edge-preserving loss to take edge and patch similarity. Comparative experiments indicate that the proposed method produces photo-realistic and plausible inpainting results in random masks as well as outperforms the state-of-the-art ones in terms of quantitative measurements and subjective evaluations. [ABSTRACT FROM AUTHOR]

• Unpooling is adopted to get over boundary ambiguity. • Convolutional blocks are replaced by dense blocks to for parameters reduction. • A dual attention mechanism is proposed for feature refinement. • A composite loss function is introduced to alleviate vanishing-gradient problem. • The proposed method can be applied to nasopharyngeal carcinoma segmentation in clinic. Precise and accurate segmentation of the most common head-and-neck tumor, nasopharyngeal carcinoma (NPC), in magnetic resonance images (MRI) sheds light on treatment and regulatory decisions making, especially on radiotherapy planning. However, segmenting NPC manually is time-consuming and expensive. Thus, automatic segmentation of NPC is highly demanded. Whereas, problems such as large variations in the lesion size and shape of NPC, boundary ambiguity, as well as limited available annotated samples, conspire NPC segmentation towards a formidable task. Consequently, existing NPC segmentation methods cannot satisfy the high requirements for medical practice. Motivated by these challenges and prevalence of deep learning, in this paper, we propose a Dual Attention-based Dense SU-net (DA-DSUnet) framework for automatic NPC segmentation. It is an encoder-decoder network taking 2D NPC MRI slices as input and outputting the corresponding segmentation results. The main innovations of our method are fourfold. First, different from the traditional decoder in the baseline (U-net) which uses upconvolution for upsamling, we argue that the restoration from low resolution features to high resolution outputs should be capable of preserving the information related to boundary localization. Therefore, we use unpooling as the upsampling method in our model. Second, to combat the vanishing-gradient problem, we introduce dense block which can facilitate feature propagation to replace the traditional convolutional block. Third, we incorporate a dual attention mechanism in our network, which models the inter-dependencies in position and channel dimensions. Fourth, using only binary cross entropy (BCE) as loss function may bring about troubles such as miss-prediction. Hence, we propose to use a loss function named BCEDice to train the network. Quantitative and qualitative comparisons are carried out extensively on in–house dataset. The experimental results show that the proposed method achieves a DSC of 0.8050, a PM of 0.8026 and a CR of 0.7065, which respectively has a relative gain of 5.17%, 13.8% and 10.3% over U-net, indicating the effectiveness of our method. [ABSTRACT FROM AUTHOR]

Image captioning means automatically generating a caption for an image. As a recently emerged research area, it is attracting more and more attention. To achieve the goal of image captioning, semantic information of images needs to be captured and expressed in natural languages. Connecting both research communities of computer vision and natural language processing, image captioning is a quite challenging task. Various approaches have been proposed to solve this problem. In this paper, we present a survey on advances in image captioning research. Based on the technique adopted, we classify image captioning approaches into different categories. Representative methods in each category are summarized, and their strengths and limitations are talked about. In this paper, we first discuss methods used in early work which are mainly retrieval and template based. Then, we focus our main attention on neural network based methods, which give state of the art results. Neural network based methods are further divided into subcategories based on the specific framework they use. Each subcategory of neural network based methods are discussed in detail. After that, state of the art methods are compared on benchmark datasets. Following that, discussions on future research directions are presented. [ABSTRACT FROM AUTHOR]

Smoke detection based on video monitoring is of great importance for early fire warning. However, most of the smoke detection methods based on neural network only consider the normal weather. The harsh weather such as the fog environment is ignored. In this paper, we propose a smoke detection in normal and fog weather, which combines attention mechanism and feature-level and decision-level fusion module. First, a new fog smoke dataset with diverse positive and hard negative samples dataset is established through online collection and offline shooting. Then, an attention mechanism module combining spatial attention and channel attention is proposed to solve the problem of small smoke detection. Next, a lightweight feature-level and decision-level fusion module is proposed, which can not only improve the discrimination of smoke, fog and other similar objects, but also ensure the real-time performance of the model. Finally, a large number of comparative experiments on the existing dataset and our self-created dataset, show that our method can obtain higher detection accuracy rate, precision rate, recall rate, and F1 score from the perspective of overall, each category, small smoke and hard negative samples detection than the existing methods. [ABSTRACT FROM AUTHOR]

This paper presents a biologically-inspired saliency prediction method to imitate two main characteristics of the human perception process: focalization and orienting. The proposed network, named ACNet is composed of two modules. The first one is an essential concentrated module (CM), which assists the network to "see" images with appropriate receptive fields by perceiving rich multi-scale multi-receptive-field contexts of high-level features. The second is a parallel attention module (PAM), which explicitly guides the network to learn "what" and "where" is salient by simultaneously capturing global and local information with channel-wise and spatial attention mechanisms. These two modules compose the core component of the proposed method, named ACBlock, which is cascaded to progressively refine the inference of saliency estimation in a manner similar to that humans zoom in their lens to focus on the saliency. Experimental results on seven public datasets demonstrate that the proposed ACNet outperforms the state-of-the-art models without any prior knowledge or post-processing. [ABSTRACT FROM AUTHOR]

Accurate traffic prediction plays an important role in Intelligent Transportation System. This problem is very challenging due to the heterogeneity and dynamic spatio-temporal dependence of large-scale traffic data. Existing models often suffer two limitations: (1) They usually only consider one type of data in the input, or simply treat other collected time series data as features, ignoring the non-linear interactions among different series. In fact, heterogeneous data at a specific location has direct impacts on the predicted series. (2) The method based on graph convolutional network uses a fixed Laplacian matrix to model spatial correlation, without considering its dynamics. The aggregations also occur only in the neighborhood, making it difficult to capture long-range dependencies. In this paper, we propose a M ulti-Stage A ttention S patial- T emporal G raph N etworks (MASTGN). First, an internal attention mechanism is designed to capture the interactions among multiple time series collected by the same sensor. Second, to model the complex spatial correlations, we apply a dynamic neighborhood-based attention mechanism. Unlike the general attention-based methods that ignore the structure information of the road network, we use the adjacency relations as a prior to divide the nodes of a road network into different neighborhood sets. In this way, attention can capture spatial correlations both within the same order neighborhood, and among different neighborhoods dynamically. Furthermore, a temporal attention mechanism is used to extract the dynamic temporal dependencies. Experiments are conducted on two real traffic datasets, and the results verify the effectiveness of the proposed model. [ABSTRACT FROM AUTHOR]

The task of multi-source web entity resolution (MSWER) aims to automatically discover entity references from multiple web sources that refer to the same real-world entity, which plays an important role in tasks such as question answering and recommendations. However, existing approaches typically suffer from three major limitations: (1) they usually treat the MSWER as an information retrieval task and focus on learning the similarity between entity references based on the associated features extracted from multiple sources; (2) they ignore the valuable implicit interactions between the associated features of different entities that cannot be directly captured based on the given data without any external knowledge; (3) they didn't consider the redundant and noisy interactions between features. To overcome these limitations, this paper presents a novel attentive interaction-driven entity resolution model (AIDER). Our theme is to capture both the explicit and implicit interactions of features associated with entity references in the form of paths, and further develop an end-to-end entity resolution model for inferring the equivalent entity references. Accordingly, an external knowledge base is leveraged to construct paths for implicit interactions, and a well-designed attention mechanism is further employed to measure the importance of each path-based interaction, which focuses on useful interactions and neglects those redundant and noisy ones. Experimental results on three real-world datasets demonstrate that AIDER outperforms the state-of-the-art approaches. [ABSTRACT FROM AUTHOR]

Recommender system is an important technique to find the information that the users may be interested by their feedbacks. However, it is still a challenge to model the preference of users due to the sparsity of user feedbacks. To alleviate this problem, many methods are developed by extracting information from various kinds of auxiliary information that are related to the users. In the auxiliary information, review is the popular one, since it can reflect both user preferences and item characteristics. Moreover, the review can generate plausible recommendation explanations in the recommendation results. In this paper, we propose a hybrid deep collaborative filtering model that jointly learns rating embedding and textural feature from ratings and reviews respectively. Specifically, two embedding layers are employed to learn rating embedding for users and items based on the interactions, and two attention-based GRU networks attempt to learn context-aware representation as textural feature for users and items from reviews. To leverage the contribution between rating embedding and textual feature and obtain the fused features for users and items, a proposed gating mechanism is used. Then an interaction-learning layer is adopted to learn the user and item interaction information based on the fused user and item features. The prediction score is obtained with the factorization machine. Experimental results on six real-world datasets demonstrate the superior performance of the proposed method over several state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Aspect category sentiment analysis (ACSA) excels at identifying the aspect categories and corresponding sentiments involved in a sentence, regardless of whether the aspect terms are explicitly mentioned or not. However, current methods tend to overinflate the original data, resulting in the introduction of unnecessary information, and fail to capture the inter-task relationship sufficiently. This paper presents a new method termed the prompt-based joint model (PBJM) to address these complications. PBJM treats the sentiment polarity prediction as binary classification and leverages a natural language prompt template, a concise sentence that guides the model to perform aspect category identification subtask and curtails the need for data augmentation. The two subtasks are jointly trained in pre-trained language models (PLMs) to capture their correlation. Further, the attention mechanism for aspect categories enables the model to concentrate selectively on significant features such as phrases and words during the predictions. In addition, the verbalizer employs a set of parameters to balance the weight of each label word while projecting between the label space and the label words space. Through experiments on four datasets, our model demonstrated remarkable performance in detecting category-sentiment pairs. [ABSTRACT FROM AUTHOR]

Point of Interest (POI) recommendation algorithms can help users find the POIs that they prefer, and they can also help merchants to find potential customers. However, most existing methods still have difficulties effectively utilizing the information in users' check-in data. Significantly, they ignore the intent behind the users' check-in behaviors, which limits the recommendation performance. In this paper, we propose an I ntent A ware G raph N eural N etwork-based model(IAGNN) to predict/recommend the next POI with which the target user may interact. Specifically, IAGNN first models the user's check-in behavior sequences as graphs and utilizes the information transmission mechanism of the graph neural network (GNN) to learn the feature vector representation (embedding) of POIs. Second, we devise a hierarchical attention network for capturing users' preferences adaptively. At the same time, we design a user intent-aware module based on disentangled representations to extract the user's intents. Finally, the user's preferences and their intents obtained by the user intent perception module are combined to recommend the POI for the user. Extensive evaluations are conducted on two real-world POI check-in datasets. The experimental results show that our proposed model IAGNN outperforms the baselines in terms of both recall and MRR. [ABSTRACT FROM AUTHOR]

In multi-modal learning tasks such as video understanding, the most important operations are feature extraction, feature enhancement for single modality and feature aggregation between modalities. In this paper, we present two attention based algorithms, the Position-embedding Non-local (PE-NL) Network and the Multi-modal Attention (MA) feature aggregation method. Inspired by Non-local Neural Networks and Transformers, our PE-NL is a self-attention liked feature enhancement operation and it can capture long-range dependencies and model relative positions. The MA aggregation method merges visual and audio modals while reduces feature dimension and the number of parameters without losing too much accuracy. Both of PE-NL and MA blocks can be plugged into many multi-modal learning architectures. Our Gated PE-NL-MA network achieves competitive results on Youtube-8M dataset. [ABSTRACT FROM AUTHOR]

Weakly supervised semantic segmentation has been attracting increasing attention as it can alleviate the need for expensive pixel-level annotations through the use of image-level labels. Relevant methods mainly rely on the implicit object localization ability of convolutional neural networks (CNNs). However, generated object attention maps remain mostly small and incomplete. In this paper, we propose an Atrous Convolutional Feature Network (ACFN) to generate dense object attention maps. This is achieved by enhancing the context representation of image classification CNNs. More specifically, cascaded atrous convolutions are used in the middle layers to retain sufficient spatial details, and pyramidal atrous convolutions are used in the last convolutional layers to provide multi-scale context information for the extraction of object attention maps. Moreover, we propose an attentive fusion strategy to adaptively fuse the multi-scale features. Our method shows improvements over existing methods on both the PASCAL VOC 2012 and MS COCO datasets, achieving state-of-the-art performance. [ABSTRACT FROM AUTHOR]

Script identification is an essential step in the text extraction pipeline for multi-lingual application. This paper presents an effective approach to identify scripts in scene text images. Due to the complicated background, various text styles, character similarity of different languages, script identification has not been solved yet. Under the general classification framework of script identification, we investigate two important components: feature extraction and classification layer. In the feature extraction, we utilize a hierarchical feature fusion block to extract the multi-scale features. Furthermore, we adopt an attention mechanism to obtain the local discriminative parts of feature maps. In the classification layer, we utilize a fully convolutional classifier to generate channel-level classifications which are then processed by a global pooling layer to improve classification efficiency. We evaluated the proposed approach on benchmark datasets of RRC-MLT2017, SIW-13, CVSI-2015 and MLe2e, and the experimental results show the effectiveness of each elaborate designed component. Finally, we achieve better performances than those competitive models, where the correct rates are 89.66%, 96.11%, 98.78% and 97.20% on PRC-MLT2017, SIW-13, CVSI-2015 and MLe2e, respectively. [ABSTRACT FROM AUTHOR]

Tag-aware recommender systems (TRS) utilize rich tagging information to better depict user portraits and item features. Recently, many efforts have been done to improve TRS with neural networks. However, existing methods construct user representations through either explicit tagging behaviors or implicit interacted items, which is inadequate to capture multi-aspect user preferences. Besides, there are still lacks of investigation about the intersection between user and item tags, which is crucial for better recommendation. In this paper, we propose AIRec, an attentive intersection model for TRS, to address the above issues. More precisely, we first project the sparse tag vectors into a latent space through multi-layer perceptron (MLP). Then, the user representations are constructed with a hierarchical attention network, where the item-level attention differentiates the contributions of interacted items and the preference-level attention discriminates the saliencies between explicit and implicit preferences. After that, the intersection between user and item tags is exploited to enhance the learning of conjunct features. Finally, the user and item representations are concatenated and fed to factorization machines (FM) for score prediction. We conduct extensive experiments on two real-world datasets, demonstrating significant improvements of AIRec over state-of-the-art methods for tag-aware top-n recommendation. [ABSTRACT FROM AUTHOR]

In the modern manufacturing industry, remaining useful life (RUL) prediction of the machining tools plays a significant role in promoting machining efficiency, ensuring product quality and reducing production costs. In recent years, many data-driven prognostic approaches have been developed for machining tools, but few of them have considered the operating conditions such as spindle load and rotating speed that may have great impact on the degradation behavior and sensor signals. It may give rise to more uncertainty and lead to an obvious decrease in prediction accuracy when operating condition changes. Besides, feature extraction from the raw signals that are nonstationary, nonlinear, and mixed with abundant noise is essential but quite challenging. To address these issues, this paper proposes a novel prognostic approach for machining tools under dynamic operating condition with varying spindle load. In the proposed approach, an adaptive variational mode decomposition (VMD) is newly developed to adaptively search the optimal parameters for processing the raw vibration data, then several components with good trendability and noise robustness are obtained for feature extraction. Furthermore, a deep learning model combining one-dimensional convolutional long short-term memory (LSTM) with attention mechanism is constructed to perform RUL prediction. Numerical experiments on a real-world case study show the effectiveness and superiority of the proposed approach in comparison with other baseline data-driven approaches. [ABSTRACT FROM AUTHOR]

Facial expression recognition is a hot research topic and can be applied in many computer vision fields, such as human–computer interaction, affective computing and so on. In this paper, we propose a novel end-to-end network with attention mechanism for automatic facial expression recognition. The new network architecture consists of four parts, i.e., the feature extraction module, the attention module, the reconstruction module and the classification module. The LBP features extract image texture information and then catch the small movements of the faces, which can improve the network performance. Attention mechanism can make the neural network pay more attention to useful features. We combine LBP features and attention mechanism to enhance the attention model to obtain better results. In addition, we collected and labelled a new facial expression dataset of seven expressions from 35 subjects aged from 20 to 25. For each subject, we captured both RGB images and depth images with a Microsoft Kinect sensor. For each image type, there are 245 image sequences, each of which contains 110 images, resulting in 26,950 images in total. We apply the newly proposed method to our own dataset and four representative expression datasets, i.e., JAFFE, CK+, FER2013 and Oulu-CASIA. The experimental results demonstrate the feasibility and effectiveness of the proposed method. [ABSTRACT FROM AUTHOR]

Modeling interpretable artificial intelligence (AI) for flood forecasting represents a serious challenge: both accuracy and interpretability are indispensable. Because of the uncertainty and nonlinearity of flood, existing hydrological solutions always achieve low prediction robustness while machine learning (ML) approaches neglect the physical interpretability of models. In this paper, we focus on the need for flood forecasting and propose an interpretable Spatio-Temporal Attention Long Short Term Memory model (STA-LSTM) based on LSTM and attention mechanism. We use dynamic attention mechanism and LSTM to build model, Max-Min method to normalize data, variable control method to select hyperparameters, and Adam algorithm to train the model. Emphasis is placed on the visualization and interpretation of attention weights. Experiment results on three small and medium basins in China suggest that the proposed STA-LSTM model outperforms Historical Average (HA), Fully Connected Network (FCN), Convolutional Neural Networks (CNN), Graph Convolutional Networks (GCN), original LSTM (LSTM), spatial attention LSTM (SA-LSTM), and temporal attention LSTM (TA-LSTM) in most cases. Visualization and interpretation of spatial and temporal attention weights reflect the reasonability of the proposed attention-based model. [ABSTRACT FROM AUTHOR]

With the prosperity of the location-based social networks, next point-of-interest (POI) recommendation has become an increasingly significant requirement since it can benefit both users and business. Obtaining insight into user mobility for the next POI recommendations is a vital yet challenging task. Existing approaches to understanding user mobility mainly gloss over the check-in sequence, making it fail to explicitly capture the subtle POI–POI interactions across the entire user check-in history and distinguish relevant check-ins from the irrelevant. In this paper, we proposed a novel recommendation approach, namely geographical-temporal awareness hierarchical attention network (GT-HAN) to resolve those issues. We first establish a geographical-temporal attention network to simultaneously uncover the overall sequence dependence and the subtle POI–POI relationships. Then, a context-specific co-attention network was designed to learn to change user preferences by adaptively selecting relevant check-in activities from check-in histories, which enabled GT-HAN to distinguish degrees of user preference for different check-ins. Finally, we make a POI recommendation using a conditional probability distribution function. Experimental results on real world datasets (obtained from Foursquare and Gowalla) show that the GT-HAN model significantly outperforms current state-of-the-art approaches, and demonstrating the benefits produced by new technologies incorporated into GT-HAN. [ABSTRACT FROM AUTHOR]

Deep neural networks have been employed to analyze the sentiment of text sequences and achieved significant effect. However, these models still face the issues of weakness of pre-trained word embeddings and weak interaction between the specific aspect and the context in attention mechanism. The pre-trained word embeddings lack the specific semantic information from the context. The weak interaction results in poor attention weights and produces limited aspect dependent sentiment representation in aspect-based sentiment analysis (ABSA). In this paper, we propose a novel end-to-end memory neural network, termed Recurrent Memory Neural Network (ReMemNN), to mitigate the above-mentioned problems. In ReMemNN, to tackle weakness of pre-trained word embeddings, a specially module named embedding adjustment learning module is designed to transfer the pre-trained word embeddings into adjustment word embeddings. To tackle weak interaction in attention mechanism, a multielement attention mechanism is designed to generate powerful attention weights and more precise aspect dependent sentiment representation. Besides, an explicit memory module is designed to store these different representations and generate hidden states and representations. Extensive experimental results on all datasets show that ReMemNN outperforms typical baselines and achieve the state-of-the-art performance. Besides, these experimental results also demonstrate that ReMemNN is language-independent and dataset type-independent. [ABSTRACT FROM AUTHOR]

Automatic heart sound auscultation is one of the common used techniques for cardiovascular diseases detection. In this paper, a novel heart sound classification method based on deep learning technologies for cardiovascular disease prediction is introduced, which is mainly comprised three parts: pre-processing, 1-D waveform heart sound patches classification using a deep convolutional neural network (CNN) with attention mechanism, and majority voting for final prediction of heart sound recordings. In order to enhance the information flow of the CNNs, a block-stacked style architecture with clique blocks is employed, and in each clique block a bidirectional connection structure is introduced in the proposed CNN. By using the stacked clique and transition blocks, the proposed CNN achieves both spatial and channel attention leading a promising classification performance. Moreover, a novel separable convolution with inverted bottleneck is utilized to decouple the spatial and channel-wise relevancy of features efficiently. Experiments on PhysioNet/CinC 2016 show that the proposed method obtains a superior classification results and excels in consumption of parameter comparing to state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Aspect-based sentiment classification aims to predict the sentiment polarity of an aspect term in a sentence instead of the sentiment polarity of the entire sentence. Neural networks have been used for this task, and most existing methods have adopted sequence models, which require more training time than other models. When an aspect term comprises several words, most methods involve a coarse-level attention mechanism to model the aspect, and this may result in information loss. In this paper, we propose a multi-attention network (MAN) to address the above problems. The proposed model uses intra- and inter-level attention mechanisms. In the former, the MAN employs a transformer encoder instead of a sequence model to reduce training time. The transformer encoder encodes the input sentence in parallel and preserves long-distance sentiment relations. In the latter, the MAN uses a global and a local attention module to capture differently grained interactive information between aspect and context. The global attention module focuses on the entire relation, whereas the local attention module considers interactions at word level; this was often neglected in previous studies. Experiments demonstrate that the proposed model achieves superior results when compared to the baseline models. [ABSTRACT FROM AUTHOR]

Semantic segmentation networks usually utilize special pyramid structure after encoder or combine low-level and high-level feature maps in decoder to capture multi-scale context information, which we term them feature combination and feature connection, respectively. However, both such frameworks are less valid with the fixed geometric structure or unsuitable interim. In this paper, we advocate Dynamic Attention Network (DAN) to solve these problems. First, we design a Deformable Attention Pyramid (DAP) module to perform a self-adjustable descriptor of high-level output, which utilizes deformable function to model geometric transformation. With DAP, semantic information can be captured effectively. Second, we propose a Fusing Attention Interim (FAI) module to guide the back-propagation of long-short range information in each level of decoder. We evaluate DAN on the challenging PASCAL VOC 2012 and Cityscapes segmentation benchmarks and find that it achieves state-of-the-art results without post-processing. Our observation can be concluded that the flexible structure that possesses dynamic attention mechanism is beneficial to learn multi-scale context information. [ABSTRACT FROM AUTHOR]

Keywords can express the main content of an article or a sentence. Keywords extraction is a critical issue in many Natural Language Processing (NLP) applications and can improve the performance of many NLP systems. The traditional methods of keywords extraction are based on machine learning or graph model. The performance of these methods is influenced by the feature selection and the manually defined rules. In recent years, with the emergence of deep learning technology, learning features automatically with the deep learning algorithm can improve the performance of many tasks. In this paper, we propose a deep neural network model for the task of keywords extraction. We make two extensions on the basis of traditional LSTM model. First, to better utilize both the historic and following contextual information of the given target word, we propose a target center-based LSTM model (TC-LSTM), which learns to encode the target word by considering its contextual information. Second, on the basis of TC-LSTM model, we apply the self-attention mechanism, which enables our model has an ability to focus on informative parts of the associated text. In addition, we also introduce a two-stage training method, which takes advantage of large-scale pseudo training data. Experimental results show the advantage of our method, our model beats all the baseline systems all across the board. And also, the two-stage training method is of great significance for improving the effectiveness of the model. [ABSTRACT FROM AUTHOR]

The identification of chromatin accessibility is a significant part of the genomics and genetics. However, high-throughput experimental techniques are costly and impractical for systematic identification of accessibility. Many computational methods were proposed to predict the functional regions of chromatin purely relying on DNA sequences, but they could not take full advantage of sequence information to capture hidden complex motifs among DNA sequences. Recently, deep learning algorithms have been incorporated into the chromatin accessibility predication and achieved the remarkable results. Nevertheless, there still exists a problem in chromatin accessibility prediction as how to effectively represent the complex features merely from DNA sequences. Thus, developing efficient computational methods is becoming increasingly urgent to identify functional regions of the genome. In this paper, combining convolutional and gated recurrent unit neural networks with attention mechanism, we develop a discriminative computational framework DeepANF to adaptively extract hidden pattern features and identify the chromatin accessibility based on distributed representation of DNA sequences. To verify the efficacy of the DeepANF framework, we conduct extensive experiments on five large scale datasets, and experimental results reveal that our framework not only consistently outperforms these published methods for chromatin accessibility prediction tasks, but also extracts more discriminative features from pure DNA sequences than published methods, especially on MCF-7 dataset. [ABSTRACT FROM AUTHOR]

• A novel Transformer-based framework is proposed for video object segmentation. • Target guidance block is explored for integrating temporal features. • Our method is superior to some existing video object segmentation methods. • Our method can deal with scale variance and distinguish similar objects. Video object segmentation automatically separates the interested objects from the background across a video sequence and was an active research area in recent years. The crucial challenge lies in investigating an effective architecture to fully exploit spatiotemporal correlation in a given video sequence for achieving accurate segmentation results. In this paper, we propose a novel semi-supervised Transformer-based framework called Target-guided Spatiotemporal Dual-stream Transformers (TSDT) with two separate streams to enable effective spatiotemporal context propagation. Technically, the temporal stream is used to aggregate rich temporal cues from past frames, while the spatial stream is trained to encode object location and appearance information stored in the current frame. To compress and integrate temporal features, a target guidance block (TGB) is designed to retrieve target information in the past video flow under the guidance of the current frame. The experimental results on video object segmentation benchmarks demonstrate the feasibility and effectiveness of the proposed framework. Codes and trained models are available at https://github.com/zhouweii234/TSDTVOS. [ABSTRACT FROM AUTHOR]

• We believe that integrating sentiment knowledge and aspects interaction is beneficial to help the model better understand the interactions of different aspects in sentences and the sentiment relations between words, which is ignored by the existing research. • We propose a model named GMF-SKIA. We first add sentiment knowledge information to word nodes through SenticNet. After that, we use an aspect-related self-attention mechanism to obtain inter-aspect feature information. Lastly, we use an information gate to dynamically fuse information. • We evaluate GMF-SKIA on four datasets, Rest14, Lap14, Rest15 and Rest16, our model outperforms the best benchmark model by an average of 2.1 % and achieves the highest accuracy of 91.56 % on the Rest16 dataset. Aspect level sentiment classification is a fine-grained sentiment analysis task that aims to identify the sentiment polarity of one or more given aspects in a sentence. In natural language, words frequently carry certain sentimental tendencies, which can be beneficial in obtaining the features between aspects and contexts. On the other hand, the dependencies between different aspects in a sentence can provide sufficient information for the sentiment polarity discrimination of a target aspect. However, existing models tend to focus on sentiment knowledge or aspect interactions individually without leveraging their converged information. Therefore, we propose a model based on G ated M echanism F using S entiment K nowledge and I nter- A spect dependency (GMF-SKIA) for Aspect-level Sentiment Classification in this paper, aiming to dynamically fuse sentiment knowledge information of words and inter-aspect dependency. Specifically, the model uses the SenticNet sentiment dictionary to add sentiment knowledge information to words during dependency tree construction, and then we introduce a graph convolutional network to obtain sentiment information of dependency tree. We utilize an aspect-related multiheaded self-attention mechanism to model the inter-aspect interactions. Moreover, we design an information gate based on gated mechanism to fuse sentiment knowledge and inter-aspect features. We performed experiments on four publicly available datasets, our model outperforms the best benchmark model by an average of 2.1 % and achieves the highest accuracy of 91.56 % on the Rest16 dataset. [ABSTRACT FROM AUTHOR]

• More comprehensive global context features are developed in our new model. • The CAA extracts mult-level visual features in a mutually beneficial way. • The DWC utilizes complementarity between multi-level visual features. • Extensive experiments show our method outperforms state-of-the-art models. Recently, image captioning models have made remarkable progress by introducing transformer architecture, which utilizes self-attention to explore intra- and inter-modal interactions. However, most existing methods only consider region-level characteristic during the attention weight calculation and ignore the image-level information. This seriously hinders the whole model from understanding the scene content. In this paper, we propose a Context-Aware Transformer (CATNet) with two novel designs, namely Context Augmented Attention (CAA) and Dual Way Controller (DWC). Concretely, CAA in encoder enables the extraction of more comprehensive visual representation through modeling the communications between multi-level visual features. DWC in decoder is used to enhance the fusion between visual features and language representation through utilizing complementarity of global context and local regions. Extensive experiments conducted on MSCOCO dataset show that the proposed CATNet has achieved state-of-the-art performance on both Karpathy test set and online test. [ABSTRACT FROM AUTHOR]

• An efficient multimodal feature fusion network for audio-visual speech enhancement. • The first to use fine-grained 3D lip landmarks as auxiliary visual information. • The MEM based on a multi-branch structure can capture critical multi-scale features. • The AFM utilizes a mutually reinforcing strategy to fuse visual and audio features. • An exhaustive experimental study on two audio-visual speech enhancement datasets. Traditional speech enhancement uses only audio signals to obtain clean speech by removing background noise. Audio-visual speech enhancement uses additional visual information to improve the intelligibility and perceptual quality of noisy speech, which can be applied to video conferencing. However, the original visual cues contain redundant information, which causes high latency and makes the model prone to overfitting. Meanwhile, related methods do not pay much attention to exploiting critical visual features and fusing multimodal audio-visual features. In this paper, we propose an efficient multimodal feature fusion network (MFF-Net) for audio-visual speech enhancement. Specifically, we are the first to use fine-grained 3D lip landmarks to represent visual features. This visual representation can provide refined 3D visual information while protecting privacy. We design a multi-scale enhancement module (MEM) with a multi-branch structure that extracts critical multi-scale features by using dilated convolutions and attention mechanisms. Besides, we propose an audio-visual fusion module (AFM) that uses a mutually reinforcing strategy to effectively fuse visual and audio features. To verify the effectiveness of our method, we construct two speech enhancement datasets based on lip landmarks and conduct related experiments. Extensive experimental results show that our proposed MFF-Net has competitive performance compared to other methods. [ABSTRACT FROM AUTHOR]

• Construct modality-shared and modality-specific encoders that effectively learn shared and specific feature representations of modalities. • Propose an end-to-end multimodal representation and fusion method for multimodal intent recognition. • Propose an adaptive multimodal fusion method based on the attention-based gated neural network, which can distinguish the contributions of different modalities and reduce possible noise. • Experimental results show that the model outperforms state-of-the-art models on multiple evaluation metrics. Intent recognition is a crucial task in natural language understanding. Current research mainly focuses on task-specific unimodal intent recognition. However, in real-world scenes, human intentions are complex and need to be judged by integrating information such as speech, tone, expression, and action. Therefore, this paper proposes an effective multimodal representation and fusion method (EMRFM) for intent recognition in real-world multimodal scenes. First, text, audio, and vision features are extracted based on pre-trained BERT, Wav2vec 2.0, and Faster R-CNN. Then, considering the complementarity and consistency among the modalities, the modality-shared and modality-specific encoders are constructed to learn shared and specific feature representations of the modalities. Finally, an adaptive multimodal fusion method based on an attention-based gated neural network is designed to eliminate noise features. Comprehensive experiments are conducted on the multimodal intent recognition MIntRec benchmark dataset. Our proposed model achieves higher accuracy, precision, recall, and F1-score than state-of-the-art multimodal learning methods. We also conduct multimodal sentiment recognition experiments on the CMU-MOSI dataset, and our model still outperforms state-of-the-art methods. In addition, the experiment demonstrates that the model's multimodal representation well learned the modality's shared and specific features. The multimodal fusion of the model achieves adaptive fusion and effectively reduces possible noise interference. [ABSTRACT FROM AUTHOR]

Feature engineering is a classical problem in recommender systems, and feature interactions is one of the most important parts of feature engineering. Factorization based models are widely used for explicit feature interactions. However, most current works utilize separate features to model cross features. Such a pattern limits the significance of cross features, since realistic recommendation scenarios are rich in associations between features. In this paper, we classify the basic feature interactions into sum-interaction and product-interaction, and improve the current general strategy of explicit feature interactions. Based on these theoretical studies, we propose a novel explicit feature interactions model Attentional Aggregative Interaction Network (AAIN), which models higher-order features using a cyclic explicit module. Specifically, we introduce attention mechanism for the reorganization of separate features, followed by product-interaction and higher-order features' compression and output. The model is efficient since: 1) AAIN automatically learns high-order feature interactions and filters them with different weights. 2) AAIN optimizes the interaction between features into the interaction between feature groups, which allows for other relevant information to be considered when performing interactions. Furthermore, we integrate AAIN model with the classical deep neural network (DNN) model into a new model Deep Attentional Aggregative Interaction Network (DAAIN). Experiments on real-world datasets show that our models achieve state-of-the-art results. [ABSTRACT FROM AUTHOR]

Highlights • For the convolutional layer, the convolution window size and the stride size affect the classification performance. • BiLSTM and attention mechanism have greater effects than the convolutional layer on the classification accuracy. • Compared to pre-trained embedding word vector, random embedding word vector requires to train more parameters and it causes relatively lower classification accuracy in limited iterations. • Our proposed approach performs better than some state-of-the-art DNNs. Abstract Neural network models have been widely used in the field of natural language processing (NLP). Recurrent neural networks (RNNs), which have the ability to process sequences of arbitrary length, are common methods for sequence modeling tasks. Long short-term memory (LSTM) is one kind of RNNs and has achieved remarkable performance in text classification. However, due to the high dimensionality and sparsity of text data, and to the complex semantics of the natural language, text classification presents difficult challenges. In order to solve the above problems, a novel and unified architecture which contains a bidirectional LSTM (BiLSTM), attention mechanism and the convolutional layer is proposed in this paper. The proposed architecture is called attention-based bidirectional long short-term memory with convolution layer (AC-BiLSTM). In AC-BiLSTM, the convolutional layer extracts the higher-level phrase representations from the word embedding vectors and BiLSTM is used to access both the preceding and succeeding context representations. Attention mechanism is employed to give different focus to the information outputted from the hidden layers of BiLSTM. Finally, the softmax classifier is used to classify the processed context information. AC-BiLSTM is able to capture both the local feature of phrases as well as global sentence semantics. Experimental verifications are conducted on six sentiment classification datasets and a question classification dataset, including detailed analysis for AC-BiLSTM. The results clearly show that AC-BiLSTM outperforms other state-of-the-art text classification methods in terms of the classification accuracy. [ABSTRACT FROM AUTHOR]