Search

Gated recurrent unit (GRU) is a variant of the recurrent neural network (RNN). It has been widely used in many applications, such as handwriting recognition and natural language processing. However, GRU can only memorize the sequential information, but lacks the capability of adaptively paying attention to important parts in the sequences. In this paper, we propose a novel RNN model, called recurrent attention unit (RAU), which can seamlessly integrate the attention mechanism into the interior of the GRU cell by adding an attention gate. The attention gate enhances the ability of RAU to remember long-term information and pay attention to important parts in the sequential data. Extensive experiments on adding problem, image classification, sentiment classification and language modeling show that RAU consistently outperforms GRU and other related models. [ABSTRACT FROM AUTHOR]

With the increasing attention to the energy crisis and global warming, solar generation has become an important way to use clean solar energy and is playing an increasingly important role. Due to the highly-variable patterns of solar generation, the estimation of prediction intervals is receiving more attention, which is conducive to the safe and stable operation of the power system. In order to further improve the performance of prediction intervals of solar generation, this paper proposes a prediction intervals estimation method for solar generation based on gated recurrent unit (GRU) neural networks and kernel density estimation (KDE). GRU, a commonly used recurrent neural networks, is utilized to obtain the deterministic forecast of solar generation. In addition, according to the characteristics of solar generation, attention mechanism is designed on the GRU prediction model to further improve the prediction performance. Then, the KDE method is used to fit the prediction errors of solar generation obtained by the deterministic forecasting method. In order to verify the effectiveness of the proposed method, we have carried out a large number of experiments on freely available datasets. The experimental results show that the proposed method outperforms competing methods and can generate high-quality prediction intervals. [ABSTRACT FROM AUTHOR]

Attention has arguably become one of the most important concepts in the deep learning field. It is inspired by the biological systems of humans that tend to focus on the distinctive parts when processing large amounts of information. With the development of deep neural networks, attention mechanism has been widely used in diverse application domains. This paper aims to give an overview of the state-of-the-art attention models proposed in recent years. Toward a better general understanding of attention mechanisms, we define a unified model that is suitable for most attention structures. Each step of the attention mechanism implemented in the model is described in detail. Furthermore, we classify existing attention models according to four criteria: the softness of attention, forms of input feature, input representation, and output representation. Besides, we summarize network architectures used in conjunction with the attention mechanism and describe some typical applications of attention mechanism. Finally, we discuss the interpretability that attention brings to deep learning and present its potential future trends. [ABSTRACT FROM AUTHOR]

Deep neural networks have been employed to analyze the sentiment of text sequences and achieved significant effect. However, these models still face the issues of weakness of pre-trained word embeddings and weak interaction between the specific aspect and the context in attention mechanism. The pre-trained word embeddings lack the specific semantic information from the context. The weak interaction results in poor attention weights and produces limited aspect dependent sentiment representation in aspect-based sentiment analysis (ABSA). In this paper, we propose a novel end-to-end memory neural network, termed Recurrent Memory Neural Network (ReMemNN), to mitigate the above-mentioned problems. In ReMemNN, to tackle weakness of pre-trained word embeddings, a specially module named embedding adjustment learning module is designed to transfer the pre-trained word embeddings into adjustment word embeddings. To tackle weak interaction in attention mechanism, a multielement attention mechanism is designed to generate powerful attention weights and more precise aspect dependent sentiment representation. Besides, an explicit memory module is designed to store these different representations and generate hidden states and representations. Extensive experimental results on all datasets show that ReMemNN outperforms typical baselines and achieve the state-of-the-art performance. Besides, these experimental results also demonstrate that ReMemNN is language-independent and dataset type-independent. [ABSTRACT FROM AUTHOR]

The identification of chromatin accessibility is a significant part of the genomics and genetics. However, high-throughput experimental techniques are costly and impractical for systematic identification of accessibility. Many computational methods were proposed to predict the functional regions of chromatin purely relying on DNA sequences, but they could not take full advantage of sequence information to capture hidden complex motifs among DNA sequences. Recently, deep learning algorithms have been incorporated into the chromatin accessibility predication and achieved the remarkable results. Nevertheless, there still exists a problem in chromatin accessibility prediction as how to effectively represent the complex features merely from DNA sequences. Thus, developing efficient computational methods is becoming increasingly urgent to identify functional regions of the genome. In this paper, combining convolutional and gated recurrent unit neural networks with attention mechanism, we develop a discriminative computational framework DeepANF to adaptively extract hidden pattern features and identify the chromatin accessibility based on distributed representation of DNA sequences. To verify the efficacy of the DeepANF framework, we conduct extensive experiments on five large scale datasets, and experimental results reveal that our framework not only consistently outperforms these published methods for chromatin accessibility prediction tasks, but also extracts more discriminative features from pure DNA sequences than published methods, especially on MCF-7 dataset. [ABSTRACT FROM AUTHOR]

Highlights • For the convolutional layer, the convolution window size and the stride size affect the classification performance. • BiLSTM and attention mechanism have greater effects than the convolutional layer on the classification accuracy. • Compared to pre-trained embedding word vector, random embedding word vector requires to train more parameters and it causes relatively lower classification accuracy in limited iterations. • Our proposed approach performs better than some state-of-the-art DNNs. Abstract Neural network models have been widely used in the field of natural language processing (NLP). Recurrent neural networks (RNNs), which have the ability to process sequences of arbitrary length, are common methods for sequence modeling tasks. Long short-term memory (LSTM) is one kind of RNNs and has achieved remarkable performance in text classification. However, due to the high dimensionality and sparsity of text data, and to the complex semantics of the natural language, text classification presents difficult challenges. In order to solve the above problems, a novel and unified architecture which contains a bidirectional LSTM (BiLSTM), attention mechanism and the convolutional layer is proposed in this paper. The proposed architecture is called attention-based bidirectional long short-term memory with convolution layer (AC-BiLSTM). In AC-BiLSTM, the convolutional layer extracts the higher-level phrase representations from the word embedding vectors and BiLSTM is used to access both the preceding and succeeding context representations. Attention mechanism is employed to give different focus to the information outputted from the hidden layers of BiLSTM. Finally, the softmax classifier is used to classify the processed context information. AC-BiLSTM is able to capture both the local feature of phrases as well as global sentence semantics. Experimental verifications are conducted on six sentiment classification datasets and a question classification dataset, including detailed analysis for AC-BiLSTM. The results clearly show that AC-BiLSTM outperforms other state-of-the-art text classification methods in terms of the classification accuracy. [ABSTRACT FROM AUTHOR]

Abstract Recurrent Neural Network (RNN) based approaches have recently shown promising in tackling Natural Language Generation (NLG) problems. This paper presents an approach to leverage gating mechanisms, in which we incrementally propose three additional semantic cells into a traditional RNN model: a Refinement cell to filter the sequential inputs before RNN computations, an Adjustment cell, and an Output cell to select semantic elements and gate a feature vector during generation. The proposed gating-based generators can learn from unaligned data by jointly training both sentence planning and surface realization to generate natural language utterances. We conducted extensive experiments on four different NLG domains in which the results empirically show that the proposed methods not only achieved better performance on all the NLG domains in comparison with previous gating-based, attention-based methods, but also obtained highly competitive results compared to a hybrid generator. [ABSTRACT FROM AUTHOR]

Long short-term memory (LSTM) is an extension of the recurrent neural network (RNN) and has achieved excellent performance in various tasks, especially sequential problems. The LSTM is chain-structured, and its architecture is limited by sequential information propagation. In practice, it is hard to solve the problems involving very long term dependencies. Recent studies have indicated that adding recurrent skip connections across multiple timescales may help the RNN improve its performance in long-term dependencies. Moreover, capturing local features can improve the performance of the RNN. In this paper, we propose a novel architecture (Att-LSTM) for the LSTM, which connects continuous hidden states of previous time steps to the current time step and applies an attention mechanism to these hidden states. This architecture can not only capture local features effectively but also help learn very long-distance correlations in an input sequence. We evaluate Att-LSTM in various sequential tasks, such as adding problem, sequence classification, and character-level language modeling. In addition, to prove the generalization and practicality of the novel architecture, we design a character-level hierarchical Att-LSTM and refine the word representation with a highway network. This hierarchical model achieved excellent performance on question classification. [ABSTRACT FROM AUTHOR]