Back to Search
Start Over
Self attention mechanism of bidirectional information enhancement
- Source :
- Applied Intelligence. 52:2530-2538
- Publication Year :
- 2021
- Publisher :
- Springer Science and Business Media LLC, 2021.
-
Abstract
- Self attention mechanism is widely used in relation extraction, emotion classification and other tasks. It can extract a wide range of relevance information in the text. The attention mode of the existing self attention mechanism is soft attention mode, that is, a dense attention matrix is generated by softmax function. However, if the sentence length is long, the weight of important information will be too small. At the same time, the softmax function assumes that all elements have a positive impact on the results by default, which makes the model unable to extract the negative effect information. We use hard attention mechanism, namely sparse attention matrix, to improve the existing self attention model and fully extract the positive and negative information of text. Our model can not only enhance the extraction of positive information, but also makes up for the blank that the traditional attention matrix cannot be negative. We evaluated our model in three tasks and seven data sets. The experimental results show that our model is superior to the traditional self attention model and superior to state-of-the-art models in some tasks.
- Subjects :
- Computer science
Mechanism (biology)
business.industry
Emotion classification
Mode (statistics)
Machine learning
computer.software_genre
Relationship extraction
Matrix (mathematics)
Range (mathematics)
Artificial Intelligence
Softmax function
Relevance (information retrieval)
Artificial intelligence
business
computer
Subjects
Details
- ISSN :
- 15737497 and 0924669X
- Volume :
- 52
- Database :
- OpenAIRE
- Journal :
- Applied Intelligence
- Accession number :
- edsair.doi...........393e0664ca2c5248792260590d909a25