Search

Your search keyword '"Processament de la parla"' showing total 4 results
Start Over Descriptor "Processament de la parla" Publisher association for computational linguistics
4 results on '"Processament de la parla"'

1. Measuring the mixing of contextual information in the transformer

Author

Universitat Politècnica de Catalunya. Doctorat en Intel·ligència Artificial, Universitat Politècnica de Catalunya. Doctorat en Teoria del Senyal i Comunicacions, Universitat Politècnica de Catalunya. IDEAI-UPC - Intelligent Data sciEnce and Artificial Intelligence Research Group, Ferrando Monsonís, Javier, Gallego Olsina, Gerard Ion, Ruiz Costa-Jussà, Marta, Universitat Politècnica de Catalunya. Doctorat en Intel·ligència Artificial, Universitat Politècnica de Catalunya. Doctorat en Teoria del Senyal i Comunicacions, Universitat Politècnica de Catalunya. IDEAI-UPC - Intelligent Data sciEnce and Artificial Intelligence Research Group, Ferrando Monsonís, Javier, Gallego Olsina, Gerard Ion, and Ruiz Costa-Jussà, Marta

Abstract

The Transformer architecture aggregates input information through the self-attention mechanism, but there is no clear understanding of how this information is mixed across the entire model. Additionally, recent works have demonstrated that attention weights alone are not enough to describe the flow of information. In this paper, we consider the whole attention block --multi-head attention, residual connection, and layer normalization-- and define a metric to measure token-to-token interactions within each layer. Then, we aggregate layer-wise interpretations to provide input attribution scores for model predictions. Experimentally, we show that our method, ALTI (Aggregation of Layer-wise Token-to-token Interactions), provides more faithful explanations and increased robustness than gradient-based methods., Javier Ferrando and Gerard I. Gállego are supported by the Spanish Ministerio de Ciencia e Innovación through the project PID2019-107579RB-I00 / AEI / 10.13039/501100011033., Peer Reviewed, Postprint (published version)

Published
2022

2. Multilingual, multi-scale and multi-layer visualization of sequence-based intermediate representations

Author

Universitat Politècnica de Catalunya. Doctorat en Teoria del Senyal i Comunicacions, Universitat Politècnica de Catalunya. Departament de Ciències de la Computació, Universitat Politècnica de Catalunya. VEU - Grup de Tractament de la Parla, Universitat Politècnica de Catalunya. ViRVIG - Grup de Recerca en Visualització, Realitat Virtual i Interacció Gràfica, Escolano Peinado, Carlos, Ruiz Costa-Jussà, Marta, Lacroux, Elora, Vázquez Alcocer, Pere Pau, Universitat Politècnica de Catalunya. Doctorat en Teoria del Senyal i Comunicacions, Universitat Politècnica de Catalunya. Departament de Ciències de la Computació, Universitat Politècnica de Catalunya. VEU - Grup de Tractament de la Parla, Universitat Politècnica de Catalunya. ViRVIG - Grup de Recerca en Visualització, Realitat Virtual i Interacció Gràfica, Escolano Peinado, Carlos, Ruiz Costa-Jussà, Marta, Lacroux, Elora, and Vázquez Alcocer, Pere Pau

Abstract

The main alternatives nowadays to dealwith sequences are Recurrent Neural Net-works (RNN), Convolutional Neural Networks(CNN) architectures and the Transformer. Inthis context, RNN’s, CNN’s and Transformerhave most commonly been used as an encoder-decoder architecture with multiple layers ineach module. Far beyond this, these architec-tures are the basis for the contextual word em-beddings which are revolutionizing most natural language downstream applications. However, intermediate layer representations insequence-based architectures can be difficultto interpret. To make each layer representation within these architectures more accessible and meaningful, we introduce a web-based toolthat visualizes them both at the sentence and token level. We present three use cases. The first analyses gender issues in contextual worde mbeddings. The second and third are show-ing multilingual intermediate representations for sentences and tokens and the evolution of these intermediate representations along the multiple layers of the decoder and in the con-text of multilingual machine translation., This work is supported by a Google Faculty Research Award. This workis also supported by the Spanish Ministerio de Economía y Competitividad, the European Regional Development Fund and the Agencia Estatal de Investigación, through the post-doctoral senior grant Ramón y Cajal, contracts TEC2015-69266-P and TIN2017-88515-C2-1-R(GEN3DLIVE) (MINECO/FEDER,EU), and contract PCIN-2017-079 (AEI/MINECO)., Peer Reviewed, Postprint (published version)

Published
2019

3. Multilingual, multi-scale and multi-layer visualization of sequence-based intermediate representations

Author

Escolano Peinado, Carlos, Ruiz Costa-Jussà, Marta, Lacroux, Elora, Vázquez Alcocer, Pere Pau, Universitat Politècnica de Catalunya. Doctorat en Teoria del Senyal i Comunicacions, Universitat Politècnica de Catalunya. Departament de Ciències de la Computació, Universitat Politècnica de Catalunya. VEU - Grup de Tractament de la Parla, and Universitat Politècnica de Catalunya. ViRVIG - Grup de Recerca en Visualització, Realitat Virtual i Interacció Gràfica

Subjects
natural language processing, Machine learning, Aprenentatge automàtic, Recurrent Neural Net-works, Processament de la parla, Speech processing systems, Encoder-decoder architecture, Enginyeria de la telecomunicació [Àrees temàtiques de la UPC]
Abstract

The main alternatives nowadays to dealwith sequences are Recurrent Neural Net-works (RNN), Convolutional Neural Networks(CNN) architectures and the Transformer. Inthis context, RNN’s, CNN’s and Transformerhave most commonly been used as an encoder-decoder architecture with multiple layers ineach module. Far beyond this, these architec-tures are the basis for the contextual word em-beddings which are revolutionizing most natural language downstream applications. However, intermediate layer representations insequence-based architectures can be difficultto interpret. To make each layer representation within these architectures more accessible and meaningful, we introduce a web-based toolthat visualizes them both at the sentence and token level. We present three use cases. The first analyses gender issues in contextual worde mbeddings. The second and third are show-ing multilingual intermediate representations for sentences and tokens and the evolution of these intermediate representations along the multiple layers of the decoder and in the con-text of multilingual machine translation. This work is supported by a Google Faculty Research Award. This workis also supported by the Spanish Ministerio de Economía y Competitividad, the European Regional Development Fund and the Agencia Estatal de Investigación, through the post-doctoral senior grant Ramón y Cajal, contracts TEC2015-69266-P and TIN2017-88515-C2-1-R(GEN3DLIVE) (MINECO/FEDER,EU), and contract PCIN-2017-079 (AEI/MINECO).

Published
2019

4. Multilingual, multi-scale and multi-layer visualization of sequence-based intermediate representations

Author

Escolano Peinado, Carlos|||0000-0001-6657-673X, Ruiz Costa-Jussà, Marta|||0000-0002-5703-520X, Lacroux, Elora, and Vázquez Alcocer, Pere Pau|||0000-0003-4638-4065

Subjects
natural language processing, Machine learning, Aprenentatge automàtic, Recurrent Neural Net-works, Processament de la parla, Speech processing systems, Encoder-decoder architecture, Enginyeria de la telecomunicació [Àrees temàtiques de la UPC]
Abstract

The main alternatives nowadays to dealwith sequences are Recurrent Neural Net-works (RNN), Convolutional Neural Networks(CNN) architectures and the Transformer. Inthis context, RNN’s, CNN’s and Transformerhave most commonly been used as an encoder-decoder architecture with multiple layers ineach module. Far beyond this, these architec-tures are the basis for the contextual word em-beddings which are revolutionizing most natural language downstream applications. However, intermediate layer representations insequence-based architectures can be difficultto interpret. To make each layer representation within these architectures more accessible and meaningful, we introduce a web-based toolthat visualizes them both at the sentence and token level. We present three use cases. The first analyses gender issues in contextual worde mbeddings. The second and third are show-ing multilingual intermediate representations for sentences and tokens and the evolution of these intermediate representations along the multiple layers of the decoder and in the con-text of multilingual machine translation. This work is supported by a Google Faculty Research Award. This workis also supported by the Spanish Ministerio de Economía y Competitividad, the European Regional Development Fund and the Agencia Estatal de Investigación, through the post-doctoral senior grant Ramón y Cajal, contracts TEC2015-69266-P and TIN2017-88515-C2-1-R(GEN3DLIVE) (MINECO/FEDER,EU), and contract PCIN-2017-079 (AEI/MINECO).

Published
2019

Catalog

Books, media, physical & digital resources
See catalog results