Multilingual, multi-scale and multi-layer visualization of sequence-based intermediate representations

The main alternatives nowadays to dealwith sequences are Recurrent Neural Net-works (RNN), Convolutional Neural Networks(CNN) architectures and the Transformer. Inthis context, RNN’s, CNN’s and Transformerhave most commonly been used as an encoder-decoder architecture with multiple layers ineach module. Far beyond this, these architec-tures are the basis for the contextual word em-beddings which are revolutionizing most natural language downstream applications. However, intermediate layer representations insequence-based architectures can be difficultto interpret. To make each layer representation within these architectures more accessible and meaningful, we introduce a web-based toolthat visualizes them both at the sentence and token level. We present three use cases. The first analyses gender issues in contextual worde mbeddings. The second and third are show-ing multilingual intermediate representations for sentences and tokens and the evolution of these intermediate representations along the multiple layers of the decoder and in the con-text of multilingual machine translation. This work is supported by a Google Faculty Research Award. This workis also supported by the Spanish Ministerio de Economía y Competitividad, the European Regional Development Fund and the Agencia Estatal de Investigación, through the post-doctoral senior grant Ramón y Cajal, contracts TEC2015-69266-P and TIN2017-88515-C2-1-R(GEN3DLIVE) (MINECO/FEDER,EU), and contract PCIN-2017-079 (AEI/MINECO).