Back to Search Start Over

Shared functional specialization in transformer-based language models and the human brain.

Authors :
Kumar S
Sumers TR
Yamakoshi T
Goldstein A
Hasson U
Norman KA
Griffiths TL
Hawkins RD
Nastase SA
Source :
Nature communications [Nat Commun] 2024 Jun 29; Vol. 15 (1), pp. 5523. Date of Electronic Publication: 2024 Jun 29.
Publication Year :
2024

Abstract

When processing language, the brain is thought to deploy specialized computations to construct meaning from complex linguistic structures. Recently, artificial neural networks based on the Transformer architecture have revolutionized the field of natural language processing. Transformers integrate contextual information across words via structured circuit computations. Prior work has focused on the internal representations ("embeddings") generated by these circuits. In this paper, we instead analyze the circuit computations directly: we deconstruct these computations into the functionally-specialized "transformations" that integrate contextual information across words. Using functional MRI data acquired while participants listened to naturalistic stories, we first verify that the transformations account for considerable variance in brain activity across the cortical language network. We then demonstrate that the emergent computations performed by individual, functionally-specialized "attention heads" differentially predict brain activity in specific cortical regions. These heads fall along gradients corresponding to different layers and context lengths in a low-dimensional cortical space.<br /> (© 2024. The Author(s).)

Details

Language :
English
ISSN :
2041-1723
Volume :
15
Issue :
1
Database :
MEDLINE
Journal :
Nature communications
Publication Type :
Academic Journal
Accession number :
38951520
Full Text :
https://doi.org/10.1038/s41467-024-49173-5