| Title: |
Shared functional specialization in transformer-based language models and the human brain. |
| Authors: |
Kumar, Sreejan, Sumers, Theodore R., Yamakoshi, Takateru, Goldstein, Ariel, Hasson, Uri, Norman, Kenneth A., Griffiths, Thomas L., Hawkins, Robert D., Nastase, Samuel A. |
| Source: |
Nature Communications; 6/29/2024, Vol. 15 Issue 1, p1-19, 19p |
| Subject Terms: |
LANGUAGE models, NATURAL language processing, ARTIFICIAL neural networks, FUNCTIONAL magnetic resonance imaging, NATURAL languages |
| Abstract: |
When processing language, the brain is thought to deploy specialized computations to construct meaning from complex linguistic structures. Recently, artificial neural networks based on the Transformer architecture have revolutionized the field of natural language processing. Transformers integrate contextual information across words via structured circuit computations. Prior work has focused on the internal representations ("embeddings") generated by these circuits. In this paper, we instead analyze the circuit computations directly: we deconstruct these computations into the functionally-specialized "transformations" that integrate contextual information across words. Using functional MRI data acquired while participants listened to naturalistic stories, we first verify that the transformations account for considerable variance in brain activity across the cortical language network. We then demonstrate that the emergent computations performed by individual, functionally-specialized "attention heads" differentially predict brain activity in specific cortical regions. These heads fall along gradients corresponding to different layers and context lengths in a low-dimensional cortical space. The extent to which transformer-based language models provide a good model of human brain activity during natural language comprehension is unclear. Here, the authors show that the internal transformations performed by the network predict brain activity in cortical language network as measured by fMRI. [ABSTRACT FROM AUTHOR] |
|
Copyright of Nature Communications is the property of Springer Nature and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.) |
| Database: |
Complementary Index |
