Recent advancements in artificial intelligence language models have demonstrated remarkable capabilities in various linguistic tasks. These cutting-edge systems particularly excel at predicting subsequent words in text sequences, a technology that powers the autocomplete features in search engines and messaging applications.
The latest generation of predictive language models goes beyond simple word prediction, developing an understanding of language's underlying meaning. These sophisticated systems can perform complex tasks that appear to require genuine comprehension, including answering questions, summarizing documents, and completing narratives.
Although these models were specifically designed to optimize text prediction without mimicking human brain processes, groundbreaking research from MIT neuroscientists reveals striking parallels between how these AI systems function and how the human brain processes language.
Interestingly, computer models that perform well on other language tasks don't demonstrate this same similarity to the human brain, suggesting that the human brain might employ next-word prediction as a fundamental mechanism for language processing.
"The more accurately a model predicts the next word, the more closely it aligns with human brain activity," explains Nancy Kanwisher, the Walter A. Rosenblith Professor of Cognitive Neuroscience at MIT, a member of MIT's McGovern Institute for Brain Research and Center for Brains, Minds, and Machines (CBMM), and co-author of the study. "It's remarkable how well these models correspond to brain function, indirectly suggesting that the human language system might operate by continuously anticipating what comes next."
The study, published this week in the Proceedings of the National Academy of Sciences, was senior-authored by Joshua Tenenbaum, a professor of computational cognitive science at MIT and member of CBMM and MIT's Artificial Intelligence Laboratory (CSAIL); and Evelina Fedorenko, the Frederick A. and Carole J. Middleton Career Development Associate Professor of Neuroscience and member of the McGovern Institute. Martin Schrimpf, an MIT graduate student working in CBMM, served as the paper's first author.
Powerful Predictive Capabilities
The advanced next-word prediction models belong to a category known as deep neural networks. These networks consist of computational "nodes" that form connections with varying strengths, organized in layers that transmit information according to specific rules.
Over the past decade, researchers have employed deep neural networks to develop vision models capable of recognizing objects as effectively as the primate brain. MIT research has previously demonstrated that the underlying function of visual object recognition models corresponds to the organization of the primate visual cortex, despite these computer models not being explicitly designed to replicate brain function.
In this innovative study, the MIT team applied a similar methodology to compare language-processing regions in the human brain with language-processing models. The scientists analyzed 43 different language models, including several optimized for next-word prediction. These models featured GPT-3 (Generative Pre-trained Transformer 3), which can generate human-like text given a prompt, alongside other models designed for various language tasks, such as filling blanks in sentences.
As each model processed word sequences, the researchers measured activity across the network's nodes. They then compared these patterns to human brain activity, recorded as subjects performed three language tasks: listening to stories, reading sentences individually, and reading sentences with words revealed one at a time. These human datasets included functional magnetic resonance (fMRI) data and intracranial electrocorticographic measurements from patients undergoing epilepsy-related brain surgery.
The researchers discovered that the most accurate next-word prediction models displayed activity patterns remarkably similar to those observed in the human brain. Activity in these same models also strongly correlated with human behavioral measures, such as reading speed.
"We found that models which effectively predict neural responses also tend to best predict human behavioral responses, specifically reading times. Both of these correlations are explained by the model's performance in next-word prediction. This triangular relationship effectively connects all these elements," Schrimpf explains.
"A crucial insight from this research is that language processing represents a highly constrained problem: The optimal solutions developed by AI engineers ultimately resemble those discovered by the evolutionary processes that shaped the human brain. Since the AI network wasn't explicitly designed to mimic the brain—yet ends up appearing brain-like—this suggests a form of convergent evolution has occurred between artificial intelligence and nature," notes Daniel Yamins, an assistant professor of psychology and computer science at Stanford University, who wasn't involved in the study.
Transformative Discovery
Among the key computational features of predictive models like GPT-3 is the forward one-way predictive transformer. This component can forecast upcoming elements based on previous sequences, with the significant ability to make predictions using extensive prior context (hundreds of words) rather than just the most recent words.
Scientists haven't yet identified brain circuits or learning mechanisms that directly correspond to this type of processing, Tenenbaum acknowledges. However, the new findings align with previously proposed hypotheses suggesting prediction serves as a critical function in language processing.
"One of language processing's fundamental challenges is its real-time nature," he states. "Language flows continuously, requiring listeners to process and comprehend it instantaneously."
The researchers now intend to develop variations of these language processing models to examine how minor architectural modifications affect their performance and their capacity to match human neural data.
"For me, this result represents a game-changing discovery," Fedorenko remarks. "It's completely transforming my research approach, as I wouldn't have anticipated that within my lifetime we would develop sufficiently explicit computational models that capture enough about brain function to leverage them in understanding how the brain operates."
The team also plans to integrate these high-performing language models with computer models previously developed in Tenenbaum's lab that can perform other tasks, such as constructing perceptual representations of the physical world.
"If we can understand what these language models do and how they connect to models that perform tasks more akin to perception and thinking, we might develop more integrative models of brain function," Tenenbaum explains. "This could lead to improved artificial intelligence systems, as well as better models of how broader brain functions operate and how general intelligence emerges—advancing beyond what we've previously achieved."
The research received funding from a Takeda Fellowship; the MIT Shoemaker Fellowship; the Semiconductor Research Corporation; the MIT Media Lab Consortia; the MIT Singleton Fellowship; the MIT Presidential Graduate Fellowship; the Friends of the McGovern Institute Fellowship; the MIT Center for Brains, Minds, and Machines, through the National Science Foundation; the National Institutes of Health; MIT's Department of Brain and Cognitive Sciences; and the McGovern Institute.
Additional paper authors include Idan Blank PhD '16 and graduate students Greta Tuckute, Carina Kauf, and Eghbal Hosseini.
