Deep learning neural networks are transforming our technological landscape, powering everything from smartphone voice assistants to self-driving vehicles that can identify and navigate around obstacles. Despite these remarkable advances, the development of deep learning networks has largely been a process of experimentation and discovery. MIT researchers have recently published their comprehensive review of contributions to the theoretical understanding of deep learning neural networks, offering valuable insights for the field's future direction.
“Deep learning emerged somewhat accidentally,” notes Tommy Poggio, investigator at the McGovern Institute for Brain Research, director of the Center for Brains, Minds, and Machines (CBMM), and the Eugene McDermott Professor in Brain and Cognitive Sciences. “We still lack fundamental understanding of why it functions effectively. A theoretical framework is developing, and I believe we're approaching a comprehensive theory. It's crucial to step back and examine recent breakthroughs.”
Navigating High-Dimensional Data Challenges
Our contemporary era is characterized by an unprecedented abundance of data — information from affordable sensors, digital text, internet content, and vast genomic datasets in biological sciences. Modern computers process these multidimensional datasets, creating what mathematician Richard Bellman termed the “curse of dimensionality.”
One significant challenge is that representing smooth, high-dimensional functions demands an astronomical number of parameters. While we know deep neural networks excel at learning to represent or approximate such complex data, the underlying reasons remain unclear. Understanding these mechanisms could potentially accelerate advancements in deep learning applications.
“Deep learning resembles electricity after Volta's battery invention but before Maxwell's theories,” explains Poggio, founding scientific advisor of The Core, MIT Quest for Intelligence, and CSAIL investigator. “Practical applications existed after Volta, but Maxwell's electromagnetic theory — that deeper understanding — paved the way for radio, television, radar, transistors, computers, and the internet.”
The theoretical framework developed by Poggio, Andrzej Banburski, and Qianli Liao explains why deep learning might overcome data challenges like “the curse of dimensionality.” Their approach begins with observing that many natural structures follow hierarchical patterns. Modeling a tree's growth doesn't require specifying every twig's position. Instead, hierarchical deep learning models can use local rules to drive branching patterns. The primate visual system employs similar processing when handling complex information. When viewing natural images — trees, cats, or faces — the brain progressively integrates local image patches, then small collections of patches, and finally collections of these collections.
“The physical world operates compositionally — meaning it's built from numerous local physical interactions,” explains Qianli Liao, study author and graduate student in Electrical Engineering and Computer Science and CBMM member. “This principle extends beyond images to language, human thought, and even neural connections. Our review theoretically explains why deep networks excel at representing this complexity.”
The hypothesis suggests that hierarchical neural networks should better approximate compositional functions than single-layer neuron networks, even with identical total neurons. The technical aspect of their research defines what “better at approximating” means and validates this hypothesis.
The Deep Network Generalization Puzzle
Another question surrounds what some call the unreasonable effectiveness of deep networks. Deep network models typically contain far more parameters than training data points, despite today's data abundance. This scenario should lead to “overfitting,” where models fit existing data perfectly but fail with new data — known as poor generalization in traditional models. The conventional solution involves constraining aspects of the fitting process. However, deep networks don't seem to require such constraints. Poggio and his team demonstrate that training deep networks implicitly “regularizes” the solution, providing built-in constraints.
This research has numerous implications for the future. While deep learning is already being widely implemented, this has occurred without comprehensive theoretical foundation. A theory explaining why and how deep networks function, along with their limitations, will likely enable development of significantly more powerful learning approaches.
“Long-term, developing superior intelligent machines will be essential for any technology-driven economy,” Poggio concludes. “Even in its current — still highly imperfect — state, deep learning impacts or will soon influence nearly every aspect of our society and daily lives.”
