Deep learning technology is transforming our digital landscape, powering everything from social media curation to search engine results. Now, this revolutionary artificial intelligence branch is poised to revolutionize how we interact with everyday objects, from monitoring vital signs to adjusting home temperatures. MIT researchers have pioneered MCUNet, an innovative system designed to bring deep learning neural networks to the realm of microcontrollers — the tiny computer chips powering wearable medical devices, household appliances, and the billions of connected devices forming the Internet of Things (IoT) ecosystem.
MCUNet represents a breakthrough in creating compact neural networks that deliver exceptional speed and accuracy for deep learning on IoT devices, despite their severe memory and processing constraints. This cutting-edge technology promises to dramatically expand the IoT universe while simultaneously reducing energy consumption and enhancing data security measures.
The groundbreaking research will be unveiled at the upcoming Conference on Neural Information Processing Systems. Led by Ji Lin, a PhD student in Song Han's lab at MIT's Department of Electrical Engineering and Computer Science, the team includes distinguished researchers Han and Yujun Lin of MIT, Wei-Ming Chen of MIT and National University Taiwan, and John Cohn and Chuang Gan of the MIT-IBM Watson AI Lab.
The Evolution of Connected Devices
The concept of Internet of Things traces back to the early 1980s when Carnegie Mellon University graduate students, including Mike Kazar '78, famously connected a Coca-Cola machine to the internet. Their motivation was simple yet revolutionary: eliminating unnecessary trips by checking the machine's inventory remotely. "This was pretty much treated as the punchline of a joke," recalls Kazar, now a Microsoft engineer. "No one expected billions of devices on the internet."
Since that humble Coke machine, everyday objects have increasingly become part of the expanding IoT network. Today, this includes everything from wearable heart monitors to intelligent refrigerators that alert you when milk supplies run low. These IoT devices typically operate on microcontrollers — simple computer chips lacking operating systems, with minimal processing power and less than one thousandth of the memory found in typical smartphones. Consequently, running complex pattern-recognition tasks like deep learning directly on these devices has been nearly impossible. Traditionally, data collected by IoT devices must be sent to the cloud for analysis, creating potential security vulnerabilities.
"How do we deploy neural nets directly on these tiny devices? It's a new research area that's getting very hot," explains Han. "Companies like Google and ARM are all working in this direction." And Han's team is at the forefront of this technological revolution.
Through MCUNet, Han's research group has codesigned two essential components for what they term "tiny deep learning" — implementing neural networks on microcontrollers. The first component is TinyEngine, an optimized inference engine that manages resources much like an operating system. TinyEngine is specifically designed to run a particular neural network structure, which is selected by MCUNet's second component: TinyNAS, a sophisticated neural architecture search algorithm.
System-Algorithm Codesign Innovation
Creating deep learning networks for microcontrollers presents significant challenges. Existing neural architecture search techniques typically begin with a large pool of potential network structures based on predefined templates, gradually identifying the optimal balance between accuracy and computational cost. While this approach works for more powerful devices, it proves inefficient for microcontrollers. "It can work pretty well for GPUs or smartphones," notes Lin. "But it's been difficult to directly apply these techniques to tiny microcontrollers, because they are too small."
To address this limitation, Lin developed TinyNAS, a neural architecture search method that creates custom-sized networks tailored to specific microcontroller constraints. "We have a lot of microcontrollers that come with different power capacities and different memory sizes," Lin explains. "So we developed the algorithm [TinyNAS] to optimize the search space for different microcontrollers." This customized approach enables TinyNAS to generate compact neural networks with optimal performance for any given microcontroller — eliminating unnecessary parameters. "Then we deliver the final, efficient model to the microcontroller," Lin adds.
To execute these compact neural networks, microcontrollers also require a streamlined inference engine. Conventional inference engines typically contain redundant code — instructions for tasks they rarely perform. While this extra code poses no problem for laptops or smartphones, it can easily overwhelm a microcontroller's limited resources. "It doesn't have off-chip memory, and it doesn't have a disk," Han points out. "Everything put together is just one megabyte of flash, so we have to really carefully manage such a small resource." This challenge led to the development of TinyEngine.
The researchers engineered their inference engine in tandem with TinyNAS. TinyEngine generates only the essential code required to run TinyNAS' customized neural networks, eliminating any superfluous code and significantly reducing compile-time. "We keep only what we need," Han states. "And since we designed the neural network, we know exactly what we need. That's the advantage of system-algorithm codesign." In performance tests, TinyEngine produced compiled binary code between 1.9 and five times smaller than comparable microcontroller inference engines from Google and ARM. Additionally, TinyEngine incorporates innovative features like in-place depth-wise convolution, which reduces peak memory usage by nearly half. After successfully codesigning TinyNAS and TinyEngine, Han's team put MCUNet through rigorous testing.
MCUNet's initial challenge was image classification. The researchers trained the system using labeled images from the ImageNet database, then tested its ability to classify novel images. On a commercial microcontroller, MCUNet achieved an impressive 70.7 percent accuracy in classifying novel images — significantly outperforming the previous state-of-the-art neural network and inference engine combination, which achieved only 54 percent accuracy. "Even a 1 percent improvement is considered significant," Lin emphasizes. "So this is a giant leap for microcontroller settings."
The team observed similar performance improvements in ImageNet tests across three additional microcontrollers. Furthermore, MCUNet outperformed competitors in both speed and accuracy for audio and visual "wake-word" tasks, where users initiate interactions through voice commands (such as "Hey, Siri") or simply by entering a room. These experiments demonstrate MCUNet's remarkable adaptability across numerous applications.
Transformative Potential
The promising test results have led Han to believe that MCUNet could become the new industry standard for microcontrollers. "It has huge potential," he asserts.
This advancement "extends the frontier of deep neural network design even farther into the computational domain of small energy-efficient microcontrollers," notes Kurt Keutzer, a computer scientist at the University of California at Berkeley, who was not involved in the research. He adds that MCUNet could "bring intelligent computer-vision capabilities to even the simplest kitchen appliances, or enable more intelligent motion sensors."
MCUNet also offers significant security benefits for IoT devices. "A key advantage is preserving privacy," Han explains. "You don't need to transmit the data to the cloud."
By processing data locally, MCUNet reduces the risk of personal information being compromised — particularly sensitive health data. Han envisions smart watches equipped with MCUNet that not only monitor users' heartbeat, blood pressure, and oxygen levels but also analyze and help interpret this information. The technology could also bring deep learning capabilities to IoT devices in vehicles and rural areas with limited internet connectivity.
Additionally, MCUNet's minimal computing requirements translate to a reduced carbon footprint. "Our big dream is for green AI," Han shares, noting that training a large neural network can consume energy equivalent to the lifetime emissions of five cars. MCUNet operating on a microcontroller would require only a fraction of that energy. "Our end goal is to enable efficient, tiny AI with less computational resources, less human resources, and less data," Han concludes.
