Leveraging cutting-edge artificial intelligence and supercomputing technology, MIT researchers have pioneered a groundbreaking model that maps global internet traffic patterns in real-time. This innovative AI-powered system serves as a comprehensive measurement tool for internet research and numerous digital applications, revolutionizing how we understand worldwide web activity.
Understanding these massive-scale traffic patterns through machine learning algorithms provides invaluable insights for shaping internet policy, preventing system outages, strengthening cyber defenses against sophisticated attacks, and designing more efficient computing infrastructure. The research team presented their findings at the prestigious IEEE High Performance Extreme Computing Conference, highlighting the intersection of AI and network security.
For this remarkable achievement, the researchers compiled the largest publicly available internet traffic dataset ever assembled, containing an astonishing 50 billion data packets exchanged across diverse global locations over multiple years. This massive dataset forms the foundation for training their sophisticated neural network traffic monitoring system.
The team processed this enormous dataset through a novel "neural network" pipeline operating across 10,000 processors of the MIT SuperCloud, an advanced system that integrates computing resources from the MIT Lincoln Laboratory and throughout the Institute. This pipeline automatically trained an AI model that captures the complex relationships between all connections in the dataset—from routine pings to tech giants like Google and Facebook, to rare links that only briefly connect yet significantly influence web traffic patterns.
This artificial intelligence internet infrastructure model can analyze any massive network dataset and generate statistical measurements about how all connections within the network influence each other. This capability reveals critical insights about peer-to-peer filesharing, identifies potentially malicious IP addresses and spamming behavior, maps the distribution of attacks in critical sectors, and highlights traffic bottlenecks to optimize computing resource allocation and maintain smooth data flow.
The concept behind this work parallels measuring the cosmic microwave background of space—the nearly uniform radio waves traveling throughout our universe that have provided crucial information for studying cosmic phenomena. "We've developed an exceptionally accurate model for measuring the background of the internet's virtual universe," explains Jeremy Kepner, a researcher at the MIT Lincoln Laboratory Supercomputing Center with a background in astronomy. "To effectively detect any variances or anomalies, you first need a comprehensive model of the background activity."
Collaborating with Kepner on the research paper were: Kenjiro Cho of the Internet Initiative Japan; KC Claffy of the Center for Applied Internet Data Analysis at the University of California at San Diego; Vijay Gadepally and Peter Michaleas of Lincoln Laboratory's Supercomputing Center; and Lauren Milechin, a researcher in MIT's Department of Earth, Atmospheric and Planetary Sciences.
Advanced Data Processing
In the realm of internet research, experts examine anomalies in web traffic that might indicate cyber threats. To effectively identify these threats, understanding normal traffic patterns is essential. However, capturing this baseline has proven challenging. Traditional "traffic-analysis" models can only process limited samples of data packets exchanged between sources and destinations, constrained by geographical limitations. This restriction significantly reduces model accuracy and reliability.
The researchers weren't initially focused on solving this traffic-analysis challenge. Instead, they had been developing new techniques specifically designed for the MIT SuperCloud to process massive network matrices. Internet traffic emerged as the ideal test case for their innovative approach.
Networks are typically studied as graphs, with actors represented by nodes and connections shown as links between these nodes. In internet traffic, nodes vary dramatically in size and location. Large supernodes function as popular hubs, such as Google or Facebook. Leaf nodes extend from these supernodes, maintaining multiple connections with each other and the central supernode. Beyond this "core" structure of supernodes and leaf nodes exist isolated nodes and links, which connect to each other only infrequently.
Capturing the complete structure of these graphs remains impossible for traditional models. "You simply can't process that data without access to a supercomputer," Kepner emphasizes.
In collaboration with the Widely Integrated Distributed Environment (WIDE) project, established by several Japanese universities, and the Center for Applied Internet Data Analysis (CAIDA) in California, the MIT researchers assembled the world's largest packet-capture dataset for internet traffic. This anonymized dataset contains nearly 50 billion unique source and destination data points between consumers and various applications and services during random days across locations in Japan and the U.S., with data dating back to 2015.
Before training their AI model on this data, the team needed to perform extensive preprocessing. They utilized previously created software called Dynamic Distributed Dimensional Data Mode (D4M), which employs advanced averaging techniques to efficiently compute and sort "hypersparse data" containing significantly more empty space than actual data points. The researchers divided the data into units of approximately 100,000 packets across 10,000 MIT SuperCloud processors. This approach generated more compact matrices of billions of rows and columns representing interactions between sources and destinations.
Identifying Network Anomalies
Despite the preprocessing, the vast majority of cells in this hypersparse dataset remained empty. To process these matrices effectively, the team ran a neural network across the same 10,000 cores. Behind the scenes, a trial-and-error technique began fitting models to the complete dataset, creating a probability distribution of potentially accurate models.
Next, the researchers employed a modified error-correction technique to further refine the parameters of each model, capturing as much data as possible. Traditional error-correcting techniques in machine learning typically reduce the significance of outlying data to make the model fit a normal probability distribution, improving overall accuracy. However, the MIT team used innovative mathematical approaches to ensure their model still recognized all outlying data—such as isolated links—as significant to the overall measurements.
The resulting neural network generates a surprisingly simple model with only two parameters that comprehensively describes the internet traffic dataset, "from extremely popular nodes to isolated ones, and the complete spectrum of everything in between," Kepner explains.
Utilizing supercomputing resources to efficiently process a "firehose stream of traffic" and identify meaningful patterns and web activity represents "groundbreaking" work, according to David Bader, a distinguished professor of computer science and director of the Institute for Data Science at the New Jersey Institute of Technology. "A grand challenge in cybersecurity is understanding global-scale trends in internet traffic for purposes such as detecting nefarious sources, identifying significant flow aggregation, and protecting against computer viruses. This research group has successfully tackled this problem and presented deep analysis of global network traffic," he notes.
The researchers are now reaching out to the scientific community to identify the next applications for their model. For instance, experts could examine the significance of the isolated links discovered during experiments that, while rare, appear to impact web traffic in core nodes.
Beyond internet applications, this neural network pipeline can analyze any hypersparse network, including biological and social networks. "We've now provided the scientific community with an exceptional tool for those seeking to build more robust networks or detect network anomalies," Kepner concludes. "Those anomalies might represent normal user behaviors, or they could indicate activities that organizations want to prevent."
