The emergence of COVID-19 presented unprecedented challenges for vaccine researchers worldwide. Unlike traditional vaccine development where scientists possess substantial background knowledge about their target, this rapidly evolving virus left researchers with limited information, forcing them to adopt innovative approaches and cutting-edge technologies to grasp even the fundamentals of the disease.
At Janssen Research & Development, the pharmaceutical powerhouse behind the Johnson & Johnson COVID-19 vaccine, scientists embraced a revolutionary approach. They harnessed real-world data and forged a strategic partnership with MIT researchers, implementing sophisticated artificial intelligence and machine learning algorithms to steer their vaccine development efforts in the right direction.
“Data science and machine learning can significantly enhance our scientific understanding of diseases,” explains Najat Khan, chief data science officer and global head of strategy and operations at Janssen Research & Development. “For COVID-19, these technologies proved invaluable because our knowledge was extremely limited. We had no preconceived hypotheses. Instead, we developed an unbiased understanding of the disease by leveraging real-world data through advanced AI/ML algorithms.”
As Janssen prepared for clinical trials of their leading vaccine candidate, Khan sought collaborators for predictive modeling initiatives. Through Regina Barzilay, the MIT School of Engineering Distinguished Professor for AI and Health, Khan connected with Dimitris Bertsimas, the Boeing Leaders for Global Operations Professor of Management at MIT Sloan. Bertsimas had developed a groundbreaking machine learning model capable of tracking COVID-19 spread and predicting patient outcomes, making him the ideal technical partner for this critical project.
THE DELPHI MODEL
When the World Health Organization officially declared COVID-19 a pandemic in March 2020, Bertsimas mobilized his team of over 25 doctoral and master's students. Their mission: to apply their collective expertise in machine learning and optimization to create powerful tools that could help combat the global health crisis.
The team documented their progress on the COVIDAnalytics platform, where their models generated precise real-time insights into the pandemic's trajectory. Among their initial achievements was the development of DELPHI, an advanced epidemiological model that predicted infection and mortality rates across different states based on their specific policy decisions.
Building upon the traditional SEIR model—which categorizes populations into susceptible, exposed, infectious, and recovered groups—DELPHI introduced a more sophisticated system with 11 possible states. This enhanced approach allowed for realistic pandemic effects analysis, including comparing hospitalization duration between recovered patients and those who succumbed to the virus.
“While certain model parameters were predetermined, such as hospital stay duration, we delved deeper into accounting for the nonlinear changes in infection rates,” notes Bertsimas. “We discovered these rates weren't constant but varied across different periods and locations. This additional modeling flexibility significantly improved our prediction accuracy.”
A groundbreaking feature of the DELPHI model was its ability to capture human behavior patterns in response to pandemic measures like lockdowns, mask mandates, and social distancing, along with their subsequent impact on transmission rates.
“By mid-2020, we had integrated these behavioral data points, dramatically enhancing the model's accuracy,” Bertsimas explains. “We also examined various governmental response scenarios, from strict restrictions to complete openness, comparing them against real-world developments. This approach enabled us to generate a spectrum of predictions. Notably, DELPHI delivers daily predictions for 120 countries and all 50 U.S. states.”
DEVELOPING A VACCINE FOR A CHANGING PANDEMIC
The ability to forecast potential COVID-19 hotspots proved crucial for the success of Janssen's clinical trials, which were “event-based”—meaning efficacy determination relied on counting specific events within the study population, such as COVID-19 infections.
“For such large-scale trials, it's essential to identify locations with anticipated high disease transmission rates,” Khan elaborates. “This strategy allows for rapid accumulation of events, accelerating the trial timeline and enabling faster vaccine delivery to market. Most importantly, it provides a robust dataset for statistically sound analysis.”
Bertsimas assembled a dedicated research team, including MIT Operations Research Center doctoral students Michael Li, who led implementation efforts, and Omar Skali Lami. The team also included Hamza Tazi Bouardi MBN '20, a former business analytics master's student, and Ali Haddad, a data research scientist at Dynamic Ideas LLC.
Beginning in May, the MIT team collaborated closely with Khan's group to forecast potential case surges. Their objective: to identify COVID-19 hotspots where Janssen could establish clinical trials and recruit participants with high exposure risk.
With trials scheduled to commence in September, the teams immediately began intensive collaboration, generating predictions four months ahead of the actual trial dates. “We established daily meetings with the Janssen team—sometimes multiple times daily, including weekends,” Bertsimas recalls.
To track global virus movement, Janssen data scientists continuously monitored worldwide data sources. They constructed a comprehensive global surveillance dashboard that aggregated country, state, and county-level data (where available) on case numbers, hospitalizations, mortality rates, and testing statistics.
The DELPHI model incorporated these datasets alongside additional information about local policies and behaviors, such as mask compliance rates, generating 300-400 daily predictions. “We received constant feedback from the Janssen team, which helped refine our model,” Bertsimas states. “Eventually, the model became central to the clinical trial process.”
Remarkably, most locations predicted by DELPHI to become COVID-19 hotspots ultimately experienced extremely high case numbers, including South Africa and Brazil, where new virus variants had emerged by the trial commencement. According to Khan, elevated incidence rates typically indicate variant presence.
“All our model predictions remain publicly accessible, allowing verification of their accuracy,” Bertsimas asserts. “The model has consistently performed exceptionally well. To this day, DELPHI stands as one of the most accurate models developed by the scientific community during this pandemic.”
“Thanks to this model, we submitted our vaccine candidate with an exceptionally data-rich package,” Khan notes. “We conducted clinical trials in South Africa and Brazil—regions where few competitors had gathered data. This proved critical as we developed a vaccine relevant to today's world, where multiple variants continue to challenge global health efforts.”
Khan emphasizes that the DELPHI model evolved with diversity considerations, incorporating biological risk factors, patient demographics, and other characteristics. “COVID-19 affects different populations differently, so recruiting diverse participants was essential,” she explains. “Our efforts resulted in one of the most diverse COVID-19 trials conducted to date. When you start with unbiased data and select the right locations, you can transform many of the paradigms that currently limit scientific progress.”
In April, the MIT and Janssen R&D Data Science teams received the 2021 Innovative Applications in Analytics Award from the Institute for Operations Research and the Management Sciences (INFORMS) for their groundbreaking work on COVID-19. Building on this achievement, both teams continue collaborating, applying their data-driven approach and technical expertise to address other infectious diseases. “This wasn't merely a nominal partnership,” Khan affirms. “Our teams truly integrated and continue working together on various data science initiatives across our development pipeline.” The team also acknowledges the crucial contributions of on-site investigators who combined their expertise with the model's insights for site selection.
“It was an incredibly fulfilling experience,” Bertsimas concludes. “I'm proud to have contributed to this effort and to have helped the world combat this devastating pandemic through the power of artificial intelligence in vaccine development.”
