Recent statistics from the U.S. Centers for Disease Control and Prevention reveal a concerning reality: approximately one in seven children across the nation experienced abuse or neglect within the past year. Child protective services agencies nationwide face an overwhelming volume of reports, receiving roughly 4.4 million allegations of mistreatment in 2019 alone. To manage this staggering caseload, numerous agencies are now turning to machine learning models to assist child welfare specialists in screening cases and determining which warrant further investigation.
However, the effectiveness of these AI tools hinges on a critical factor: the understanding and trust of the human professionals they're designed to support. Without clear comprehension of how these systems arrive at their conclusions, even the most sophisticated algorithms remain underutilized.
Addressing this challenge head-on, researchers from MIT and partner institutions launched an innovative research initiative focused on identifying and resolving machine learning explainability issues specifically within child welfare screening processes. Working in close collaboration with a child welfare department in Colorado, the team conducted in-depth studies examining how call screeners evaluate cases both with and without the assistance of machine learning predictions. Drawing directly from feedback provided by these frontline professionals, the researchers engineered a sophisticated visual analytics tool that employs intuitive bar graphs to illustrate how specific case factors contribute to the predicted risk of a child being removed from their home within a two-year timeframe.
The research yielded fascinating insights: screeners demonstrated greater interest in understanding how individual factors—such as a child's age—influence predictions, rather than delving into the computational mechanics underlying the model's operations. Additionally, the findings revealed that even seemingly straightforward models could generate confusion if their features aren't described using clear, accessible language.
According to senior author Kalyan Veeramachaneni, principal research scientist in the Laboratory for Information and Decision Systems (LIDS), these discoveries have far-reaching implications beyond child welfare. "Our findings could transform how AI transparency is implemented across numerous high-risk fields where professionals rely on machine learning models to guide decisions, yet lack specialized data science training," he explains.
"The field of explainable AI often focuses on dissecting the model's inner workings to clarify its decision-making process. However, a crucial insight from our project is that domain experts aren't necessarily interested in learning the technical details of machine learning operations. Instead, they seek to understand why the model might produce predictions that contradict their intuition or which specific factors it's prioritizing. They want information that helps them reconcile points of agreement or disagreement with the model, or confirms their professional judgment," Veeramachaneni elaborates.
The research team includes electrical engineering and computer science PhD student Alexandra Zytek, who served as lead author; postdoc Dongyu Liu; and Rhema Vaithianathan, professor of economics and director of the Center for Social Data Analytics at the Auckland University of Technology and professor of social data analytics at the University of Queensland. The team presented their groundbreaking findings at the prestigious IEEE Visualization Conference later this month.
Practical Research in Real-World Settings
The research journey commenced over two years ago when the team identified seven key factors that diminish machine learning model usability, including distrust in prediction sources and discrepancies between user intuition and model outputs.
Armed with these insights, Zytek and Liu traveled to Colorado in winter 2019 to gain firsthand understanding from call screeners within a child welfare department. This particular department was implementing a machine learning system developed by Vaithianathan that generates a risk score for each report, forecasting the probability of a child being removed from their home. This risk assessment relies on more than 100 demographic and historical factors, including parental ages and previous court involvements.
"As you might imagine, receiving merely a number between one and 20 with instructions to incorporate it into your workflow can present significant challenges," Zytek notes.
Through observation of screener teams processing cases in approximately 10-minute sessions, the researchers noted that professionals dedicated most of their time to discussing risk factors associated with each case. This observation inspired the development of a case-specific details interface, which illustrates how each factor influences the overall risk score through color-coded, horizontal bar graphs indicating the magnitude of each contribution—whether positive or negative.
Building on these observations and detailed interviews, the researchers constructed four additional interfaces providing model explanations, including one that compares current cases to historical cases with similar risk profiles. Subsequently, they conducted comprehensive user studies to evaluate effectiveness.
The results were striking: more than 90% of screeners found the case-specific details interface highly valuable, and it generally enhanced their confidence in the model's predictions. Conversely, participants expressed dissatisfaction with the case comparison interface. Despite the researchers' expectation that this feature would boost trust in the model, screeners expressed concerns that it might encourage decisions based on historical precedents rather than the unique circumstances of the current report.
"Perhaps the most intriguing discovery was that the features we presented—the information the model utilizes—needed to be inherently interpretable from the outset. The model incorporates over 100 different features to generate predictions, and many of these were somewhat perplexing to users," Zytek explains.
Maintaining continuous engagement with screeners throughout the iterative development process proved invaluable, enabling researchers to make informed decisions about which elements to incorporate into the machine learning explanation tool, ultimately named Sibyl.
As they refined the Sibyl interfaces, the researchers remained vigilant about how providing explanations might inadvertently introduce cognitive biases or potentially undermine screeners' trust in the model.
For example, since explanations rely on averages from a database of child abuse and neglect cases, having three past abuse referrals might actually decrease a child's risk score, as averages in this database could be substantially higher. A screener encountering this explanation might question the model's reliability despite its correct functioning, Zytek clarifies. Furthermore, because humans typically place greater emphasis on recent information, the sequence in which factors are presented could also influence decision-making outcomes.
Advancing Interpretability
Incorporating feedback from call screeners, the research team is currently refining the explanation model to ensure its features are more readily explainable to end-users.
Looking ahead, the team plans to enhance the interfaces they've created based on additional user input, followed by quantitative user studies to evaluate the tool's impact on real-world decision-making with actual cases. Once these assessments are complete, they can prepare for the official deployment of Sibyl, according to Zytek.
"The opportunity to collaborate so closely with these screeners proved exceptionally valuable. We gained genuine insight into the challenges they face daily. While we observed some initial reservations, what stood out more was their enthusiasm regarding how useful these explanations proved in certain scenarios. That feedback was incredibly gratifying," Zytek reflects.
This significant research initiative received support, in part, from the National Science Foundation.
