While humans can easily distinguish between a turtle and a rifle, Google's artificial intelligence struggled with this differentiation just two years ago. For years, computer science researchers have dedicated significant resources to understanding how machine-learning models respond to "adversarial" attacks—deliberately crafted inputs designed to deceive or trick AI algorithms.
Although most research in this field has concentrated on speech and image recognition, a groundbreaking team from MIT's Computer Science and Artificial Intelligence Laboratory (CSAIL) recently pushed the boundaries with text-based systems. They developed "TextFooler," an innovative framework capable of successfully attacking natural language processing (NLP) systems—the same technology powering voice assistants like Siri and Alexa—and misleading them into generating incorrect predictions.
The potential applications for TextFooler span numerous internet safety domains, including email spam filtering, hate speech detection, and identification of "sensitive" political content—all of which rely on text classification models.
"If these tools are susceptible to intentional adversarial attacks, the consequences could be catastrophic," explains Di Jin, MIT PhD student and lead author of the groundbreaking TextFooler research paper. "These systems require robust defense mechanisms to protect themselves, and developing such safeguards necessitates a thorough understanding of adversarial methods first."
TextFooler operates through a two-stage process: modifying a given text, then utilizing that altered text to evaluate two distinct language tasks to determine if the system can effectively deceive machine-learning models.
The framework first identifies the most influential words affecting the target model's prediction, then selects contextually appropriate synonyms. Throughout this process, it maintains grammatical correctness and preserves the original meaning to appear sufficiently "human," continuing until the prediction changes.
Subsequently, the framework is applied to two different tasks—text classification and entailment (the relationship between text fragments within a sentence)—with the objective of altering the classification or invalidating the entailment judgment of the original models.
In one demonstration, TextFooler transformed the following input:
"The characters, cast in impossibly contrived situations, are totally estranged from reality."
To:
"The characters, cast in impossibly engineered circumstances, are fully estranged from reality."
In this example, when tested on an NLP model, the system correctly interpreted the original input but incorrectly processed the modified version.
Overall, TextFooler successfully attacked three target models, including the widely-used open-source NLP model "BERT." It reduced the target models' accuracy from over 90 percent to under 20 percent by altering merely 10 percent of words in a given text. The team assessed success based on three criteria: changing the model's prediction for classification or entailment; maintaining semantic similarity to the original text as judged by human readers; and preserving natural language flow.
The researchers emphasize that while attacking existing models isn't their ultimate goal, they hope this work will enable more abstract models to generalize better to new, unseen data.
"This framework can be utilized or extended to attack any classification-based NLP models to evaluate their robustness," notes Jin. "Conversely, the generated adversarial examples can enhance the robustness and generalization of deep-learning models through adversarial training—a crucial direction for this research."
Jin authored the paper alongside MIT Professor Peter Szolovits, Zhijing Jin from the University of Hong Kong, and Joey Tianyi Zhou of A*STAR, Singapore. They presented their findings at the AAAI Conference on Artificial Intelligence in New York.
