In an ideal digital landscape, artificial intelligence systems would process information with absolute certainty. However, the reality our AI systems navigate is far from perfect, filled with uncertainties and potential manipulations that challenge their decision-making capabilities.
Consider autonomous vehicles equipped with collision avoidance systems. These self-driving cars rely on visual input from onboard cameras to make split-second decisions—steering right, left, or maintaining course—to prevent accidents with pedestrians detected in their path. But what happens when camera sensors experience minor glitches or adversarial interference that subtly shifts an image by just a few pixels? Without proper defensive mechanisms, these vehicles might make unnecessary and potentially hazardous maneuvers.
Enter CARRL (Certified Adversarial Robustness for Deep Reinforcement Learning), an innovative deep-learning algorithm developed by MIT researchers. This groundbreaking approach is specifically designed to help machines operate effectively in our imperfect world by cultivating a healthy "skepticism" toward the measurements and inputs they receive.
The research team ingeniously combined a reinforcement-learning algorithm with a deep neural network—techniques previously used separately to train computers in mastering complex games like Go and chess. This fusion has resulted in a powerful framework that significantly enhances AI systems' ability to withstand adversarial attacks.
When tested across multiple scenarios, including simulated collision-avoidance situations and the classic video game Pong, CARRL consistently outperformed conventional machine-learning techniques. Even when confronted with uncertain, adversarial inputs, the algorithm demonstrated superior performance—successfully avoiding collisions and winning more Pong games than traditional approaches.
"You often think of an adversary being someone who's hacking your computer, but it could also just be that your sensors are not great, or your measurements aren't perfect, which is often the case," explains Michael Everett, a postdoc in MIT's Department of Aeronautics and Astronautics (AeroAstro). "Our approach helps to account for that imperfection and make a safe decision. In any safety-critical domain, this is an important approach to be thinking about."
Everett, the lead author of the study published in IEEE's Transactions on Neural Networks and Learning Systems, emphasizes that this research originated from MIT PhD student Björn Lütjens' master's thesis, under the guidance of MIT AeroAstro Professor Jonathan How.
Navigating Multiple Realities
Traditional approaches to creating robust AI systems have focused on supervised learning defenses. Typically, a neural network is trained to associate specific labels or actions with given inputs. For example, after processing thousands of images labeled as cats, houses, and hot dogs, the network should correctly identify a new image as a cat.
In robust AI systems, similar supervised-learning techniques might be tested with numerous slightly altered versions of an image. If the network consistently identifies the image as a cat despite these modifications, it demonstrates robustness against adversarial influences.
However, this method presents significant challenges. Running through every possible image alteration is computationally exhaustive and impractical for time-sensitive tasks like collision avoidance. Furthermore, existing approaches don't provide guidance on what label to use or action to take when the network's robustness fails and it misidentifies altered images.
"In order to use neural networks in safety-critical scenarios, we had to find out how to take real-time decisions based on worst-case assumptions on these possible realities," Lütjens explains.
Optimizing for Maximum Reward
The MIT team instead focused on reinforcement learning, a machine learning approach that doesn't require associating labeled inputs with outputs. Instead, it reinforces certain actions in response to specific inputs based on resulting rewards. This technique has been successfully used to train computers to master games like chess and Go.
Reinforcement learning has primarily been applied to situations where inputs are assumed to be accurate. Everett and his colleagues claim to be the first to introduce "certifiable robustness" to uncertain, adversarial inputs in reinforcement learning contexts.
Their approach, CARRL, utilizes an existing deep-reinforcement-learning algorithm to train a deep Q-network (DQN)—a multi-layered neural network that associates inputs with Q values representing reward levels.
The system processes an input, such as an image with a single dot, and considers potential adversarial influences by examining a region around the dot where it might actually be located. Every possible position of the dot within this region is processed through the DQN to identify the action that would yield the most optimal worst-case reward, based on a technique developed by recent MIT graduate Tsui-Wei "Lily" Weng PhD '20.
Thriving in Adversarial Environments
In tests involving the video game Pong, where two players control paddles on opposite sides of a screen to hit a ball back and forth, the researchers introduced an "adversary" that slightly displaced the ball's position. They discovered that CARRL won significantly more games than standard techniques as the adversary's influence increased.
"If we know that a measurement shouldn't be trusted exactly, and the ball could be anywhere within a certain region, then our approach tells the computer that it should position the paddle in the middle of that region, ensuring we hit the ball even in the worst-case deviation," Everett explains.
The method proved equally robust in collision avoidance tests, where the team simulated blue and orange agents attempting to switch positions without colliding. As the team disrupted the orange agent's observation of the blue agent's position, CARRL guided the orange agent around the other agent, taking a wider path as the adversary's influence grew and the blue agent's position became more uncertain.
The researchers did identify a point where CARRL became overly conservative, causing the orange agent to assume the other agent could be anywhere in its vicinity, leading it to completely avoid its destination. Everett notes that this extreme conservatism is actually beneficial, as researchers can use it as a benchmark to tune the algorithm's robustness—adjusting it to consider smaller deviations or regions of uncertainty that still allow agents to achieve high rewards and reach their destinations.
Beyond addressing imperfect sensors, Everett suggests that CARRL represents an initial step toward helping robots safely handle unpredictable interactions in the real world.
"People can be adversarial, like getting in front of a robot to block its sensors, or interacting with them, not necessarily with the best intentions," Everett observes. "How can a robot think of all the things people might try to do, and try to avoid them? What sort of adversarial models do we want to defend against? That's something we're actively working to address."
This research was supported, in part, by Ford Motor Company as part of the Ford-MIT Alliance.
