Have you ever imagined what a loaf of bread would look like transformed into a cat? This whimsical concept became reality through edges2cats, one of many innovative creations inspired by Phillip Isola's groundbreaking image-to-image translation software developed during the early days of generative adversarial networks (GANs). In a landmark 2016 paper, Isola and his research team demonstrated how a revolutionary type of GAN could convert a simple hand-drawn sketch of a shoe into a photorealistic fashion image, or transform an aerial photograph into a detailed grayscale map. Subsequently, the researchers showcased how landscape photographs could be reimagined using the distinctive impressionist brushstrokes of legendary artists like Monet and Van Gogh. Now serving as an assistant professor in MIT's prestigious Department of Electrical Engineering and Computer Science, Isola continues to push the boundaries of what GANs can accomplish.
GANs operate through an ingenious mechanism that pairs two neural networks, both trained on extensive image datasets. The first network, known as the generator, produces images patterned after the training examples. The second network, called the discriminator, evaluates how convincingly the generator's output resembles the authentic training data. When the discriminator identifies an image as artificially generated, the generator iteratively refines its approach until the output becomes virtually indistinguishable from genuine examples. When Isola first encountered GANs, he was experimenting with nearest-neighbor algorithms in an attempt to infer the underlying structural patterns of objects and scenes.
What makes GANs particularly remarkable is their extraordinary ability to capture the essential structure of places, faces, or objects, thereby facilitating more effective structured prediction. Since their introduction five years ago, GANs have been employed to visualize the devastating impacts of climate change, create more realistic computer simulations, and safeguard sensitive data, among numerous other applications.
To connect the expanding community of GAN enthusiasts at MIT and beyond, Isola recently helped organize GANocracy, a comprehensive day of talks, tutorials, and poster presentations held at MIT on May 31. This event was co-sponsored by the MIT Quest for Intelligence and the MIT-IBM Watson AI Lab. Isola recently shared his insights about the future trajectory of GANs and their evolving role in artificial intelligence.
When asked about his influential image-to-image translation paper, which has garnered over 2,000 citations, Isola explained: "It was among the pioneering papers to demonstrate the practical utility of GANs for predicting visual data. We illustrated that this framework is remarkably versatile and can be conceptualized as translating between different visual representations of the world—a process we termed image-to-image translation. While GANs were initially proposed as a model for generating realistic images from scratch, their most valuable application appears to be structured prediction, which represents the primary use case for GANs in today's landscape."
Regarding the creative applications of GANs that have gained popularity on social media, Isola highlighted several favorites: "#Edges2cats is probably my favorite, and it significantly contributed to popularizing the framework in its early stages. Architect Nono Martínez Alonso has utilized pix2pix to explore innovative tools for sketch-based design. I admire everything created by Mario Klingemann; Alternative Face is particularly thought-provoking. It places one person's words into another's mouth, suggesting a potential future of 'alternative facts.' Scott Eaton is pushing the boundaries of GAN technology by transforming sketches into three-dimensional sculptures."
When discussing other GAN-generated art that captures his attention, Isola expressed enthusiasm: "I genuinely appreciate all of it. One extraordinary example is GANbreeder. It represents a human-curated evolution of GAN-generated images, where the community collectively decides which images to breed or eliminate. Over multiple generations, this process yields beautiful and unexpected artistic creations."
Beyond the artistic realm, GANs are finding applications in numerous fields. "In medical imaging, they're being employed to generate CT scans from MRIs," Isola noted. "While there's significant potential here, it can be easy to misinterpret the results: GANs are making predictions, not revealing absolute truth. We haven't yet developed robust methods to measure the uncertainty of their predictions. I'm also excited about the application of GANs for simulations. Robots are frequently trained in simulators to reduce costs, which creates complications when deploying them in real-world environments. GANs can help bridge the gap between simulation and reality."
Addressing whether GANs might redefine the concept of artistry, Isola reflected: "I don't know, but it's an incredibly fascinating question. Several of our GANocracy speakers are artists, and I hope they will explore this topic. GANs and other generative models differ from other forms of algorithmic art because they are trained to imitate, so the individuals being imitated likely deserve some recognition. The art collective Obvious recently sold a GAN-generated image at Christie's for $432,500. While Obvious selected, signed, and framed the image, the underlying code was derived from work by then-17-year-old Robbie Barrat, with Ian Goodfellow contributing to the development of the fundamental algorithm."
Looking toward the future of the field, Isola offered his perspective: "As remarkable as GANs are, they represent merely one type of generative model. GANs might eventually decline in popularity, but generative models are here to stay. As models of high-dimensional structured data, generative models approach what we mean when we use terms like 'create,' 'visualize,' and 'imagine.' I believe they will increasingly be employed to approximate capabilities that still seem uniquely human. However, GANs do possess some distinctive properties. For one, they address the generative modeling challenge through a two-player competition, creating a generator-discriminator arms race that leads to emergent complexity. These competitive dynamics appear throughout machine learning, including in the AI that achieved superhuman performance in the game of Go."
When questioned about potential misuse of GAN technology, Isola expressed concern: "I'm definitely worried about the use of GANs to generate and disseminate misleading content, or so-called fake news. GANs make it significantly easier to create manipulated photos and videos, where you no longer need to be a video editing expert to make it appear as though a politician is saying something they never actually said."
Addressing the concept of 'GANtidotes' advocated by Isola and other GANocracy organizers, he explained: "We would like to inoculate society against the misuse of GANs. Everyone could simply stop trusting what we see online, but then we'd risk losing touch with reality. I'd prefer to preserve a future in which 'seeing is believing.' Fortunately, many researchers are developing technical antidotes ranging from detectors that identify telltale artifacts in GAN-manipulated images to cryptographic signatures that verify that a photo hasn't been altered since its capture. There are numerous promising approaches being explored, so I'm optimistic that this challenge can be addressed."
