Future Timeline

What exactly is lost in translation when TV shows and films are subbed or dubbed into a new language? It’s a hard question to answer, but for the team at AI startup Flawless, it may be one we don’t have to think about in the future. The company claims it has the solution to this particular language barrier; a technical innovation that could help TV shows and films effortlessly reach new markets around the world: deepfake dubs.

We often think of deepfakes as manipulating the entire image of a person or scene, but Flawless’ technology focuses on just a single element: the mouth. Customers feed the company’s software with video from a film or TV show along with dubbed dialogue recorded by humans. Flawless’ machine learning models then create new lip movements that match the translated speech and paste them automatically onto the actor’s head.

"When someone’s watching this dubbed footage, they’re not jolted out of the performance by a jarring word or a mistimed mouth movement,” Flawless’ co-founder Nick Lynes tells The Verge. “It’s all about retaining the performance and retaining the original style.”

Creativity knows no barrier and is found across all spectrums. However, most often, language is found to be a significant roadblock in conveying that creativity to the masses. TV shows and films are one arena where widespread translation is used to reach wider audiences. But the question that arises is, does that change the viewing experience of the audience? The answer might be different for different individuals, but it could possibly be insignificant given the fact that the AI Startup Flawless claims to have found a technology to override this language barrier. With the help of their Deepfake Dubs, the performance quality and the emotion would be retained, and the film or show, even when translated, would be as authentic as the original.

Update, 5/30/21: It's Memorial Day weekend in the US, and staff are trying to stay away from the keyboard accordingly. As such, we're resurfacing a few classic pieces from our archives. Since catching a holiday weekend movie in 2021 is a much different proposition than in years past, we thought a second theatrical front page run for Sunspring, a short film starring Thomas Middleditch and written in conjunction with an algorithm, should be in order. The film originally debuted on Ars on June 9, 2016, and our interview with the humans behind the project appears unchanged below.

Ars is excited to be hosting this online debut of Sunspring, a short science fiction film that's not entirely what it seems. It's about three people living in a weird future, possibly on a space station, probably in a love triangle. You know it's the future because H (played with neurotic gravity by Silicon Valley's Thomas Middleditch) is wearing a shiny gold jacket, H2 (Elisabeth Gray) is playing with computers, and C (Humphrey Ker) announces that he has to "go to the skull" before sticking his face into a bunch of green lights. It sounds like your typical sci-fi B-movie, complete with an incoherent plot. Except Sunspring isn't the product of Hollywood hacks—it was written entirely by an AI. To be specific, it was authored by a recurrent neural network called long short-term memory, or LSTM for short. At least, that's what we'd call it. The AI named itself Benjamin.