Watch the Skies Is The First Movie to Be Visually Dubbed Using AI – A Breakthrough or Threat for Actor?

The Swedish sci-fi adventure Watch the Skies is breaking new ground as the first feature film to use a cutting-edge technique called “visual dubbing” for its international release. This innovative approach aims to eliminate the awkwardness often associated with traditional dubbing, where dialogue rarely syncs up with the actors’ lip movements. 

Instead, it matches their lip movements to the dubbed English dialogue without requiring any reshoots. The makers have even tried to set new precedents when it comes to AI and copyright, ensuring dubbing is only carried out with the consent of actors.

The innovation behind this groundbreaking technique is TrueSync, a technology developed by Los Angeles-based movie-making AI firm Flawless. TrueSync works by analyzing an actor’s entire performance, building a volumetric 3D model of their face, and then adjusting their lip movements in post-production to match the new dialogue track. It is similar to technology used in gaming to create realistic dialog scenes. The result is a seamless experience for audiences, allowing them to fully immerse themselves in the story without distraction.

TrueSync has been around for a few years, but its most notable use until now was to clean up dialogue in the 2022 British survival horror movie Fall, which happened to be directed by Flawless CEO Scott Mann. In that instance, the technology was used to swap out 35 swearing examples with PG-13-friendly phrases for a censored edit of the movie. According to Mann, this allowed for significant changes to the film’s dialogue without the need for expensive and time-consuming reshoots.

Watch the Skies: A Game-Changer for Global Releases

Now, Flawless is taking things a step further by visually dubbing an entire movie. Watch the Skies, originally released as UFO Sweden in 2022, follows a teenager searching for her missing father, who she believes may have been abducted by aliens. She teams up with a UFO watchers’ club to uncover the truth, leading them on a perilous journey into a realm of unexplained phenomena and conspiracies.

Critically, “UFO Sweden” has garnered positive feedback. On Rotten Tomatoes, it holds a 100% approval rating based on seven reviews, with an average score of 7.3/10. Critics have praised its nostalgic nod to ’80s and ’90s family-friendly sci-fi adventures, likening its charm to that of “Stranger Things.”

Andrew Parker from The Gate described it as “a delightfully charming throwback,” so it is little surprise that there was a demand to turn it into an English language film and see how it fared with audiences in the rest of the world. This will be tested when it gets a theatrical release in May.

For the English-language release, the original actors dubbed their own performances at a recording studio. This approach ensures that their authentic vocal expressions are preserved, complete with Swedish accents, adding another layer of authenticity to the overall experience. Instead of the clunky and disjointed dialogue often associated with foreign film dubbing, TrueSync’s technology provides smooth lip-syncing that aligns perfectly with the English audio.

You can see the results of the technology yourself below.

“Our aim has always been to make films feel as if they were performed in the audience’s own language,” said Scott Mann. “By using TrueSync, we can help filmmakers reach wider audiences without sacrificing the original performances that make their work special.”

Bridging the Language Barrier with AI

While subtitles have long been a go-to solution for foreign films, they can detract from the viewing experience by forcing audiences to split their attention between text and visuals. Visual dubbing, by contrast, allows viewers to stay fully immersed in the storytelling without being jarred by mismatched audio and visual cues.

By enhancing the lip-syncing process, Flawless hopes to open new doors for international film distribution. It could mean more foreign films finding success in English-speaking markets and vice versa. It also offers filmmakers greater creative freedom, allowing them to craft dialogue that best serves the story rather than being limited by the constraints of traditional dubbing.

TrueSync’s use in Watch the Skies is the most ambitious application of the technology to date. If it proves successful, it could pave the way for a broader adoption of visual dubbing in both the independent and mainstream film industries.

A Brighter View of the Future of AI in Cinema

Visual dubbing might sound like a futuristic concept, but it’s here now, and the potential applications are staggering. It’s not just about making foreign films more accessible; it’s about preserving the integrity of performances across languages and breaking down barriers that have long limited the reach of non-English cinema.

Scott Mann and his team at Flawless have already demonstrated the technology’s versatility with Fall. Now, Watch the Skies is the first full-length feature to take advantage of the technique, and all eyes are on its international reception. As AI continues to infiltrate various aspects of the film industry, visual dubbing could be one of the most practical and impactful tools to emerge in years. 

A lot of people are very worried about what AI could mean for creativity. The film industry’s ongoing relationship with AI has often been met with skepticism and concern, but innovations like TrueSync offer a glimpse at how these tools can be used to enhance creativity rather than stifle it. As Scott Mann puts it, “We’re not looking to replace actors or their performances. We’re trying to amplify them, to give their work the broadest reach possible.”

With the success of Watch the Skies hanging in the balance, visual dubbing could soon become a standard practice rather than a novel experiment. And as audiences worldwide continue to demand more diverse storytelling, technologies like TrueSync could be the key to making that possible.