Flawless AI’s ‘True Sync’ technology revolutionizes movie dubbing

When British director Scott Mann’s latest film, Fall, was in danger of receiving an ‘R’ rating from the MPAA for the number of ‘F’ bombs dropped in its 1 hour and 47 minute run. , what would he have done if he was a rational person? would: he used artificial intelligence Digitally alter an actor’s performance to turn swear words into more palatable words.stroke of flicking Genius, if you ask us.

For those interested: about 35 “F” words It was between a PG-13 rating and an R rating. Mann’s dilemma then led him to try to find a way to preserve the film’s integrity without reshoots or dubbing. Of course, this is harder than it sounds.

In traditional movies, you just bite the bullet and reshoot. It’s expensive, but it’s better than the audience losing their sense of immersion just by hearing different lines from the actors saying them clearly on screen. Reading lips in 4K is much easier.

But Autumn is not a traditional movie. It’s a horror movie about her two women climbing a 2,000 foot radio tower. I won’t spoil anything, but it was actually filmed. So they built a huge tower and filmed the actors on top of it.

Thinking about the logistics involved, it becomes clear why reshoots aren’t really an option. Not only did they cost even more than usual, but often the vulgar language was ad-libbed by the actors because of the emotion of the scene.

This means Mann likely got the performance he was looking for. It can be assumed that no director will be happy to erase such trifles as bad language.Unfortunately, usually R-rated movies only about half What their PG-13 counterparts should pay.

Luckily for Mann, he happens to be in a unique position among Hollywood directors. perfect AIis a technology company that specializes in digitally altering video to match an actor’s performance to a dubbed audio track.

For Fall, Mann and the Flawless AI team had to record actors saying different phrases, such as “flick,” to replace the film’s vulgar language. Both the new audio and video were then run through the company’s bespoke neural network, or what it calls “True Sync” technology. This essentially allows the team to deepfake the actor’s face and say new lines.

As far as the results are concerned, “Certified Fresh” Rating No one on Rotten Tomatoes noticed the AI ​​at work when they saw the drama unfold on the big screen.

Not only is the system more cost-effective than reshoots, Mann says it is also relatively easy to implement in production. He told his TNW in an email that about 80% of the workflow is automated and designed to handle everything at Post. No special equipment or instructions are required during filmmaking. He even says that using “dubbing” or video dubbing can save up to 50% on reshoots.

But Flawless AI wasn’t just founded to help filmmakers save money in post-production. In fact, the company’s main focus is accessibility. Mann said:

The company was founded after Mann saw one of his previous films, Heist, starring Robert Deniro, in a traditional dubbed format. He found the current dubbing method to be “limited and broken”.

“We saw how traditional foreign-language dubbing is hurting films, with scripts, performances and meaning lost,” Mann told TNW. “This led Max to read White’s paper on neural networks at the Planck Institute, which ultimately led him to found Flawless.”

Moving forward, Mann and the Flawless AI team hope their work will help storytellers reach audiences around the world in a more intimate way. It’s one thing for him to know what the actors are saying because he’s listening to audio dubbed in a language he understands, but with all the nuance and character given in the original performance. It’s a whole different experience to watch them deliver those lines.

We salute Mann and the Flawless AI team for using AI for good. Art belongs to everyone.