Articles, Blog

Face2Face: Real-Time Facial Reenactment | Two Minute Papers #63

November 26, 2019

Dear Fellow Scholars, this is Two Minute Papers
with Károly Zsolnai-Fehér. There was a previous episode on a technique
where the inputs were a source video of ourselves, and a target actor. And the output was a video
of this target actor with our facial gestures. With such an algorithm, one can edit pre-recorded
videos in real time, and the current version only needs a consumer webcam to do that. This new version addresses two major shortcomings:
One: the previous work relied on depth information, which means that we needed to know how far
different parts of the image were from the camera. This newer version only relies on
color information and does not need anything beyond that. Whoa!
Two: Previous techniques often resorted to copying the footage from the mouth and adding
synthetic proxies for teeth. Not anymore with this one! I tip my hat to the authors, who came up with
a vastly improved version of their previous method so quickly, and it is probably needless
to say that the ramifications of such an existing technique are far reaching, and are hopefully
pointed in a positive direction. However, we should bear in mind that from
now on, we may be one step closer to an era where a video of something happening won’t
be taken as proper evidence. I wonder how this will affect legal decision-making in
the future. Thanks for watching, and for your generous
support, and I’ll see you next time!

You Might Also Like


  • Reply Pedro Afonso Oitaven de Sousa May 4, 2016 at 5:31 pm


  • Reply StormCrow May 4, 2016 at 5:59 pm

    Videos of "Hello, I'm an African prince and I would like to give you all my money" incoming.

  • Reply Joseph Gericke May 4, 2016 at 6:43 pm

    the only thing i can think of is what awesome parodys of movies u can make with this

  • Reply Tommy C May 4, 2016 at 9:27 pm

    This reminds me of a process described in Henry Kuttner's "Fury", where a video falsified in such a fashion was used to defame a public figure. Feels surreal how this science fiction technology is no longer fiction.

  • Reply Neoshaman Fulgurant May 4, 2016 at 10:31 pm

    More impressed by the idea of High Quality™ markerless facial motion capture using commodity camera.

  • Reply Kane May 5, 2016 at 12:38 am

    wow, super impressive results

  • Reply Russell Cox May 5, 2016 at 2:59 am

    You bring up a good point about the legal ramifications regarding evidence. What about impersonating someone's likeness?

  • Reply David McSween May 5, 2016 at 7:10 am

    Its been quite some time now since people have considered video to be infallible. Modern CGI can make anything seem real. However it's the context and details that separate forgery from reality. Perhaps a form of Morellian Analysis and demonstration of the "Chain of Custody" that will show veracity of a document.

  • Reply David McSween May 5, 2016 at 7:11 am

    I wonder if the recent deep thought painting tool could be turned to discover forgeries instead of making them?

  • Reply Krishna Penukonda May 5, 2016 at 1:02 pm

    Is there a place where we can get the code to play around with it?

  • Reply UnknownSentinel May 6, 2016 at 12:06 pm

    Is the source code for the demo accesible somewhere?

  • Reply ProGamerGov May 7, 2016 at 12:59 am

    So has the code been released? Or is someone working on implementation of this? Can't find a download link of the software anywhere.

  • Reply Geo Synchronous July 17, 2016 at 12:28 pm

    It will only be used to deceive people. This planet is getting way to far out of control.

  • Reply Andrei Brănescu April 20, 2017 at 8:49 am

    Wait until we'll have the same feature for voice. You'll be able to make Trump say anything you want. 🙂

  • Reply Death Barrier May 29, 2017 at 12:55 am

    So you just use RGB data, how about how do you transfer and modify the mesh from the actor to the target without having the mesh of the actor overlapped on top of the target?

  • Reply Hans Sprungfeld July 28, 2017 at 12:14 am

    How can this possibly be used in a benign way?!?

  • Reply HotchiwawaChronicles August 5, 2017 at 1:03 pm

    Don't believe pictures, don't believe video anymore.

  • Reply Tymski August 18, 2017 at 10:00 am

    How can I get this going on my machine?

  • Reply Adam Lavigne October 15, 2017 at 6:05 am

    love this.i have more opportunities to make movies or series

  • Reply Ha Vu Tran November 7, 2017 at 11:01 pm

    Now films can be lip-synced to different languages 🙂 Towards a better poly-lingual hollywood

  • Reply M. Manel June 30, 2018 at 3:49 am

    where we can download ???

  • Reply Union Village Media November 22, 2018 at 2:52 pm

    but the real question is… when is it going to be in the app store!? LOL

  • Leave a Reply