Create Your Favourite Actor From Nothing But Photos
Written by David Conrad
Sunday, 13 December 2015
Graphics techniques are becoming so sophisticated that it really does seem like magic is being done. Now a team of researchers has implemented a way of constructing a 3D model of a well known actor and using it to create scenes they never played in.
If you always wanted to see John Wayne play the lead in The Good, the Bad and the Ugly you might not have to wait much longer.
The model is first created using photos. The method relies on facial recognition to locate the standard points that make up a face. The face is then warped to place the standard points in the same orientation so that all of the photos are reduced to face-on shots. Then they are wrapped around a standard 3D model customized to give the required distances between the standard points. The warped photos are then used to construct a texture to overlay the 3D model to make it look natural and more like the target.
This amazing enough, but the next stage is to analyse a video of the "driver", i.e. to work out the deformations in the puppet needed to make it follow the driver. This is the real innovation - working out how to convert the movement of one person into those of the actor.
To quote the paper:
"Our answer to the question of what makes Tom Hanks look like Tom Hanks is in the form of a demonstration, i.e., a system that is capable of very convincing renderings of one actor believably mimicking the behavior of another. Making this work well is challenging, as we need to determine what aspects are preserved from actor A’s performance and actor B’s personality. For example, if actor A smiles, should actor B smile in the exact same manner? Or use actor B’s own particular brand of smile? After a great deal of experimentation, we obtained surprisingly convincing results using the following simple recipe: use actor B’s shape, B’s texture, and A’s motion (adjusted for the geometry of B’s face)."
So it seems that it is sufficient to use the movements generated by the driver and simply modify them to take account of the geometry of the actor's face.
Although the researchers don't draw any conclusion from this, it suggests that the character of an actor comes about from displaying a universal human facial expression using the physiology they happen to have. So much for acting...
Take a look a the video to see if you find the techniques convincing:
The possiblities are clear, endless, and slightly frightening if you happen to be an actor.
Microsoft's Project Malmo platform for AI experimentation is now available on Github. The Malmo platform is built on top of Minecraft and uses its experiences and interactions to learn how to interact [ ... ]