Realistic Face Markerless Motion Capture

Hello Unreal developers! I´m developing an in-game cinematic trailer for my own IP, so I created this test that has nothing to do with the story of the project, but a lot with the quality I´m trying to achieve! This is markerless motion-capture, then rendered inside Unreal Engine 4. If you want to know more about my future projects, feel free to add/follow me at: Redirecting... or https://www.instagram.com/brav_fg/

I want to thank David Marmor for the base model and blenshapes, and Baongoc Vu for letting me use and tweak his eye material. I was in charge of making the UVs, all the texturing, materials, secondary blenshapes, re-rigging the whole thing for game engine, lighting, mocap setup and animation. One of the main goals while planning this test was not only to show quality, but to know how the engine would manage to handle a complex model. To see if a real time solution could compete with pre-render quality. I was surprised that the engine handles it pretty well, running at 50-60fps at 1080p, and bare in mind that is a very high poly model because is a test intended for cinematic quality, also have complex shaders and some textures are even 8k!

Creation Process

More images of the creation process:

408013f8bc17cf4c61c07b23ef6e8859de950db5.jpeg

c155bb2b3686e9b4d2751adaffa22058c221d29f.jpeg

ac749597a74af31dfc34e638bdbb9bd86f7d1b35.jpeg

749ec27a1855cd45f68451da18baaff8bb737bf6.jpeg

27898eb4e5cfedc6d733907b913e718518a6bbf5.jpeg

How you capure the video? Kinect?

cool
capture face rig?

Yes I use a Kinect for Xbox 360 along with the software Faceshift, an old version I have from 2014. Sadly this software is no longer for sale, but another similar solution could be Brekel Face 2.0 :slight_smile:

Sorry, do you mean how I capture the movement? or how does the rig of the face (joints, blendshapes) looks like?

This looks great. For me I couldn’t stop focusing on how still the nose is compared to the lovely natural ‘twitching’ of the rest of the face though. There are moments that would cause the nose to wrinkle a little bit, or the nostrils to flare and twitch. This isn’t something that could be captured probably, but would need to be added after by the animator. Great job though

level God! that looks very awesome. please @bravFG, can you tell us how do you did the rig of face, how you organize the blendshapes and a peek view of joints.

Outstanding work! Congratz!

I loved the model, I would like to see another video with the main light moving around the head… because it looks like you are not using a mask to tell where subsurface will be more relevant (maybe you are using, but cant tell by the light position) as the ears looks too much solid look. As for the eyes, I really loved how the pupils contracted after a blink, thats hard to see around in many works… need a little bit of work on the moisture at them, they look a bit dry, besides the specular visible there… I really believe you will achieve what you want soon! Keep going and I hope see your work end results!

Thanks, in the rig there’s a subtle movements that are associated to other movements of the face. For example, when the mouth smiles the nose move a little bit. So when the animation is recorded, those secondary or associated movements also move along, I decide to leave them, but yeah I could also clean them up. :smiley:

Thanks so much! sure no problem, hope this image helps!

Thanks! I was using a translucency map to tell the subsurface where to be more relevant, but I also have several mask that controls parts of the face independently. One of those mask controls the ears, I could easily tone the roughness or transculency back up. :slight_smile: Thanks for your suggestion on the eyes, I´ll work on that, eyes are a difficult subject. I animate the pupils via sequencer :smiley:

Do you plan to release the project file?

Not really, this its just a test for me, so no plan on releasing or selling or anything like that. But I was given the oportunity to create a breakdown involving each aspect in the creation of this test, you can read it here :slight_smile: Realistic Human Character Study

Here are some gifs to better understand how it all works:

Mocap recording:

http://i.makeagif.com/media/8-20-2017/YSed_K.gif

Roughness control:

http://i.makeagif.com/media/8-20-2017/SjOkMV.gif

Normals control:

http://i.makeagif.com/media/8-20-2017/BuYLSZ.gif

Mask control:

http://i.makeagif.com/media/8-20-2017/PxsIjD.gif

Wow - I had Faceshift in 2014 as well. Loved it. I thought there was no way to use it anymore though?

You can still use it with an old licence, but sadly, since Apple bought it, you cant purshase it anymore :frowning:

Hi BraV,
excellet job!

Can we have the skin material?

Thanks.

Thanks for sharing!

The lack of detail/expression in the eyes in disturbing…