Hello everyone!
In name of Third Move Studios, I’m so excited to share a collab that I did with Deep3dstudio, a photogrammetry studio in Kyiv, Ukraine.
They scanned an old man called Gregory with 86 DSLR cameras using Reality Capture for model reconstruction an Agisoft Metashape for texturing.
After that they sent this 10M polygons model to me (and the custom hair in alembic file), so I could transform Gregory into a Metahuman with Mesh to Metahuman plugin.
I did the script and then used AI generated voices (Replica Studios) and AI lip sync (NVIDIA Omniverse Audio2Face) integrated with iPhone Arkit facial expressions, also using the smartphone as a virtual camera. I played with the lights and then rendered out in EXR files with ambient occlusion pass. Compositing was done in After Effects and color grading in Premiere Pro.
I had a lot of fun with this project and I learned a lot.
Here is the final video + making of where I show a bit about the process.
Hope you like it
Thanks!
Wow! Look at all that expensive-looking photogrammetry equipment?! Your montage makes this look so simple, but I am sure this project took many hours of work and technical know-how.
Yes, this is certainly AI-mazing! This collaboration is so zen! Shout out to the old man named Gregory for volunteering his time and his likeness! I wish them peace in Kyiv!
What was the file size of the 10M polygons model? What did you find the most fun with this project, and what are the top 3 things that you’ve learned?
Hi @Get_DOVAH_it
Thank you so much for your comment!
Yeah this project took a lot of work, but I had so much fun with that!
I think the most fun was getting in touch with the guys at Deep3dstudio where they kept sending me some files for testing on Unreal.
Top 3 things that I’ve learned:
Integrating the iPhone arkit with the AI lip sync into one single sequence. Very manual process here, but it was cool.
Getting a custom alembic hair to work with a metahuman.
Blending original metahuman texture (upscaled to 16k) with Gregory’s original 16k texture for more realism.
The file size of the 10M polygons model is 1,09 GB.
Final Unreal project has only 5,72 GB, but the entire folder containing all additional files (for making of as well) has 154 GB.
Hey man, looks neat, but let me give you some feedback.
The pose of the man on Unreal looks like a younger version of the real man. Giving it that extra sense of age-related pose and environment would of gave it more inmersion. On the second hand, the lips (and maybe tongue?) need some manual work done to match the words closer. Also for rendering, I would make sure to export at 24fps or higher (including metahuman facial animation +24fms) so it’s smoother on the facial gestures. Try also the pathtracer rendering mode, might surprise you.
Besides that everything looks great.
I’m currently working on pushing the Metahumans to the limit of realism. Playing around with teeth opacity, adding sculpted scars and extra normal maps.
Cheers
This is extremely beautiful. I’m currently attending college for my computer Animation Degree. I’ve been an artist for about 20 years, and now deciding to take on computer animation in digital art. What would you recommend being such a refined artist for me to focus on, mostly approaching my computer animation?
I really recommend you focus on UE5 for animation, because it has a lot of cool tools and Epic is always improving them. I suggest you take a look at the Learning tab here on Forums and also some “metahuman”, “control rig” and “sequencer” tutorials on Youtube.
Thank you Anderson. This seems very intimidating at first but I believe once I put it in the practice with my art background everything should fall into place. I’m honored to receive your feedback and look forward to create. Maybe I can show you some of my projects and get feedback if possible, I have the excitement of a child, knowing that I’m about to be able to do these sort of things hopefully as good as you . Thank you again. And best regards.
Also, a brilliant job at capturing a scan of a person!
Look at all that expensive-looking photogrammetry equipment?! Your montage makes this look so simple, but I am sure this project took many hours of work and technical know-how.
This collaboration is so zen! Shout out to the old man named Gregory for volunteering his time and his likeness! I wish them peace in Kyiv!