I’ve reached a milestone integrating Face Mocap into Mesh Morpher. This is still an experiment and i have no idea if or when will be available in the public version.
How does it work?
I’m capturing webcam input (in my case, video stream sent from my phone as a webcam).
The stream gets processed using OpenCV : resizing, setting up grayscale and equalizing histogram on the result.
I’m using dlib to identify faces and find face landmarks. Landmark points are used to create Delaunay triangulation (blue lines on the screen).
Then the results gets written to texture.
Well, the hardest part. We need to identify face landmarks on the 3D mesh model we want to create morph targets for.
I think these landmarks points should be manually configured as it should be a one-time task and allows for extra precision.
Using those points we create the Delaunay triangulation.
Here comes the fun part. I’m going to grab a delta for each triangle generated by the landmark points: Before and After.
If delta exceeds a certain threshold then the corresponding triangle in the 3D Delaunay triangle data will apply it.
From here we just need to identify the vertices on the mesh that are intersecting with the closest triangle and apply the delta.
Sounds a bit crazy, but I think this will work.
There are some extra things to finish as of right now:
- Improve Speed
- Improve precision