[CONTRACT or ROYALTY or UNPAID] UE Beginner needs help porting a cluster-based facial animation pipeline from Blender to Unreal Engine. I am willing to pay for the work.
Description:
I am using a Python script which was created by an other person in their doctoral thesis consisting of various sub-scripts to implement a cluster-based facial animation pipeline. The whole script and its functionalities should be ported to Unreal Engine, in order to build a small application with a Graphical User Interface (GUI) to run the same animation pipeline in UE which gets Motion Capture Markers and the 3D model as input and outputs the animation applied to the 3D mesh in a .uproject file.
**Rough explanation on how it works: **
After importing cleaned motion capture markers of the desired facial expression into an empty Blender-project, which all have uniformly labeled markers, you can run the script and it will call the following processes, to apply the expression onto the 3d model. The following steps will be made while running the Blenderpython scripts:
- importing the 3D model from a human on which the motion capture markers are to be mapped.
- retargeting: the motion capture markers are transferred to the face and scaled accordingly to fit the new face. For this to work, a manually created correspondence file must be used to establish the correspondence between the marker and the 3D mesh vertices. To avoid having to set all markers manually, a Radial Basis Function network is used to set some markers automatically. After training the RBF network, any point in the source space can be introduced into the system and is converted into a point in the target space by calculating its position according to the trained points and their weights.
- Motion retargeting: In the second step, the motion capture markers were retargeted to the face for the first frame. For all subsequent frames, the motions are scaled accordingly. For this purpose, a scaling factor is calculated and then applied to the motions. After retargeting the motion capture markers, the ratio between new and old markers (before retargeting) is calculated and thus a scaling factor sx, sy, sz is formed, which then scales the motion trajectories correctly. Singular Value Decomposition is used to calculate the frame-by-frame (frame-to-frame) motion displacements of the markers and scale these displacements with the calculated sx, sy, sz factors.
- Skinning: The mesh is bound to the rig after the retargeting process, Motion capture markers are used as the rig, i.e. those that move the bones. Each bone has a specific area of influence on the network, which we also call a cluster. To cluster the face according to the face markers, a space partitioning technique called Voronoi diagrams is used. To determine the influence of each bone on a clustered vertex, interpolation techniques for scattered data such as Natural Neighbour Interpolation and Gaussian Interpolation are used in this script. In all strategies, markers are pinned to their corresponding clusters in the face using Blenders Hook Modifiers. The Voronoi diagram helps to find the same corresponding region on the virtual face and thus enables a correct transfer from human faces to virtual representatives. Nearest Neighbour Interpolation or Gaussian Interpolation is used to determine the weight of the bones in the skinning process. For both techniques, the previously calculated Voronoi cluster is then used to limit the range of influence of the bones on the face mesh.
- Surface Restriction: In order to avoid having wrong areas of influence around the eyes a reweighting is used to restrict the surface to the right area.
Goal:
The whole facial animation pipeline should be ported to work completely in Unreal Engine. I would like to discuss with the person which procedure or idea can solve my problem most efficiently, simply and/or quickly, (for example with using Blueprints, Python Scripting in UE or C++ programming) in order to then be able to implement it together. Unfortunately I’m really new to UE and would like to get help tackling this big “impossible” task with me. I would be glad if we could work on this together on Discord or similar platforms in order that I learn how to use Unreal Engine to finish the task together.
Contact:
E-mail: mb37829214@gmail.com
Discord: Martin#4132
Thank you for your help!