FACE ANIMATION PROCESS
The value of the blendshape weight is something that can be moved from one character to another. So what’s the trick in creating hundreds of blendshapes automatically for all character types with the click of a button? To use a preset file for all the joint transforms. The joint transform presets file contains the min/max xyz translation and rotation values of driven keys for each joint that is associated to a control. This settings file is easily customizable per character, and will allow animations to be moved from one character to another without much issue. The ‘driven keys’ provide a barrier so the tracked locators will never transform the mesh into an undesirable position. A blend shape is created after setting each control value to a -x/+x/-y/+y. The control rig is used as an in between for retargeting between characters. The motion capture footage is used to drive the X and y translations of the control rig. The scale of the tracked locators will determine how exaggerated the animation will appear. The rig x and y translations can be exported into UE4 to drive the values of the blendshapes to get the identical animation. In UE4, use the TICK() function and lock the fps to 30 to play the animation. To be in sync with audio you have to take into account dropped frames by using DeltaTime and skip those frames.
Whether you will be creating blendshapes or using fbx skeletal animations, it should not impact the animation process for setting keyframes, transfering animations between character or how anything is going to be setup(it’s just an export process). The control rig, or the imported motion capture data are the only things that should ever be animated directly. If custom blendshapes are built in zbrush initially, or not built until the very end after the animation is completed, the keyframes on the rig will remain the same.
//-------------------------------------------
// Skeletal animations
//-------------------------------------------
Before exporting a skeletal animation, bake the keyframes on the joint hierarchy and export as fbx.
Play the face skeletal animation on a separate animation state as to not interfere with the slot node or montage that will drive the body animations.
Skeletal face animations should be baked on the base skeleton if the character is retargeted onto other characters. The face animations won’t retarget correctly using any of the retargeting methods if the face structure is far different unless you’re transforms are based on rotation in which ‘AnimationRelative’ seems to work the best. For best results, skeletal face animations should be baked onto the character they’re made for and setting the retargeting method to skeleton or animation. To prevent disastrous transforms that occur on retargeted characters that have no animation playing on the face joints, create an idle state that takes in no animation.
Playing a face animation is just about setting a reference to an animation into the final pose. One thing to consider is that UE4 will crash if you’re changing out animation references that are being accessed in the animation state. You can prevent this by setting a boolean value to blend out of the current animation being played and call an event that allows the animation file to be switched out.
//-------------------------------------------
// Blendshape animations
//-------------------------------------------
Blend shape animations are baked onto the control rig and the animation is exported to a text file that will drive the blend shape weights.
The text file can be imported into Unreal by dragging and dropping the text file into the content browser and defining a data type.
I have chosen to have each row in the data file represent a keyframe, and each column to represent a control name for the rig.
The min and max values of the animation can be scaled up or down at runtime by multiplying a float value in order to increase it’s exaggeration.
The animation is played by referencing the data file and then pulling out all the rows of the animation(total keyframes) and then setting the weight value of the morph target with “Set Morph Target”
Here is the blueprint diagram I used to play blendshape animations with synced audio that can loop a set amount of times.
How do you setup blendshapes for a face that contains multiple meshes?
As far as I know, you have to create a blendshape for each of these objects that need to deform. If they are to transform in the same fashion then it’s beneficial to combine the meshes. This is obviously not possible in all cases when you’re wanting to define multiple materials or otherwise but this is no real issue and there is an easy approach to solving this. All the blendshapes for all the meshes attached to the skeleton are all accessable from the skeletal mesh that owns them. So after they’re created you can think of them as a global variable.
The blendshapes that are created have to be identified in some way in order to be used. So you run into issues when you have two objects (teeth and face for example) that use the same animation values but have different blend shape names. There is a simple fix for this, which is to have an adjustable two dimensional array of all the names of the blendshapes that you want to use and their associated animation control name. It definitely becomes an issue when you have different and additional blendshape names for each of the characters but luckily it doesn’t make a difference if you’re trying to modify a blendshape value that doesn’t exist. Although, blueprint makes it very simple to quickly adjust an array containing all the blendshapes that are being used on a character.
In this example, the eyes are a single mesh, the teeth are a single mesh, the face is a single mesh. This shows how to scale a blendshape animation, and scale past the bounds from the 0-1 range creating undesirable animation.
Blendshapes can be supplemental to the face skeletal animations already in place. They can also entirely remove the bones in the face allowing you to quickly skin a character and make adjustments to the face details easy in the future and allowing you to animate the characters face shapes off of the body mesh without having to retarget anything.
This is just my process, I know there are a lot of really good options out there. I have created a tool that automates the process of creating the blendshapes, a face animation rig, setting relationships between mocap locators and rig controls, setting the driven keys from a presets file, saving driven key presets per character, transferring the face animations between characters with different joint placements, cleaning up skeleton transforms and so forth. I have also built a face animation player in blueprint that will allow x and y rig translations to drive the blendshapes weight.
-DOWNLOAD](BFPTools)- For Maya
*Use with caution still undergoing development
Installation: place contents into C:\BFPTools\
run the command in maya: source “C:\BFPTools\UnrealTools\FaceMocapMain_2015.mel”;
Auto create blendshapes: All blend shapes for the character are created with a single button click.
Automatic face rig creation and easy associate mocap data to rig
