Metahuman Animator - curve optimisations

Development Character & Animation
, ,
1

Hi, I was hoping to get some guidance on some optimisations I’m looking into for Metahuman Animator.

The aim is to reduce the Disk Space + Memory size as well as runtime performance of RigLogic. With the main motivation being to allow Metahuman Animator-generated facial performances to be used for crowd NPCs’ dialogue, banter and barks.

Metahuman Animator produces anim sequences that drive facial animation through curves which are parsed through RigLogic to drive the bone positions and morphtargets, which makes them much more Disk Space and memory-efficient than normal anim sequences which have Animation Track data instead of only Float Curve data. Though this comes at some runtime performance cost from RigLogic actually parsing the curves.

However, MHA still produces a lot of float curves still that maintain a consistent value of 0 because the controls aren’t actually being animated. Removing these curves doesn’t seem to break the animation and further reduces the Disk Size/Memory Size of the asset.

Is there a reason to keep these seemingly useless curves around when they’re not actually animated? Would removing them through an AnimModifier or editor utility be a viable method of further reducing disk and memory sizes of the animSequences?

With enough animations these can become sizeable Disk Space savings.

Secondly, removing unneeded curves should also positively influence the costs of RigLogic as you’re feeding it less data to iterate through.

Is there an out-of-the-box way to remove curves through the AnimGraph? I could only find ways to override curves but this doesn’t actually ‘remove’ the curve from being passed into RigLogic.

Although mesh LOD-ing reduces costs of updating bone positions, the RigLogic costs are unaffected as long as it parses all curves present in the facial animation.

To reduce RigLogic costs I’m considering the viability of LOD-ing the curves themselves. So based on the predicted LOD (or through Significance Manager’s calculated significance) you could filter out curves that don’t do much at larger distances.

The idea would be to have e.g. LOD 4 remove all curves except ctrl_expressions_jawopen, at LOD 3 we also pass through ctrl_expressions_eyeblinkl and ctrl_expressions_eyeblinkr etc. etc. Basically LOD-ing our curves to reduce the work for RigLogic to actually parse them.

Additionally, it seems Livelink and stADA isn’t designed for use with crowds of NPCs as the system relies on pre-configured Livelink Sources. The blueprint exposed API doesn’t allow for creation of new Sources at runtime.

Is there future intent that streaming audio could be used for simultaneous lipsync of crowd NPCs?

This could avoid the hassle of needing to maintain 1000s of assets for all lipsync in all languages of NPC barks and banter.

So far it seems streaming audio based lipsync is mostly not intended for use at large scale.

To recap, my questions are:

  • is there a need to keep seemingly pointless Anim curves on MHA-generated lipsync?
  • does LOD-ing curves to reduce RigLogic costs make sense?
  • is there a built-in way to remove curves in animBP?
  • is Livelink/stADA intended for current or future use of multiple characters doing streaming-audio based LipSync simultaneously?

Thanks for reading through all that and I appreciate any help you can provide! :smiley:

Btw, Matt Lake’s comments on Content Auditing in the UnrealFest talk on Character and Animation Optimization helped me a lot in realising the benefits MHA’s curve-based approach provides for disk size and memory :smiley: Great talk!

[Attachment Removed]

2

Steps to Reproduce[Attachment Removed]

3

Hi Gaethan,

There’s quite a few questions in here so I will do my best to answer them. Please let me know if I miss something though.

is there a need to keep seemingly pointless Anim curves on MHA-generated lipsync?

These can be removed without impact. In 5.7 we added a new option on export (Remove Redundant Keys) that removes redundant keys from the curves. It’s on by default and means those curves will only have a single key and take up little space. Also we’d recommend using the Animation Compression Library to further reduce the animation asset sizes.

does LOD-ing curves to reduce RigLogic costs make sense?

Not to reduce RigLogic costs, only to reduce size. The curves passed into RigLogic have no impact on the computations. For curves not passed in it will take a default value and then use that for the computations as a value is still needed.

There are 3 options to improve RigLogic performance by reducing the number of curves used in calculations:

  1. As of UE5.6 RigLogic has additional optimizations. Curves that are not needed for the LOD being calculated are now excluded entirely. So within RigLogic the “LODing” of curves happens automatically to improve performance.
  2. In UE5.5 you could try and achieve similar by reducing the number of LOD’s present in the DNA entirely. For curves that are not used in any LOD within the DNA, they are excluded from RigLogics calculations. So if you are only ever showing a MH at LOD 5, removing the unused LOD’s will improve RigLogic performance.
  3. Remove that curve from the DNA entirely. You could manually modify a DNA to remove a curve and exclude it from use at any LOD. However, this is a complicated process to do right and you are starting to impact visual fidelity, so it’s not recommended.

Is there a built-in way to remove curves in animBP?

No. With the 5.6 optimisations to riglogic we don’t think this should be necessary.

Is Livelink/stADA intended for current or future use of multiple characters doing streaming-audio based LipSync simultaneously?

There’s no reason you shouldn’t be able to have multiple LiveLink sources driving characters at the same time. However, as you say the blueprint interface was limited, meaning this would need you to write C++ code to achieve it. As of 5.7 the blueprint interface was updated so creating these sources in blueprints should now be possible. You can see examples here https://dev.epicgames.com/documentation/en\-us/metahuman/blueprint\-examples.

My hesitation here would be the processing cost of these streams. So far we’ve focused on content creation use cases, rather than in-game streaming and so you may find that the overhead of processing all these streams is a lot.

Glad to hear the UnrealFest talk was useful! A lot of effort goes into them so always good to hear that’s been worthwhile.

[Attachment Removed]

4

Hi Colin,

Thank you for answering my questions. After testing out some of the things you mentioned, I found the following:

Remove Redundant Keys

Remove Redundant Keys does not actually remove the curves themselves. This means there are still curves with constant values of 1 from start to end of the animation that don’t actually contribute anything. Would love an option in the future to Remove Redundant Curves as from my findings that would save additional Disk Size as previously mentioned.

Regarding ACL, since this became the default in UE5.3 we are using this. Though I’ve not looked into tweaking ACL’s compression settings further… Appreciate the reminder that this is an option.

LOD-ing curves

It’s great to hear there have already been improvements here.

From the sounds of it, the DNA contains the info on which LODs actually use certain curves.

I’m assuming that means if you were to combine LOD 0 for the Metahuman meshes (Face,Body etc.) through Mutable and then auto-generate further LODs from that merged LOD 0, you run the risk of RigLogic assuming (based on the Metahuman DNA Data found on the AssetUserData for the Skeletal Mesh?) certain LODs have bones even though the merged mesh you re-generated LODs for might retain different bones for which different curves are relevant. Is that correct?

As far as I’m aware there are no additional options for specifying LOD behaviour (e.g. target tris/bone counts for specific LODs) in the Metahuman Character asset. So the ‘relevant curve’ info in the DNA would be based on which settings you used during Assembly? E.g. Assembly: UE Cine (Complete) -> DNA indicates at LOD 0 all curves are relevant. Whereas Assembly: UE Optimized (Low) -> DNA indicates at LOD 0 only a fraction of all curves are relevant.

It sounds like using Mutable and afterwards auto-generating further LODs might not really be compatible with RigLogic’s optimizations in this case.

I’m aware Mutable can merge other LODs as well but this has its own issues. Any other meshes such as custom clothing and accessories would need their own LODs already set up. Tweaking the merged mesh’s LODs afterwards (e.g. to further reduce tri count) would also become more complex in this case as you’d need to tweak the tri counts on all your individual meshes and then merge them through Mutable again.

Auto-generation approach from a merged LOD 0 initially seemed like the most flexible way to go about merging Metahuman meshes for these reasons.

multiple LiveLink sources driving characters at the same time

After testing the new blueprint nodes in UE5.7, I find this is technically true but not actually in a way useful to games. Metahuman (Audio) sources use Audio Devices found on the hardware. On a PC this is typically the Microphone and Headset, virtual Audio Cable and the like. However, it doesn’t seem possible to feed .wav files or SoundWave assets through as an audio source (at least not without delving into the c++ API). These would be the kind of assets I’d expect video games to use moreso than mic audio.

As you say, the system seems focused on content creation use cases as of now.

I’ve shared your doubts regarding the processing overhead. My main goal atm is looking into ease of setup and performance stress testing to determine feasibility of streaming audio for lipsync. As you can guess, it’s currently looking like streaming audio is only feasible in a limited amount of scenarios.

Assuming an offline generation approach (e.g. through Metahuman Performance batch-processing audio to create animSequences) is the better solution, would you have any recommendations on ensuring animations stay up-to-date?

.wav or Soundwaves could get updated after the generation resulting in out-of-date lipsync animations. Do other licensees typically create custom validation and/or utilities that check the revision history of files to auto-detect discrepancies? I’ve got some ideas here already but am curious what approach to managing the generated assets others have found success with so far.

Many thanks for all the info so far Colin! Been very useful :smiley:

[Attachment Removed]

5

Hi Gaethan,

Remove Redundant Keys - Correct, this doesn’t remove the curves. However those curves will now only have a single key so the disk space used should be very small. As a workaround you could probably write a small script that strips those curves out based on them having only a single key on the first frame. Thanks for the feedback though, we’ll consider that as an option for the future.

Mutable/LODS/LOD behaviour - Yes, if you start modifying the bone counts or adding/removing LOD’s these would become out of sync with the DNA data potentially causing problems.

One option is to use the DNA Calibration libraries, the most recent versions of which ship as part of MetaHuman for Maya. These can programmatically modify the DNA through python so you could change the DNA to match your updates. Obviously this does require a good understanding of the DNA structure and so you would want to spend some time getting familiar with it.

While a little old, a good place to start would be these resources:

https://www.metahuman.com/en-US/learning/the-tech-behind-metahuman-creator-face-rigs

https://github.com/EpicGames/MetaHuman-DNA-Calibration/tree/main/examples

These reference the old DNA Calibration library which is no longer compatible with MetaHuman characters created in Unreal Engine 5.6 or later. However, the same API is available in MetaHuman for Maya, changing the import statement to import dnacalib2.

We are aware people would like more flexibility in this area and so it is something we are looking at making easier in the future.

Audio to Animation in Game - Yes, the focus right now has definitely been content creation use cases rather than in-game. So while technically possible, I expect you would need to do quite a bit of work for this to be viable beyond a proof of concept use case right now.

Offline Generation/Asset tracking - There is no built in way to track how up to date the output animation is relative to the input audio. You could stamp a hash of the source audio into the output animation sequence as Asset Metadata and scan your project for “out of date” animation assets using something like python. All of MetaHuman Animator is blueprint/python scriptable so in theory you could even create an automated process to then update out of date assets. There’s nothing built in right now to do this for you though.

[Attachment Removed]

6

Hi Colin,

Apologies for the belated reply.

Thank you, this answers basically all my questions :smiley: Haven’t looked too closely at the DNA calibration yet so I appreciate the links.

Will probably look into automating some tools to find out of date animation assets as you suggested, this seems the most viable approach at the moment.

Feel free to mark this post as answered and thanks again for the detailed response to all these questions.

[Attachment Removed]