I run an experiment that might be fun to dissect: a fully autonomous, interactive live talk show hosted by two MetaHuman twins in UE 5.6 — on air 12 hours a day, 7 days a week.
Topics come from real news feeds, dialogue and voices are generated locally, NVIDIA Audio2Face drives the lipsync via LiveLink, and viewers can talk to the twins in chat: they greet you and answer questions live on air.
The part I want to improve: when the twins speak, the lipsync is accurate but the rest of the face stays too quiet. Real speech carries into cheeks, brows, micro head motion, emphasis nods. Right now it reads as “a talking mouth on a calm face.”
What I’m doing: A2F blendshapes via LiveLink, blended with an idle body anim; I add procedural blinks, gaze shifts and small head motion in the idle layer. My question for people who’ve fought this: what are your favorite techniques to make speech propagate naturally into the whole face and head? Additive animation layers on top of A2F? Control Rig post-processing driven by audio amplitude? Curve remapping of A2F output onto brow/cheek shapes? Something else entirely?
And honestly, tips on anything else you notice are just as welcome — fresh eyes catch things I’ve long stopped seeing.
I think your instinct to treat the A2F result as the base layer and add secondary motion on top is probably the direction I’d explore first. If the lip sync is already accurate, I wouldn’t try to make the audio signal drive the entire face directly, because amplitude alone doesn’t really tell you whether a moment should produce a brow raise, cheek movement, or a head nod.
I’d separate the motion into a few layers: A2F for the speech-driven mouth and facial shapes, procedural systems for things like blinking and gaze, and a small additive layer for conversational emphasis and head movement. The important part would be keeping that additive layer subtle enough that it doesn’t fight the solved facial animation.
For the brow and cheek movement, curve remapping could be interesting if you derive it from specific phoneme or expression information rather than simply overall volume. You could also introduce some smoothing and a small amount of temporal variation so the secondary motion doesn’t look mechanically synchronized with every syllable.
For head motion, I’d probably avoid directly mapping amplitude to rotation. A slower envelope with thresholds and cooldowns could work better: occasional nods or small rotations during emphasized phrases rather than continuous movement whenever the voice gets louder.
Since you’re running this continuously for such long periods, I’d also pay attention to repetition. Even a good procedural blink or nod system can become noticeable if the same timing patterns recur too often. A little controlled randomness within sensible limits could make the behavior feel much less synthetic.
The MetaHuman facial rig and Control Rig give you room to build these additional layers without replacing the Live Link/A2F input, so I think that separation is worth experimenting with.
I’d be particularly interested in seeing whether you get a bigger improvement from better facial-emphasis curves or from very small head/neck movements. My guess is that the latter will make the biggest difference once the lips are already convincing.
The layering you describe matches how it’s set up now (A2F as the solved base, procedural blinks/gaze in an idle layer), so it’s encouraging that the advice points toward building on that rather than replacing it.
The “slower envelope with thresholds and cooldowns” idea for head motion clicks with me the pipeline already uses cooldown logic elsewhere, so occasional emphasis nods instead of continuous audio-driven rotation feels very doable. And noted on repetition: at 12 hours a day, any fixed timing pattern gets exposed sooner or later.
To make it concrete, the two most visible issues on air right now are: (1) the quiet face during speech cheeks and brows barely joining in and (2) the head transition when a twin looks down at her laptop, or snaps back up the moment the other starts talking. Sometimes that transition is smooth, sometimes abrupt and a bit jerky. Curious how people here would attack that one: longer blend times, inertialization, or driving the gaze/head target through an interpolator with its own easing?
One more thought on detecting “emphasized phrases” for the nods and brow accents: since the dialogue text is generated before TTS, I technically know where the emphasis should be before the audio even exists. Has anyone driven emphasis animation from the script side rather than from audio analysis?
I’ll experiment with the additive layer first and report back here with results.
Quick update, faster than promised: the additive layer is in.
I took the script-side route I mentioned above. The dialogue generator was already writing stage directions with most lines, things like (nods), (smirking), (raising an eyebrow). Those now get parsed out and translated into a small additive layer on top of A2F: head deltas for nods and shakes, brow and eye-squint blendshapes for the rest, each played once per line with a soft sine envelope so it never fights the lipsync. First tests today: an emphasis nod landing exactly on “that’s a solid point” does more for liveliness than I expected. I also had to boost the amplitudes about 50 percent from my careful first values. Talk show hosts apparently play their reactions bigger than real life, who knew.
Any suggestions on reaction shots? Right now directions only animate the twin who is speaking, but half of real talk show energy is the listener reacting. I can see two routes: simple procedural rules (mirror the laugh a beat later, small brow raise when the other asks a question), or letting the dialogue model write explicit directions for the listener too. Has anyone done listener reactions for a two-character setup? Curious whether procedural rules read as alive or as robotic, and whether an animated listener starts stealing focus from the speaker.
