odysseyzjh - MetaHuman Audio to Face Runtime

Distribution Product Discussion
1

MetaHumanAudioToFaceRuntime

🎙️ Speech in → 🧑 MetaHuman talking. In real time. Zero baking.

Turn any audio into fully animated MetaHuman facial performance at runtime. An NNE / ONNX inference solver converts speech directly into viseme curves and drives your MetaHuman through LiveLink — no offline processing, no pre-baked animation, no waiting.

Feed it a WAV file or a live PCM stream, and your character lip-syncs instantly. Perfect for realtime voice agents, TTS-driven NPCs, and interactive digital humans.

✨ Key Features

🔊 Runtime Audio-to-Face
Speech → viseme curves, generated and played live. Nothing baked, nothing precomputed.

📡 Dual Input — Files or Live Stream
Play from WAV, or push streaming PCM with BeginAudioStream / FeedAudioStream / EndAudioStream — built for realtime voice & TTS pipelines.

🌿 Procedural Liveliness
Automatic blinks, gaze drift, and subtle head micro-motion keep your character alive in both idle and talking states.

😊 Emotion Layer
Rule-based emotion + optional ML emotion recognition (NNERuntimeORT), with full manual override when you want precise control.

🎚️ Tuning Presets
One-click looks — Realistic Dialogue, Cinematic Close-Up, Expressive Stylized, Live Stream Presenter — or go fully custom.

🧩 100% Blueprint-Driven
~90 tunable parameters and complete events (frame generated, playback state, emotion recognized, errors) — all exposed to Blueprint.

👁️ In-Editor Preview
See facial output instantly, without entering Play mode.

⚡ One-Call Function Library
Apply generated frames to a SkeletalMesh in a single Blueprint node.

📋 Requirements

PlatformWin64DependenciesLiveLink · MetaHumanCoreTech · NNERuntimeORTAssetAn existing MetaHuman character in your project

🤝 Build a Complete Digital Human

Pairs seamlessly with the RealtimeVoice plugin:

🗣️ RealtimeVoice → 👄 MetaHumanAudioToFaceRuntime
realtime voice + realtime lip-sync = a fully interactive talking digital human.

2

如果需要测试麦克风对话，需要去火山引擎开通服务

火山方舟—模型广场—Doubao-实时语音交互----点击 立即使用—开通之后会显示 服务详情：实时语音大模型—填入APP ID和Access Token

插件对应的api文档是https://www.volcengine.com/docs/6561/1594356?lang=zh

If you need to test microphone conversations, you will need to purchase the RealtimeVoice plugin: https://www.fab.com/listings/0c8543ac-88f2-4bf9-ad4f-ecbffde6fded

Then, you need to activate the service on Volcengine:

Volcengine Ark → Model Square → Doubao - Real-time Voice Interaction → Click “Use Now”. After activation, it will display “Service Details: Real-time Voice Large Model” → Fill in the APP ID and Access Token.

The corresponding API documentation for the plugin is: https://www.volcengine.com/docs/6561/1594356?lang=zh

image
image2064×951 89.9 KB

image
image2205×681 60.7 KB

image
image2199×1446 239 KB

3

Hi, thanks for sharing this plugin. A few technical questions before I consider it. Does it work in a packaged Shipping build, with no editor present, and have you validated that? The listing says UE 5.6 to 5.8, but the demo video looks like 5.7 — have you tested on 5.8 since the June release? Does the solver use Epic’s own StreamingADA and ISpeechAnimationSolver, or your own ONNX model? And for the streaming path (BeginAudioStream, FeedAudioStream, EndAudioStream), what PCM format does it expect (sample rate, channels, bit depth), and is there a minimum chunk size or a measured latency figure? My audio comes from a TTS service as streamed PCM chunks — no microphone, no audio device. Thanks.

4

Hi, thanks for the detailed questions.

Yes, the plugin is designed for runtime use and has been validated in packaged Win64 builds, including Shipping builds, with no editor required.

The demo video was recorded in UE 5.6, but the plugin has separate builds for UE 5.6, 5.7, and 5.8. The UE 5.8 version has also been validated separately.

For the solver side, the UE 5.8 version uses Epic’s runtime speech-animation path. It does not require a cloud service or an external audio device; the plugin runs locally.

The streaming path works with PCM chunks from a TTS service, so a microphone is not required. The recommended input is mono, 16-bit PCM, commonly at 24 kHz; other TTS output settings can be passed when starting the stream. There is no requirement that the chunks come from an audio device, as long as they are fed in order. Actual latency depends on the TTS chunk cadence, frame rate, and prebuffer setting; the plugin is built for low-latency streaming and can be tuned for either responsiveness or extra stability.

Currently, the MetaHuman series voice dialogue plugins are divided into the following modules:

  1. MetaHuman Facial Driving Plugin: MetaHuman Audio to Face Runtime | Fab

    Responsible for receiving audio data to drive facial expressions.

  2. Real-Time Voice Plugins:

  3. Optional Plugin - Wake Word Plugin: WakeWord Plugin for UE5 - Offline Keyword Spotting (EN/CN) | Fab

    Allows for customizable Chinese or English wake words.

To check the overall latency and performance, you can test the demo yourself:

Here you can find the step-by-step screenshots for activating Doubao Voice. Once activated, you can enter the key into the demo to test the dialogue.

For all other tutorials, please refer to the online documentation linked within the corresponding Fab pages.

5

Aliyun Realtime Voice 插件的workspace id和apikey按照图中的位置即可找到

You can find the Aliyun plugin’s workspace ID and API key exactly as shown in the picture.

ScreenShot_2026-08-20_090630_467
ScreenShot_2026-08-20_090630_4671920×987 170 KB

ScreenShot_2026-08-20_090915_701
ScreenShot_2026-08-20_090915_7012736×1290 154 KB

ScreenShot_2026-08-20_090806_973
ScreenShot_2026-08-20_090806_9732724×1263 97.8 KB

如果需要配合MetaHuman Audio to Face Runtime插件使用，那么按照下图设置即可
To use this in conjunction with the MetaHuman Audio to Face Runtime plugin, please configure the settings as shown in the image below.

ScreenShot_2026-08-20_093313_058
ScreenShot_2026-08-20_093313_0581920×973 261 KB

ScreenShot_2026-08-20_091939_574
ScreenShot_2026-08-20_091939_5742510×1304 455 KB