odysseyzjh - MetaHuman Audio to Face Runtime

Distribution Product Discussion
1

MetaHumanAudioToFaceRuntime

🎙️ Speech in → 🧑 MetaHuman talking. In real time. Zero baking.

Turn any audio into fully animated MetaHuman facial performance at runtime. An NNE / ONNX inference solver converts speech directly into viseme curves and drives your MetaHuman through LiveLink — no offline processing, no pre-baked animation, no waiting.

Feed it a WAV file or a live PCM stream, and your character lip-syncs instantly. Perfect for realtime voice agents, TTS-driven NPCs, and interactive digital humans.

✨ Key Features

🔊 Runtime Audio-to-Face
Speech → viseme curves, generated and played live. Nothing baked, nothing precomputed.

📡 Dual Input — Files or Live Stream
Play from WAV, or push streaming PCM with BeginAudioStream / FeedAudioStream / EndAudioStream — built for realtime voice & TTS pipelines.

🌿 Procedural Liveliness
Automatic blinks, gaze drift, and subtle head micro-motion keep your character alive in both idle and talking states.

😊 Emotion Layer
Rule-based emotion + optional ML emotion recognition (NNERuntimeORT), with full manual override when you want precise control.

🎚️ Tuning Presets
One-click looks — Realistic Dialogue, Cinematic Close-Up, Expressive Stylized, Live Stream Presenter — or go fully custom.

🧩 100% Blueprint-Driven
~90 tunable parameters and complete events (frame generated, playback state, emotion recognized, errors) — all exposed to Blueprint.

👁️ In-Editor Preview
See facial output instantly, without entering Play mode.

⚡ One-Call Function Library
Apply generated frames to a SkeletalMesh in a single Blueprint node.

📋 Requirements

PlatformWin64DependenciesLiveLink · MetaHumanCoreTech · NNERuntimeORTAssetAn existing MetaHuman character in your project

🤝 Build a Complete Digital Human

Pairs seamlessly with the RealtimeVoice plugin:

🗣️ RealtimeVoice → 👄 MetaHumanAudioToFaceRuntime
realtime voice + realtime lip-sync = a fully interactive talking digital human.

2

如果需要测试麦克风对话，需要去火山引擎开通服务

火山方舟—模型广场—Doubao-实时语音交互----点击 立即使用—开通之后会显示 服务详情：实时语音大模型—填入APP ID和Access Token

插件对应的api文档是https://www.volcengine.com/docs/6561/1594356?lang=zh

If you need to test microphone conversations, you will need to purchase the RealtimeVoice plugin: https://www.fab.com/listings/0c8543ac-88f2-4bf9-ad4f-ecbffde6fded

Then, you need to activate the service on Volcengine:

Volcengine Ark → Model Square → Doubao - Real-time Voice Interaction → Click “Use Now”. After activation, it will display “Service Details: Real-time Voice Large Model” → Fill in the APP ID and Access Token.

The corresponding API documentation for the plugin is: https://www.volcengine.com/docs/6561/1594356?lang=zh

image
image2064×951 89.9 KB

image
image2205×681 60.7 KB

image
image2199×1446 239 KB

3

Hi, thanks for sharing this plugin. A few technical questions before I consider it. Does it work in a packaged Shipping build, with no editor present, and have you validated that? The listing says UE 5.6 to 5.8, but the demo video looks like 5.7 — have you tested on 5.8 since the June release? Does the solver use Epic’s own StreamingADA and ISpeechAnimationSolver, or your own ONNX model? And for the streaming path (BeginAudioStream, FeedAudioStream, EndAudioStream), what PCM format does it expect (sample rate, channels, bit depth), and is there a minimum chunk size or a measured latency figure? My audio comes from a TTS service as streamed PCM chunks — no microphone, no audio device. Thanks.