Hi everyone!
I’ve been building a plugin that links any UE5 project to a local AI “brain”, so a MetaHuman can talk and act inside the level, with no cloud AI.
Video: https://www.youtube.com/watch?v=ho8m2GuVW3k
In this demo she’s a safety trainer: she walks to an electric motor, explains its controls, and presses the buttons herself, so the motor really starts and stops.
How it works on the Unreal side:
- Body motion is generated live by NVIDIA’s ARDY (text-to-motion) and retargeted onto the MetaHuman, with no animation assets. Walking uses the generated root motion along the NavMesh, so the feet don’t slide.
- She operates the level by calling its own Blueprint functions (StartMotor, StopMotor…). The plugin discovers them automatically and reads the Blueprint’s variables (IsRunning, EmergencyLatched) and its Print String messages, so she knows when a machine refuses (e.g. “press RESET first”).
- Tap anything on the stream to ask “what is this?” or “go there” (line trace from the view).
- Phoneme-timed lip sync, facial expressions, a portrait light rig that follows the camera, and a face close-up with depth of field.
- Pixel Streaming to phones, plus a JavaScript API (e.g. setExpression(“uncertain”, 0.4)).
The AI side runs on the same machine (one RTX GPU): local speech recognition, a local LLM, neural TTS, and a knowledge graph built from the company’s own documents.
Feedback is very welcome, especially on the motion and the lip sync!
