I’m currently working on implementing real-time procedural facial animation for my MetaHuman character, driven by audio or text input. Initially, I tried using the OVR Lip Sync plugin, which performed flawlessly in the editor environment but encountered limitations during runtime due to frame sequence requirements.
While many suggest using the alternative solution of Audio2Face, it too requires Live Link and fails to function even after packaging the game. Additionally, I explored ConvAI AI and MetaHuman SDK plugins, but they lack offline support and instead offer only monthly subscription options.
I greatly appreciate any assistance you can provide in resolving this challenge.
I recommend using this item.https://github.com/xszyou/Fay/tree/fay-assistant-editionThe project uses the native OVR Lip Sync service and plays the generated lip data in unreal using websocket. The limitation is that OVR Lip Sync is only available on Windows, so I hope it helps.
You can take iPhone data, extract it and put folder in dataset/data and it will automatically strip the audio and create a dataset.
Just ensure you calibrate your face in the iPhone LiveLink app and use the ARKit shapes (NOT Metahuman Animator).
New updated architecture is coming at the end of the month with software and plugins in the works after that - took a while but we finally have a model trainable with less than 30mins of data in a few hours on a 4090 so its usable in many situations.
In case anyone needs it, I created a plugin yesterday called Runtime MetaHuman Lip Sync that enables lip sync for MetaHuman-based characters across UE 5.0 to 5.5. It supports real-time microphone capture with lip sync, separate capture with lip sync during playback, and text-to-speech lip sync.
