Hi. May i know how u do this? Do u provide the service?
What exactly are you interested in? Speech, animation, tracking or dialogue?
Speech recognition is open-source locally working toolkit Vosk, speech synthesis is cloud service (I used Yandex Speech Kit, but Google should work better for English).
Facial animation has three layers.
- a few pre-defined states: friendly, annoyed, thinking (when awaiting for speech syntesis)
- lip-sync (I use my plugin, it’s on the marketplace now)
- facial animation while speaking. This is poorly trained neural net. I used iPhone live link to capture facial animation and PyTorch to train it.
For dialugue I used another my old plugin, but it’s possible to afford on UE4 without third-party solutions. Plus, it makes sense to connect Dialogflow instead.
Tracking & AR. iPhone with NDI HX app is a camera. Is isn’t good solution. You can see a mistiming between video and tracking. For tracking I use SteamVR (Vive Trackers) attached to both chair and iPhone.
No, I don’t provide any service. I just had an interesting idea and I did it.
I wud like to integrate dialogflo n lipsync. Your lipsyn plugin cannot run runtime. Any suggestions or solytions for this?
My lipsync works in runtime (and you can see it in the video), it doesn’t run in real-time. I.e. it can’t animate lips from microphone input. It takes some time to recognize a word.
Hi Thanks for ur reply. Can we keep on feeding new wave files? Will metahuman keep speaking new waves files when we replace the files at runtime? ( published game )
Yes, and you actually can download executable demo from the marketplace page and test it. It can play wave files from your PC.