How to animate speech?

Hi. May i know how u do this? Do u provide the service?

What exactly are you interested in? Speech, animation, tracking or dialogue?

Speech recognition is open-source locally working toolkit Vosk, speech synthesis is cloud service (I used Yandex Speech Kit, but Google should work better for English).

Facial animation has three layers.

  • a few pre-defined states: friendly, annoyed, thinking (when awaiting for speech syntesis)
  • lip-sync (I use my plugin, it’s on the marketplace now)
  • facial animation while speaking. This is poorly trained neural net. I used iPhone live link to capture facial animation and PyTorch to train it.

For dialugue I used another my old plugin, but it’s possible to afford on UE4 without third-party solutions. Plus, it makes sense to connect Dialogflow instead.

Tracking & AR. iPhone with NDI HX app is a camera. Is isn’t good solution. You can see a mistiming between video and tracking. For tracking I use SteamVR (Vive Trackers) attached to both chair and iPhone.

No, I don’t provide any service. I just had an interesting idea and I did it.

Good luck!

I wud like to integrate dialogflo n lipsync. Your lipsyn plugin cannot run runtime. Any suggestions or solytions for this?

My lipsync works in runtime (and you can see it in the video), it doesn’t run in real-time. I.e. it can’t animate lips from microphone input. It takes some time to recognize a word.

Hi Thanks for ur reply. Can we keep on feeding new wave files? Will metahuman keep speaking new waves files when we replace the files at runtime? ( published game )

Yes, and you actually can download executable demo from the marketplace page and test it. It can play wave files from your PC.