Issues with Audio2Face Animation Sync and Audio Transmission in UE After Complex Execution

Development Character & Animation
,
1

Hi everyone,

I’m currently working on a project that involves using NVIDIA’s Audio2Face to sync animations to Unreal Engine. The pipeline also includes audio transmission and processing within UE. However, I’m encountering two major issues:

  1. Animation Sync Fails After Running Complex Functions in UE:
    When I run some complex functions in UE (such as speech-to-text, AI processing, converting text to a wav file, and streaming it to Audio2Face), the animation from Audio2Face fails to properly sync with UE. However, if I skip these functions, the animation syncs correctly. This suggests that the additional processing tasks in UE might be interfering with the animation synchronization.
  2. Audio Transmission Issue:
    While the animations sync properly during a clean pipeline (without running the additional UE functions), the audio sent from Audio2Face over TCP does not arrive in UE. I have verified the audio is being sent in the correct format (WAVE:48000:1:16 and WAVE:48000:1:32), and the socket connection appears to be established successfully. However, UE doesn’t seem to process or play the audio.Audio transmission and animation seem to be completely independent (audio over TCP sockets, animation over LiveLink), but there appears to be some issue in handling both simultaneously.

Here is a summary of my setup:

Animation: Sent via LiveLink from Audio2Face to UE.
Audio: Sent via TCP from Audio2Face to a listening socket in UE.

What I’ve Tried:

  • Tested different audio formats (48000 Hz, 16-bit PCM, 32-bit float) with no success.
  • Verified audio data transmission via logs in Audio2Face (audiosender.py shows data being sent).
  • Reduced the complexity of UE functions to identify the issue.
  • Ensured that UE is listening to the correct socket and port (default 12031).

Questions:

  1. Why does the animation sync fail when complex functions are executed in UE? Is there any way to prioritize LiveLink updates to avoid interference?
  2. What could be causing the audio sent from Audio2Face to not be processed or played in UE? How should the TCP stream be handled for audio in UE?

If anyone has faced similar issues or has insights into handling audio and animation synchronization in this setup, your help would be greatly appreciated!