Inconsistent timing when playing sequential sounds

I am working on a simple dynamic music routine:

  • I have lots of short sound clips designed to be played sequentially (where the order can be changed to alter the mood of the music).
  • They each have their own sound cues and a ticking actor is used to decide which plays next.
  • The sound clips have 1 second of silence at the start and a tail of noise, i.e., a new clip might be played every 10s, but each clip will have 1 second at the start, 10s of tune, and then 2-3 seconds tail - so they overlap.
  • After the actor has ticked to over 10s since the last play, it plays the next sound but jumps forward into the silent part based on the time overshoot (so if 10.02s has passed, I use MyAudioComponent->Play(0.02); where 0.02 is the delay )

This seemed to work well at first, but I noticed that every so often, there would still be a noticeable bad timing (even though the clips overlap, if the timing is not perfect (within about 10-20ms), the beat can be a bit disjoint during the transition). After some trial and error I noticed that the time taken to begin playing the audio component was effected by the framerate considerably. If the framerate was constant (be it high or low), the system worked fine, but if it decreased between plays, the next sound would be late, and if it increased, the next sound would be early.
The time taken to begin playing the audio component from the point “Play” is run on the tick seemed to be relative to 2DeltaTime (this was reached entirely by trial and error listening to the beat while varying the FPS in a stress test setup).
For now I have added in this 2
DeltaTime onto my delay like this: MyAudioComponent->Play(calculatedDelay + 2*DeltaTime); and I hate it, it looks like a truly awful bodge.
What is worse is that every so often I am still getting a delay and I can’t easily recreate it in my stress test scenario. The delay is rare, sometimes I can listen for over 50 transitions and not hear it, then out of the blue there will be a gap. It is not a cumulative effect either as I have known it to happen very early on.
I think my assumption that the delay between the value of UGameplayStatics::GetTimeSeconds(GetWorld()); in the actor tick, and the time the audio file actually starts playing is constant (or a linear function of DeltaTime) is a bad idea.

Is there a more robust way I can implement this sort of accurately timed audio system?

Haven’t gone down that road far enough yet myself, but research so far hints at **timelines **being much better for the exact timing we need for music.

Have you seen this thread? Dynamic music - Work in Progress - Unreal Engine Forums

It looks like this is a good way to deal with timing the start of the sounds, however, this method still seems to rely on the idea that the sound plays immediately once the timeline bound event is fired.

I seem to experience a delay between the point MyAudioComponent->Play() is reached in the code, and the moment sound actually starts playing.

You’re right, I was thinking about accurate triggering, not end result. Have you seen this thread? Delayed Audio in 4.15? - Audio - Unreal Engine Forums

I did go through that thread before posting, I wasn’t sure if it was directly related to my problem; my clips are all of similar length and the problem occurs in the built game similarly to in the editor.

In the meantime I did try out the method with timelines just to see if there was any difference. Here is the blueprint:

The timeline is set to fire an event every 4.57s (It should be 4.57143… but I couldn’t set the timeline length more granularly than this; 1.4ms is not noticeable though). In this example it just flips between two slightly different tracks. Unfortunately, if anything, it is less robust than the setup I originally posted.

I made a recording of this setup, I think it puts the problem into context nicely (this is a packaged build):

BTW - the framerate drop caused by the particle effect is just used for the sake of stress testing, it’s not actually part of the game (thought I would mention this in case I start getting particle system advice:p)

Hmm, hopefully you will find out. My setup: I made all my loops the same length, and used timelines to fade volume in/out on each. The loops were all playing in the background, synced. Had to load them in, wait a sec, then play them, to have all play/sync. Never stress tested it or anything, it was just a tiny project.

I could do things this way, but the system to handle it would quite complex in my case. I think I would need 20+ tracks running in unison to be able to get the equivalent level of control over the music and I’m not sure how resource intensive that is. The other issue is, if for any reason one or more of the tracks ended up out of alignment, the timings would be bad for the entire remainder of the level (I don’t know why there would be an alignment problem, but I’m sure something could go wrong in rare cases, especially at the point of looping). At least in a system where the tracks play sequentially, if there is a hiccup, it is rectified when the next track is played just a few seconds later.

I could do a looping + fade in /out system with as few as 8 tracks running in unison if I could accurately jump forward and backward in time on individual tracks. This just sounds like another great way to introduce a load more potential syncing issues though!

If you’re on Windows and using the old audio engine, the timings are not guaranteed to be accurate in the XAudio2 mixer. This is because, when our audio backend was originally written, the original implementer wasn’t aware of the fact that XAudio2 defaults to doing their commands “as fast as possible” in the interest of reducing latency. What that means is that 2 events fired on the game thread (e.g. from BP) will not necessarily be played at the exact time on the audio render thread. Xaudio2 has support for fixing this through a somewhat complex design using what they call “operation sets”. I had a changelist I was working on where I switched our XAudio2 implementation over to using OperationSets (to sync our audio events in the audio render thread) but it ended up requiring a significant amount of refactoring due to a number of assumptions in our code about XAudio2 calls being synchronous to the render thread. Furthermore, I was in the process of writing a new audio renderer which would solve this problem in a simple way.

This is all to say that this issue should be resolved in the new audio engine. Any 2 audio events which happen at the same time on the game thread are gauranteed to happen on the audio thread at the same time.

As for more generalized sample-accurate timing, I am planning to do a sample accurate scheduler for stitching music systems using a “music component” which allows you to schedule/time precise events.

This sounds very promising, though I am using 4.16 with “AudioMixerXAudio2” enabled at the moment. In an attempt to avoid the problem I was having, I re-worked the music system to deal with synchronous loops as kjekkbart suggested. It is a simpler approach and it has been sounding fine for the last 2 days whilst I have been working on it, however I just now noticed the first bad synchronization in the editor. It is probably an extremely rare event and difficult to track down but I would like to know if there is an approach or something I need to flag to make my new setup more robust.

It is very simple, I just need to make sure that 8 WAV files begin playing at exactly the same time. Once they are going, they loop without issue and I can edit the volumes during the game. Here is the code (on an Actor in BeginPlay):


	
for (i = 0; i <= 7; i++)
	{
		SongComponents.Add(NewObject<UAudioComponent>(this));
		SongComponents*->bAutoActivate = false;
		SongComponents*->bAlwaysPlay = true;
		SongComponents*->bIsMusic = true;
		SongComponents*->SetSound(AllCues*);
		SongComponents*->Play();
	}	


AllCues* is a TArray of SoundCues. The cues are very simple, just wavs set to “looping” connected to the output.

I have been testing a system built around this to death over the last couple of days and out of maybe 200 tries with no problems, I just had one go where several of the wavs were about 0.5s later than the others. I can’t repeat the problem now so it seems to be a rare coincidence, but I certainly didn’t imagine it.

I have a plan where I could use OnAudioPlaybackPercent to check that they are all at the same point on frame 1, and if not, restart them all, and repeat until they are lined up. However, if there is a more elegant solution, I would appreciate it. Is there any way to set these up so they play at exactly the same time without issue?

I relatively recently uncovered an issue with 4.15/4.16 and audio event timing from the game thread. There was a check in that reduced the audio thread priority to below normal so that during your game other higher priority threads can preempt the audio thread. During worse-case scenarios this seems to be causing timing problems. We had a game internally that had issues with their music system and we were able to resolve it by bumping up the thread priority.

If you change some engine code, you can patch your 4.15 version I believe fairly easy with this change I made for 4.17:

https://github.com/EpicGames/UnrealEngine/commit/c1420f923cba85dacd0f57fbad268bda7ecfa328

Basically add this code and then set the audio thread priority in your game’s .ini file to be above normal.

Thanks for this, I think I might be too late to grab it though as that link is giving me a 404. Also we are likely to make the switch to 4.17 as soon as it is ready. 4.15 to 4.16 was very smooth.