Input latency / lag: where is it coming from? how to reduce it?

[BitShapeSoft]

I would like to understand where input latency comes from in UE. I’ve see a big input lag in my initial testing, and I don’t know why that is.

I have a basic project that takes either keyboard, mouse, or gamepad input and triggers a sound. My setup is this:

t.MaxFPS: 120
Laptop display refresh rate: 165Hz
Audio sample rate: 48000
Audio buffer size: 128
Buffers to enqueue: 1

With these settings, my expectations would be an input latency of about 20-30ms (just a guess, based on maybe a two or three 8.33ms frame delay, and +2.6ms audio buffer delay).

However, no matter what I change, I get a delay of about 150-160ms. You can see a video of that here, and I measured the input delay between using the audio:

I am triggering a sound cue from a blueprint, with a single actor instance in my level:

I ran the test at a few different t.MaxFPS frame rates. Here are the results, each with a +/- 5ms deviation:

165fps == 150ms delay
120fps == 155ms delay
60fps == 160ms delay
30fps == 175ms delay
15fps == 240ms delay

Before I move forward, can anyone explain exactly where the latency is coming from? I’ve read this thread, and I’ve tried the different settings on this page, but nothing I do changes makes this delay approach the numbers they mention.

[BitShapeSoft]

So, I’m kind of desperate on this one. I would really, really like to use Metasounds and stay within the UE world for this project. But I have no idea why the latency is so bad, or where to even look to fix it.

Does anyone have any ideas?

[BitShapeSoft]

I don’t have answers yet, but I’ll post my investigations here just as a log.

I followed this audio plugin tutorial (very nice, btw) to run my own synth code. In the OnGenerateAudio function, I checked the NumSamples requested:

int32 ULibpdSynthComponent::OnGenerateAudio(float* OutAudio, int32 NumSamples)
{
	//each tick is one block of audio, typically 64 samples
	int32 ticks = NumSamples / pd.blockSize();
	pd.processFloat(ticks, InBufferPlaceholder, OutAudio);
	pd.receiveMessages();

	return NumSamples;
}

What I was expecting to see was the buffer size I requested in Project Settings >> Platforms >> Windows >> Callback Buffer Size, i.e., something like 256 or 512. However, no matter what settings I input there, the NumSamples requested was always 1024.

Diving into the engine source, I found in the AudioMixerSourceDecode.cpp DoWork() function that the ProceduralTaskData.NumSamples that filters down to my generate audio function is always set to… 8192!

So, my current understanding is that something in my system is setting the actual buffer size to 8192, and my plugin OnGenerateAudio() function is being run in blocks of 1024 samples multiple times to fill it up.

Next step… who is setting the buffer size to 8192?

[BitShapeSoft]

Digging around the source code, here is what I’ve learned.

The platform settings from your project are essentially ignored, when it comes to buffer size. Instead of using the settings provided, the engine just uses a macro to determine buffer size:

#define MONO_PCM_BUFFER_SAMPLES		8192

So, setting the buffer size in your engine project settings doesn’t matter. Changing this macro value from 8192 to 256 and rebuilding the source code gave me a 31ms improvement (128ms to 97ms, from mouse click to sound).

I’m digging around more now, and it’s rough. I’ll post progress here as I find out more.

[echolevel]

Interesting! I know that changing the Callback Buffer Size does dictate the buffer size used by some parts of the Audio Mixer, e.g. SynthComponent and DSP effects, and I thought that this define only applied to PCM data that needs to be decoded async/non-realtime, in advance of being played.

If that were the case, then I’d say that input latency would be improved in your test example by either messing with the precache/loading options on the SoundWave asset or by using some cunning Quartz logic to preload the SoundCue.

But the fact that MONO_PCM_BUFFER_SAMPLES is referenced in AudioMixerSourceBuffer in relation to procedural wave generators makes me wonder… At a glance it seems like that value is being used so that AsyncRealtimeAudioTasks can be done in bigger chunks in order to fill a buffer which, later, will be used for playback at the project’s custom Callback Buffer Size - I know this is how I do it in my soundwave-based multisample synth component. And sure, my input latency’s huge, but I’ve been blaming that on the MIDI plugin

So the really interesting thing is that you got an improvement after changing the define. I think to be sure of what’s going on, it might be best to compare this with what happens when you try different SoundWave loading options; Quartz-based triggering; an AudioComponent playing a SoundWave asset and not a SoundCue (SoundCues are old and weird and occasionally badly behaved).

I’m not naysaying btw - it IS a problem and I’d also like to get to the bottom of it.

[edit] I just re-read and realised you made your own plugin, and discovered your OnGenerateAudio is being run multiple times (in terms of the custom callback size) to fill the buffer (in terms of the hardcoded callback size). So presumably that’s happening for me as well, but I just haven’t noticed it because my focus hasn’t been on low input latency - my tool is for composing and playing back, but since the MIDI input latency is such a big deal to overcome I just haven’t been able to invest any time in it yet. Sort of sidestepping it for now. So yeah…mind blown! Some of what I said above may therefore be irrelevant but I’m following this with interest, anyway…

[BitShapeSoft]

The MONO_PCM_BUFFER_SAMPLES macro is actually used in a number of initializers, including the FMixerSourceBuffer used for source effects (procedural synth components).

My current understanding is that the AudioMixer has its own update cycle / buffer that is on top of the hardware render buffer cycle. So… okay, but if we then set the buffer sizes the same shouldn’t any additional delay go away?

And yeah, things like precache / loading options don’t apply to procedural source effects, and aren’t used in the source code when loading them.

So here is where I stand. I took a slow-motion video to watch the actual delay, then lined it up with the logged frame count / time:

(Note: I read a post that said that running in the editor introduces a lot of latency. So I packaged my app, and ran it standalone, and it had exactly the same latency.)

From the mouse click to the visual update is 41ms. I’m guessing this is just hardware latency, and probably what this page is talking about.

In my example, the screen update and audio ‘go!’ command are done on the same frame. You can see when the screen visually updates, and then there is a 58 ms delay before the audio comes out of the speakers.

(Note #2: this delay happens on both the built-in speakers, and my Focusrite interface which reports that it is running at 192 sample buffer.)

I have no explanation for why the delay from screen update to audio output is so insanely long. None of the docs mention anything that could be causing this.

My next step is to find wherever the actual hardware buffer request is, and just hijack that and put my own buffer in, skipping Audio Mixer entirely. I don’t know what else to try, tbh.

[Bradical]

Wow, quite a detailed documentation of your findings. I have also been struggling with input latency in UE5. Thanks for continuing to share your progress. This gives me a jumping off point to explore myself. I will share anything I learn.

[Bradical]

It turns out that decreasing the Windows Callback Buffer Size from 1028 to something smaller like 256 or 128 was allI I needed to do for good results in my project. I am going for more of a “Live Quantized” feeling so I don’t need razor sharp precision. But I can definitely feel a huge difference between the buffer sizes. I am using 5.0.3. Anyway, thanks again for your thorough exploration of this topic. It definitely helped me out. And good luck on your project!

[Penguinburger]

Hey! I found a solution that worked for me!

I was using Bluetooth headphones (Apple AirPods Gen 2) with a USB Bluetooth adapter. Once I switched to my wired headphones (hooked up to the rear audio output), the audio output felt pretty much instant.

This has driven me crazy and I did the same video recording test to count the frames between click and sound. I hope this helps anybody out there surfing the wild world wide web for answers to their prayers

[vfXander]

Great thread! Any updates on this issue? Did you file a bug ticket?