Hi! While investigating the slow PIE startup times, I’ve found that quite a bit of time is spent waiting for shader compilation tasks, even when all shaders have already been compiled on the previous runs. What happens is the following:
At some point during startup process, multiple shader compilation jobs (5212 in my example, see attached screenshot) are kicked off, fully saturating 12 worker threads for almost 10 seconds.
A bit later, main thread executes `UVolumetricCloudToolsGeneratorComponent::CreateRenderState_Concurrent`, which calls `UMaterialInterface::EnsureIsComplete`, which blocks main thread until specific shader is compiled. In my example, it blocked the main thread for 4.5 seconds.
From what I see, there’s no mechanism to prioritize shader compilation tasks that some other thread is blocked on.
The worker thread spends ~90% of that time preprocessing the shaders - then it uses preprocessed code to calculate the DDC key. This DDC key is then used to lookup the compiled shader in the cache.
Requiring full preprocessing pass just to calculate the key seems very wasteful…
To clarify, is this occurring on every startup (back to back without engine restarts)? Doing shader compilation every PIE launch doesn’t like intended behaviour, and I don’t believe I’m seeing the same thing.
Also, this appears to be under PIE Interact, so PIE has actually started at this point. Based on what appears to be the bar for PIE startup to the left of the current bar, this appears to be the first frame of gameplay, so BeginPlay would be going off at this point.
I’m not seeing UVolumetricCloudToolsGeneratorComponent in the stock engine, so I can’t see the code involved. One place I’m familiar with EnsureIsComplete is for writes to a canvas. It’s possible that’s causing a block on the first frame until the shader is complete.
> To clarify, is this occurring on every startup (back to back without engine restarts)? Doing shader compilation every PIE launch doesn’t like intended behaviour, and I don’t believe I’m seeing the same thing.
No, this occurs on first PIE startup in a given editor session (subsequent PIE startups are generally fast enough to not be an issue).
> Also, this appears to be under PIE Interact, so PIE has actually started at this point. Based on what appears to be the bar for PIE startup to the left of the current bar, this appears to be the first frame of gameplay, so BeginPlay would be going off at this point.
Yes, this indeed happens during BeginPlay.
> I’m not seeing UVolumetricCloudToolsGeneratorComponent in the stock engine, so I can’t see the code involved.
Thanks - I didn’t realize it’s a customization (since someone placed it into Engine directory) - that’s good, I might be able to fix this specific component to remove the stall. Hopefully nothing else would do a similar wait right after It still feels very inefficient to preprocess all the shaders on every editor boot, but if it’s not on a critical path, that’s not a big deal
Sure, but wouldn’t it be much faster to use raw shader sources without any preprocessing + the defines as inputs to the hash function that generates DDC key?
This is the way it used to work and it led to a lot more shaders being compiled. The DDC key is based on the preprocessed source so we can deduplicate, if preprocessing is skipped or done later a lot more shaders are compiled.
Ideally we’d be doing this work in the background and using the default material until the unfinished jobs are completed.
Just wanted add a reminder that Epic is on holiday break from 6/30 - 7/11, returning on 7/14. Confidential issues will be unanswered during that time but non-confidential issues may still be answered by other licensees.
Preprocessing shaders every editor start some sense. Changes to .usf and .ush files are picked up on editor restart (There is a command to recompile as well). Since these could have changed on disk between boots, and define parameters passed to said shaders may have changed with code changes, validating all shaders makes a degree of sense.
Sure, but wouldn’t it be much faster to use raw shader sources without any preprocessing + the defines as inputs to the hash function that generates DDC key?
It still feels very inefficient to preprocess all the shaders on every editor boot, but if it’s not on a critical path, that’s not a big deal