Reliable shader compilation for automated testing

Development Platform & Builds
, ,
1

Hello, as part of our development and CI/CD process, we run a series of screenshot capture rendering tests against the versions of the engine we build (currently based on 5.4) leveraging the Automation system. We’re finding as we make certain engine modifications, shader compile times are becoming slightly longer such that, occasionally, materials are not yet visible in time for our screenshot capture during the rendering tests as the shader compile time has increased (more frequent since upgrading to 5.4). Typically, this happens for materials that are loaded more dynamically, like material applied to skeletal meshes or particle system materials where the system start is triggered by a Level Sequencer on play. We have a mechanism to wait a certain number of frames before taking a screenshot and are finding we need to adjust this number off and on as things change.

Ideally, we’d want to find a robust configuration such that shaders are guaranteed to be compiled by the time we run our tests/take a screenshot and we don’t have to make adjustments to account for variable timing. I’ve seen past discussion about the JobCacheDDC CVar ([All Shaders compliation on editor [Content removed] which should leverage previous behavior where shader compilation is done as a pre-step vs. during runtime. We tried setting this but still saw some amount of shaders being compiled on game startup, even with the CVar adjusted. We also tried pre-cooking the test projects on the machine prior to running tests but with no discernable change in behavior either (using a relatively standard cook CL: `UnrealEditor-Cmd.exe “path\to\project.uproject” -run=cook -targetplatform=Windows -cookall`).

Is there anything else we can try via configuration or a pre-cook command arguments that could help here and lead to more guaranteed shader pre-compilation, especially for more dynamically assigned materials? Or is this just the way the engine works now?

Thanks,

2

Hello, I’m checking into the specifics of our render tests that have dynamic materials and will update with that information later. You do have the option to force and wait for shader compilation using:

UMaterialInterface::SubmitRemainingJobsForWorld(CurrentWorld); FAssetCompilingManager::Get().FinishAllCompilation();And you can check the remaining number of shader jobs using:

GShaderCompilingManager->GetNumRemainingJobs();However, we usually have a precompile shaders step before our tests using the compile shaders commandlet that takes care of the majority of global shaders and the default world material.

3

To follow up on this, for some of our tests we compile most common shaders with the CompileShadersTestBed commandlet with a command like this:

-Run=CompileShadersTestBed -TargetPlatform=Windows -dpcvars=r.AreShaderErrorsFatal=0 -ExcludeDefaultMaterials -materials=/Engine/EngineMaterials/WorldGridMaterial.WorldGridMaterial -unattendedBut ultimately, we compile all the shaders with SubmitRemainingJobsForWorld() and then block on the work with FinishAllCompilation(). There are additional methods to wait for certain materials to compile, but we wait for all of them before our render tests.

4

Thanks Alex, we’re looking into this to see if it will work for our purposes.