Reliable shader compilation for test cont.

Development Platform & Builds
, ,
1

This question was created in reference to: [Reliable shader compilation for automated [Content removed]

Hi [Content removed] which looks as follows in 5.4:

`/**

  • This will cause the test to wait for the shaders to finish compiling before moving on.
    */
    bool FWaitForShadersToFinishCompilingInGame::Update()
    {
    #if WITH_EDITOR
    static double TimeShadersFinishedCompiling = 0;
    static double LastReportTime = FPlatformTime::Seconds();
    static bool DidCompileSomething = false;
    const double TimeToWaitForJobsToStart = 2.0;

bool ShadersCompiling = GShaderCompilingManager && GShaderCompilingManager->IsCompiling();
bool AssetsCompiling = FAssetCompilingManager::Get().GetNumRemainingAssets() > 0;

double TimeNow = FPlatformTime::Seconds();

if (ShadersCompiling || AssetsCompiling)
{
if (TimeNow - LastReportTime > 5.0)
{
LastReportTime = TimeNow;

if (ShadersCompiling)
{
UE_LOG(LogEditorAutomationTests, Log, TEXT(“Waiting for %i shaders to finish.”), GShaderCompilingManager->GetNumRemainingJobs() + GShaderCompilingManager->GetNumPendingJobs());
}

if (AssetsCompiling)
{
UE_LOG(LogEditorAutomationTests, Log, TEXT(“Waiting for %i assets to finish.”), FAssetCompilingManager::Get().GetNumRemainingAssets());
}
}

TimeShadersFinishedCompiling = 0;
DidCompileSomething = true;

return false;
}

// Current jobs are done, but things may still come in on subsequent frames..
if (TimeShadersFinishedCompiling == 0)
{
TimeShadersFinishedCompiling = FPlatformTime::Seconds();
}

if (FPlatformTime::Seconds() - TimeShadersFinishedCompiling < TimeToWaitForJobsToStart)
{
return false;
}

// may not be necessary, but just double-check everything is finished and ready
GShaderCompilingManager->FinishAllCompilation();
FAssetCompilingManager::Get().FinishAllCompilation();

if (DidCompileSomething)
{
UE_LOG(LogEditorAutomationTests, Log, TEXT(“Done waiting for shaders to finish.”));
DidCompileSomething = false;
}
#endif

return true;
}`Should we additionally call UMaterialInterface::SubmitRemainingJobsForWorld(CurrentWorld) and wait on FinishAllCompilation() as you suggest or should FWaitForShadersToFinishCompilingInGame latent command effectively do the same thing?

Also, I’m concerned the machines we are running on may be a factor here as they are relatively old. As you can see, the CPU alone is about a decade old. I’m imaging that could also be a factor here? (see attached aboutPC.PNG)

[Image Removed]

For additional context, here is an representative example of the command line we run for tests as well leveraging software rendering (which is our standard for core tests to eliminate factors related to GPU HW variability):

C:\path\to\project.uproject LevelName -game -ExecCmds="Automation Now; RunTests System.Engine.Rendering Framework" -FIXEDSEED -BENCHMARK -FPS=60 -WARP -AllowSoftwareRendering -ResX=1440 -ResY=810 -ForceRes -Windowed -unattended -nopause -stdout -RunningLocal -FullSizeScreenshots -d3ddebug -testexit="Automation Test Queue Empty" -ABSLOG="C:\path\to\log.txt" -d3d11

2

Hello,

In the original post you mentioned “Typically, this happens for materials that are loaded more dynamically” so it sounds like UMaterialInterface::SubmitRemainingJobsForWorld(CurrentWorld) may be more applicable to help here because it iterates over all materials and submits them so their shaders are picked up for compiling. FWaitForShadersToFinishCompilingInGame should wait for all known shaders to finish, but perhaps there are Materials that aren’t submitted for compiling and those shaders are missed. If you have a repro case, I recommend testing with UMaterialInterface::SubmitRemainingJobsForWorld(CurrentWorld) because if that doesn’t work then there is likely some additionally dynamically loaded Materials that we need to look at and see why they’re not picked up. Also please ensure that the FAssetCompilingManager::Get().FinishAllCompilation(); call happens after UMaterialInterface::SubmitRemainingJobsForWorld(CurrentWorld).

A couple other relevant functions in the Automation codebase:

UAutomationBlueprintFunctionLibrary::FinishLoadingBeforeScreenshot()

UAutomationUtilsBlueprintLibrary::TakeGameplayAutomationScreenshot()

Is there a particular reason for running with -d3ddebug? That can have an impact on render times.

3

Thanks Alex, we’ll test this further.

Thus far, we’ve maintained -d3ddebug on the command line for our tests just to have more robust debug logging in case there are any graphics-related crashes as we make mods.

4

Thus far, we’ve maintained -d3ddebug on the command line for our tests just to have more robust debug logging in case there are any graphics-related crashes as we make mods.

OK, that makes sense, thanks for the info. If you do find you are getting those kind of crashes only on your servers, you can add even more logging with -d3dlogwarnings which can really add a lot of noise to your logs which is why I only recommend it if you have an issue that you cannot repro locally on a dev machine. Also, -d3ddebug enables a debug layer which can change or prevent Aftermath crash dump results (if you’re using NVidia GPUs) - too see if this is happening you can look in your logs for an error that looks like this:

Aftermath may fail to initialize because of the debug layer (LogNvidiaAftermath: Aftermath enabled but failed to initialize (bad0000a))This is also a common gotcha if you’re trying to debug GPU crashes locally and have -attachPix set on the command line which also enables a debug layer.