PipelineStateCache.RebuildFromDiskCache() Can Causes Extreme App Launch Time.

Development Rendering
, ,
1

Hello,

after a bigger post release patch we get reports from players that their game is “no longer launching”.

Based on our investigations we are sure the game does launch, but can take several minutes to do so on first launch after applying the patch and were able to find reproduction steps to cause a extremely long app start up times (see Steps to Reproduce).

Unfortunately, we noticed, that our platform splash screen is not packaged into the game and we never noticed, so currently there is not even a splash screen, but at least that is an easy fix. Our emergency solution would be to fix the missing splash screen and show a warning in the splash screen, that sometimes app launch can sometimes take several minutes.

However, that is not a real solution since there might be other reasons than a patch that could cause the cache to rebuild from disk, we assume, like driver changes.

We already familiarized ourselves with https://dev.epicgames.com/documentation/en-us/unreal-engine/optimizing-rendering-with-pso-caches-in-unreal-engine and found the Unreal Insights Video about Game Engines & Shader Stuttering and believe to our knowledge we set everything up correctly. We have PSO Precaching enabled and have a runtime loading screen step that loads assets of which we believe will be most effective in reducing Shader Stuttering in the released version of the game.

We do not bundle compiled shaders into the game.

Do you have any pointers what we can do, to either make PipelineStateCache.RebuildFromDiskCache() faster or if that is not possible, show progress information about it to the user?

We tried updating the information in the splash screen but the information does not seem to update.

I do not know what files to attach to help, so if you think of something, please let me know and I will provide.

Thanks in advance,

Ann

[Attachment Removed]

2

Steps to Reproduce

Observed on Win64 Platform:

  1. Launch the Application for the first time, or ensure it starts as it would for a first time User (-clearPSODriverCache, delete Saved folder and most importantly any .ushaderprecache files)
  2. Wait for the Application to Launch.
  3. Close the Application after FShaderPipelineCache::NumPrecompilesRemaining() reached 0.
  4. Launch the Application again with -clearPSODriveCache flag and observe the Application will appear to stall as soon as the PipelineStateCache.RebuildFromDiskCache() call in D3D12Adapter.cpp is reached. Eventually the Application will continue to launch after the rebuild is done.

Additional Info:

  • Deleting the .ushaderprecache will fix the stalling.
  • We have currently no FPreLoadScreen set up that would wait for NumPrecompilesRemaining before launching. We implemented one for testing purpose to understand timing difference between the PipelineStateCache.RebuildFromDiskCache() and a fresh start recompile. Even if we only waited for GlobalShaders to be precompiled the PipelineStateCache.RebuildFromDiskCache() takes significantly longer than letting everything recompile.
  • FPreLoadScreen are not displayed until after PipelineStateCache.RebuildFromDiskCache() finished running.
  • On reaching our Main Menu our .ushaderprecache files are usually between 30.000 to 50.000 KB in size.
  • After reaching the In Game our .ushaderprecache will usually have increased to a size of 50.000 to 90.000 KB after a Shader Pre Caching step in our loading screen.
  • They might grow further since we only load a subset of all our assets when we run the Pre Caching step in our game loading screens.
    [Attachment Removed]
3

Hi Ann-Kathrin,

It looks like you are using the D3D12 PSO disk cache, which we do not recommend due to reported instability issues reported by other licensees. How are you hitting that code path to PipelineStateCache.RebuildFromDiskCache()? Are you using any custom engine modifications or CVars to enable it?

[Attachment Removed]

4

Hi Tim, thanks for your answer!

Can you elaborate on “D3D12 PSO disk cache, which we do not recommend due to reported instability issues reported by other licensees”. We followed documentation (linked above) on how to set up PSO Pre Caching and nothing beyond and our intention was to only use PSO Pre Caching.

Is the PSO disk cache something else then the “normal” GPU Hardware Cache? Or does this refere to the .ushaderprecache that get written by the engine?

We are hitting PipelineStateCache.RebuildFromDiskCache() during normal launch:

…

FEngineLoop::PreInitPreStartupScreen (LaunchEngineLoop.cpp)

RHIInit (LaunchEngineLoop.cpp 2971)

FD3D12DynamicRHI::Init() (DynamicRHI.cpp 319)

FD3D12Adapter::InitializeDevices (WindowsD3D12Device.cpp 1385)

FD3D12PipelineStateCache::RebuildFromDiskCache() (D3D12Adapter.cpp 1400)

Let me know if you need to know about anything above PreInitPreStartupScreen.

We didn’t do engine modifications.

We hit it from vanilla launch code in 5.6

Regarding CVars: Are you thinking about specific CVars that I could check the state of?

As far as we know, we did not take intentional measures to enable anything but PSO Pre Caching as described in the linked documentation. But the person that set it up is no longer in the company and I am only looking into the issue now.

[Attachment Removed]

5

Update, I did check my colleagues commit history again and it looks like he explicitly enabled these two CVars:

+CVars=D3D12.PSO.DiskCache=1

+CVars=D3D12.PSO.DriverOptimizedDiskCache=1

[Attachment Removed]

6

Unfortunately, the term PSO cache is extremely overloaded, so I totally understand the confusion. Those two cvars should be turned off, since we now have a separate PSO cache file manager (FPipelineFileCacheManager) that replaces the driver’s management of the bundled PSOs of the disk cache. How does your game behave with those cvars turned off? I just checked the code path, and if you turn off the cvars, the Init() function will skip over opening the disk cache.

[Attachment Removed]

7

I disabled the two CVars as proposed.

I defined some test cases and performed a Setup Step before each Test Case. All starts after the test cases had around 400 Precompiles Remaining and started very quickly.

Expected Observations: No more stalling between PlatformSplashScreen and PreLoadScreen. *.ushaderprecache files and *.upipelinecache files no longer get created.

Unexpected Observations: NumPrecompilesRemaining seems to get stuck at some point, if the Disk Cache files still exist.

Setup:

1. Acquire the Released Build (via Steam betas).

2. Delete Saved Folder.

3. Launch with -clearPSODriverCache.

4. Start New Game to Run Game Loading Step that Pre Loads Assets to trigger Pre Caching.

5. Assumption: We are starting into the game now as if we had a first install experience.

Test Case 1:

1. Switch to Patch Build (via Steam betas).

2. Run the Patch Build without -clearPSODriverCache.

3. No Stalling observed between Platform Splash Screen and Preload Screen.

4. Preload Screen Displays:

4.1. FShaderPipelineCache::IsPrecompiling() is 1

4.2. FShaderPipelineCache::NumPrecompilesRemaining() counts down, but appears to gets stuck at some point between 400 to 600. Process is still responsive according to the Task Manager.

5. Kill the process in Task Manager. And delete lingering .ushaderprecache files and .upipelinecachefile

6. Restart. Still no stalling and NumPrecompilesRemaining go down to 0 now.

Test Case 2:

1. Switch to Patch Build (via Steam betas).

2. Run the Patch Build with -clearPSODriverCache.

3. No Stalling observed between Platform Splash Screen and Preload Screen.

4. Preload Screen Displays:

4.1. FShaderPipelineCache::IsPrecompiling() is 1

4.2. FShaderPipelineCache::NumPrecompilesRemaining() counts down, but appears to gets stuck at some point between 400 to 600. Process is still responsive according to the Task Manager.

5. Kill the process in Task Manager. And delete lingering .ushaderprecache files and .upipelinecachefile

6. Restart. Still no stalling and NumPrecompilesRemaining go down to 0 now.

Test Case 3:

1. Switch to Patch Build (via Steam betas).

2. Delete lingering .ushaderprecache files and .upipelinecachefile.

2. Run the Patch Build with -clearPSODriverCache.

3. No Stalling observed between Start Up Screen and Preload Screen.

4. Preload Screen Displays:

4.1. FShaderPipelineCache::IsPrecompiling() is 0

4.2. FShaderPipelineCache::NumPrecompilesRemaining() counts down to 0 as expected. Start Value was around 5000.

Test Case 4:

1. Switch to Patch Build (via Steam betas).

2. Delete lingering .ushaderprecache files and .upipelinecachefile.

2. Run the Patch Build without -clearPSODriverCache.

3. No Stalling observed between Start Up Screen and Preload Screen.

4. Preload Screen Displays:

4.1. FShaderPipelineCache::IsPrecompiling() is 0

4.2. FShaderPipelineCache::NumPrecompilesRemaining() counts down to 0 as expected. Start Value was around 2000.

[Attachment Removed]

8

Hi Ann-Kathrin,

From your tests, everything is working as expected. What I mean is that your tests confirmed the disk cache is broken and should be turned off. Also, you’ll need to delete any remaining .ushaderpipeline and .upipelinecache files for this upcoming patch to generate a clean PSO cache on top of a clean driver cache. You won’t need to delete your caches on subsequent patches, though. I hope that clears up your worries. Feel free to let me know if you have any additional questions.

[Attachment Removed]

9

Hi [mention removed]​ thanks, for confirming this is to be expected!

Thanks a lot for the fast response times and the pointers in the right direction.

My solution for deleting the files would be, to have a Module with a very early LoadingPase (PostSplashScreen) and delete the files in StartUpModule, so players do not have to do that manually.

My test already confirmed that it gets rid of the issues described in Test Case 1 and 2.

Remainig questions I have left, that I’d appreciate confirmation on is:

  • Would you recommend another LoadingPhase than PostSplashScreen?
  • Permanently delete: .ushaderprecache files if we find them.
  • Only fist start up after patch: .upipelinecache and .ushaderpipeline, e.g. if we find ushaderprecache files we assume it is a player with the broken cache state. See below for Code Snippet of deleting the files.
  • We actually never saw a ushaderpipeline file only ever upipelinecache and the upipelinecache still gets created on runtime on App Exit after disabling PSO Disk Cache. I assume that is expected behaviour?
bool DeleteFiles(const FString& Extension)
{
	const FString SavedDir = FPaths::ProjectSavedDir();
	const FString FilePattern = FPaths::Combine(SavedDir, Extension);
	TArray<FString> Files;
	IFileManager::Get().FindFiles(Files, *FilePattern, true, false);
 
	for (const auto& File : Files)
	{
		const FString Path = FPaths::Combine(SavedDir, File);
		IFileManager::Get().Delete(*Path);
	}
 
	return Files.Num() > 0;
}
 
void FStartUpHookModule::StartupModule()
{
	// clean up PSO Disk Cache files, if they exist [Content removed]
#if PLATFORM_WINDOWS || WITH_EDITOR
	if (DeleteFiles(TEXT("*.ushaderprecache")))
	{
		DeleteFiles(TEXT("*.upipelinecache"));
		DeleteFiles(TEXT("*.ushaderpipeline"));
	}
#endif
}

[Attachment Removed]

10

Hi Ann-Kathrin,

I took a look at the code and your questions, and overall, it makes sense to me. I ended up adding in some extra logic. Mainly, it’s about logging when you fail to delete a file. You can find the code here:

bool DeleteFiles(const FString& Extension)
{
	const FString SavedDir = FPaths::ProjectSavedDir();
	const FString FilePattern = FPaths::Combine(SavedDir, Extension);
	TArray<FString> Files;
	IFileManager::Get().FindFiles(Files, *FilePattern, true, false);
 
	for (const auto& File : Files)
	{
		const FString Path = FPaths::Combine(SavedDir, File);
		if (IFileManager::Get().Delete(*Path))
		{
			UE_LOG(LogInit, Log, TEXT("Deleted cache file: %s"), *Path);
		}
		else
		{
			UE_LOG(LogInit, Warning, TEXT("Failed to delete cache file: %s"), *Path);
		}
	}
 
	return Files.Num() > 0;
}
 
void FStartUpHookModule::StartupModule()
{
	// Clean up broken D3D12 PSO Disk Cache files to fix extreme launch times.
	// [Content removed]
#if PLATFORM_WINDOWS && !WITH_EDITOR
	if (DeleteFiles(TEXT("*.ushaderprecache")))
	{
		DeleteFiles(TEXT("*.upipelinecache"));
	}
#endif
}

Q1: Would you recommend another LoadingPhase than PostSplashScreen?

Yes, that’s a good idea, but I would use PostConfigInit instead. The .ushaderprecache files are loaded during D3D12 RHI initialization, which happens around the PostSplashScreen/PreEarlyLoadingScreen window. PostConfigInit is guaranteed to run before RHI init, removing any ambiguity about timing. File deletion doesn’t need anything beyond the config system being ready, which PostConfigInit provides.

Q2: Permanently delete .ushaderprecache files if we find them?

Yes, that is the correct approach. These files are the broken D3D12 pipeline library blobs stored in FPaths::ProjectSavedDir() (defined by PIPELINE_STATE_FILE_LOCATION in D3D12RHI.h).

Q3: Only delete .upipelinecache and .ushaderpipeline on first startup after patch?

That should work, but with a minor caveat. If a user manually deleted only the precache files beforehand, they’d still have a broken pipeline cache, but this logic wouldn’t clean it. However, it’s unlikely your users will delete those files themselves, so it’s safe to ship as-is.

Q4: We never saw a .ushaderpipeline file, only .upipelinecache. The .upipelinecache still gets created at runtime on app exit after disabling PSO Disk Cache. Is that expected?

Yes, both observations are expected. .ushaderpipelines is a cook-time artifact used by the shader code library system and is never generated in the Saved/ directory at runtime. You can safely remove it from the deletion list.

The .upipelinecache file being recreated on exit is normal behavior. That is our custom PSO system, which always saves newly discovered PSOs on shutdown, and is a separate system from the D3D12 pipeline library (“PSO Disk Cache”) that was broken. The D3D12 pipeline library refers to the D3D12 ID3D12PipelineLibrary blobs, not our own UE PSO file cache.

[Attachment Removed]

11

Perfect, thanks!

Yes, the intention is to make the fix as hands off for as many players as possible.

Also now that we know the cause 100% sure and handled follow up issues, we can give official recommendations for those that might still suffer issues because they messed with their files before and maybe even give a hint in the patch notes on what the game logic does to fix the broken cache.

[Attachment Removed]

12

Ok, that sounds great to me. I assume you have everything you need to ship this patch. If you need any more help, please do not hesitate to reach out again.

[Attachment Removed]