Issues with Bundled PSO Cache When Using Chunks

Development Pipeline & Plugins
, ,
1

Hello! We use the bundled PSO cache approach described in the documentation:

https://dev.epicgames.com/documentation/en\-us/unreal\-engine/manually\-creating\-bundled\-pso\-caches\-in\-unreal\-engine

We also use DLC/Chunks to split our content, and the issue is that the bundled PSO cache does not seem to work correctly with chunks.

In the logs, I see that our bundled cache file is opened very early during application startup, while the chunks are not yet mounted. As a result, most entries report missing shaders:

LogRHI: Display: FShaderPipelineCache starting pipeline cache ‘Blitz’ and enqueued 450 tasks for precompile. (cache contains 2532, 2532 eligible, 2082 had missing shaders. 0 already compiled). BatchSize 5 and BatchTime 16.000000.

Later, when the chunks are mounted and their shader libraries are loaded, the engine attempts to open dedicated cache files for the chunks, but those files do not exist:

LogShaderLibrary: Display: Using IoDispatcher for shader code library Blitz_Chunk103. Total 1911 unique shaders.

LogShaderLibrary: Display: Cooked Context: Using Shared Shader Library Blitz_Chunk103

LogRHI: Could not open FPipelineCacheFile: ../../../Blitz/Content/PipelineCaches/Windows/Blitz_Chunk103_PCD3D_SM6.stable.upipelinecache

Based on these attempts to open chunk-specific caches, I assume that UE is capable of generating such caches. However, I could not find any information about this in the documentation.

Could you please clarify the correct setup for using bundled PSO caches together with chunks?

[Attachment Removed]

2

Hi Alexander,

It looks like your shader bundles are not picked up correctly due to BeginPrecompilingPipelineCache() running in ShaderPipelineCache.cpp, which filters all PSO entries around lines 2097-2104 using FShaderCodeLibrary::ContainsShaderCode(). Since chunk shader libraries aren’t loaded yet, your initial PSO bundle entries are permanently removed from the precompile list.

However, you are correct, we do support chunk-based PSO caches, but a platform config variable gates it. Can you try the following steps to see if you can generate a set of bundled caches that are compatible with chunk loading?

1. Override the config setting

In your platform engine config file, add:

[DevOptions.Shaders]

bDoNotChunkPSOCache=False

This overrides the default in Engine/Config/BaseEngine.ini, which disables the feature.

2. Full cook

Run a full cook for your target platform. During cooking:

- FPipelineCacheChunkDataGenerator will produce .cacheinfo JSON files for each chunk, listing its packages and expected output filenames

- The ShaderPipelineCacheToolsCommandlet build command will find those .cacheinfo files and split the merged PSO set into per-chunk files using the stable shader key map to determine which shaders belong to which chunk

3. Verify cooked output

In Content/PipelineCaches/{Platform}/, you should see per-chunk files alongside the base cache:

{LibraryName}_Chunk0_{ShaderFormat}.stable.upipelinecache

{LibraryName}_Chunk42_{ShaderFormat}.stable.upipelinecache

{LibraryName}_Chunk103_{ShaderFormat}.stable.upipelinecache

…

4. Verify runtime behavior

Run your game and check the logs. You should see:

- Fewer “missing shaders” at startup for the base cache

- Per-chunk caches opening as chunks mount:

Shader library state change 1, pipeline cache {Name}_Chunk103 was opened=1

FShaderPipelineCache starting pipeline cache ‘{Name}_Chunk103’ and enqueued N tasks for precompile…

Feel free to let me know if that works or if you need any clarification.

[Attachment Removed]

3

Hello,

Thank you! It seems to be working. Just a small fix still needs to be applied:

https://github.com/EpicGames/UnrealEngine/commit/db7412b4621053a9bc85ae57680b5ac6abf1a3e4

For now, I’ve tested it only on Windows. I’m going to test it on other platforms and will let you know if any issues arise.

By the way, I noticed the following comment in BaseEngine.ini: “Temporarily disable chunking of the PSO cache as the runtime code isn’t yet in place.”

Are there any known issues with this feature enabled that we should be aware of?

[Attachment Removed]

4

Ok, thanks for the callout on the fix and also that comment. From what I know, we already have the code in place, so we should update the wording. I have reached out to some folks to verify this assumption, though.

[Attachment Removed]

5

Hi again, we updated the comment now to reflect that you can use the feature. The change is in UE5 Main 51462008. Once you have done some more testing on your end, could you share your feedback on the PSO cache chunking? We want to gain more insights into how the system works, since we have not had many eyes on it since moving entirely to PSO precaching.

[Attachment Removed]

6

Hello Tim,

I’ve checked this more deeply, and it seems that there are some issues with splitting the PSO cache into chunks.

I did the following:

  1. Cooked a local build without any .spc files in game/Build/Windows/PipelineCaches/.
  2. Played a specific scenario and observed multiple “Encountered a new graphics/compute/raytracing PSOs” logs. I then saved the encountered PSOs using the r.ShaderPipelineCache.Save command.
  3. After that, I expanded the collected PSOs and used the resulting .spc file for the next build with bDoNotChunkPSOCache=False.
  4. Then I played the same scenario with the new build and encountered a problem: I still see some of the same “Encountered a new graphics/compute/raytracing PSOs” logs, even though the same PSO hashes are present in the rec.upipelinecache files saved during the previous iteration. I verified this using the ShaderPipelineCacheTools Dump command. I also ran ShaderPipelineCacheTools Dump for all chunk stable.upipelinecache files and did not find those PSO hashes there.
  5. Then I rebuilt the project again with bDoNotChunkPSOCache=True and with custom code changes that restart the base (Blitz) cache each time any chunk is mounted to fix the original problem described in this ticket. With this modification, the issue did not occur: there were no “Encountered a new graphics/compute/raytracing PSOs” logs with PSO hashes from the initial run. Additionally, running ShaderPipelineCacheTools Dump on the single stable.upipelinecache generated during this iteration showed that all PSO hashes from the rec.upipelinecache files were present.

I’m attaching an archive with all the rec/stable.upipelinecache files, their dumps, and logs from the game sessions.

In the screenshot below you can see an example of a PSO hash that is missing from the chunked stable cache.

As shown, Graphics PSO: 2823639361 was encountered in all three sessions during cache recording, and it is also present in the dumps of the recorded caches. In the run with a single PSO cache, it appears in the logs as an entry in the opened cache and is later precompiled. As a result, there is no “Encountered a new graphics PSO: 2823639361” log in that run.

However, this PSO does not appear in the dumps of any chunked stable caches. As a result, when running with the chunked PSO cache, it is reported again as “Encountered a new graphics PSO: 2823639361”.

[Image Removed]Here is the code changes I made to restart base PSO cache when PSO cache chunks don’t exist

[Image Removed]

[Attachment Removed]

7

Hello Tim,

During my testing I also noticed another issue that I had already reported earlier in another ticket: [Content removed]

Here is the issue I mentioned in that ticket:

It seems that I found the reason why not all the encountered PSOs are saved into the recorded cache. The reason is the opening of another PipelineFileCaches during mounting paks and reopening UserPipelineCacheFile afterwards as a result, which causes the cleanup of all NewPSOs encountered previously (see FPipelineFileCacheManager::CloseUserPipelineFileCache).

This time I again logged the number of NewPSOs (NewPSOs Num = %d) and observed the same behavior: it gets cleared when a new chunk is mounted, so only the PSOs encountered after the latest chunk is mounted are saved to the file. Please find the log attached for your reference.

[Attachment Removed]

8

Hi Alexander, that’s a good catch. I am trying to catch up with all of these findings now. Are you suggesting that the file chunk-loading/unloading logic can interfere with how we build the PSO bundle? That is, when we add newly encountered PSOs and a chunk unloading event occurs, we clear the entire list of gathered PSOs, which causes us to skip PSOs. Does the screenshot of the code you wrote fix this issue? Maybe we can integrate that into our codebase in that case. Let me know if I understood all of that info correctly.

[Attachment Removed]

9

Hi Tim, generally there are two separate issues:

  1. For some reason, chunked PSO caches miss some of the PSO entries comparing to the single PSO cache, but I didn’t investigate this deeply. Instead, I implemented a solution that allows using a non-chunked (single) PSO cache together with DLC chunks. I simply reprocess this single PSO cache (restart the compilation) when new shaders from the chunks become available - that’s what you can see in the screenshot with the code.
  2. Yes, your understanding is correct. When chunks are mounted, the UserPipelineCacheFile is reopened, which causes the NewPSOs list to be cleared, but I haven’t found any workaround for this so far.
    [Attachment Removed]
10

Hi Alexander,

Ok, thanks for the info. I have investigated these issues a bit further. Still, it’s difficult to verify the information because we don’t have a setup that reliably uses chunked PSO bundles for testing. From what I found, your workaround is ok for the time being, though we should find a cleaner solution. Would it be possible to open a pull request for this change so someone on the dev team can review it?

As for the second issue, how much trouble would it be for you to send an isolated repro project? I understand that might not be possible, and in that case, we will look into other avenues for a repro.

[Attachment Removed]

11

Hello Tim,

Here is the PR for the proposed workaround:

https://github.com/EpicGames/UnrealEngine/pull/14540

Regarding the repro project, I’ll check with the team responsible for DLC to see how feasible it is to set up a minimal sample project with DLC chunks.

[Attachment Removed]

12

Hello Tim,

After my changes were merged and widely tested, we found additional issues with the bundled PSO cache:

  1. On Windows there is a random crash when precompiling PSO in ShaderCodeArchive.cpp#L2381 with “I/O Error (Not Found)”. It happens both with my workaround and with bDoNotChunkPSOCache=False, so I assume it’s another PSO Cache + DLC bug that was previously hidden by the bundled cache not being compiled properly.
  2. In my task I also enabled PSO Cache masking for Android with OpenGL and it seems that it doesn’t work when r.PSOPrecaching is disabled. There is a crash in OpenGLProgramBinaryFileCache.cpp#L535 that happens right on app startup when SetGameUsageMaskWithComparison is called because it triggers FShaderPipelineCache::OnCachedOpened for the same cache. My guess is that it could be fixed by an early return when CurrentShaderPipelineProperties.CacheVersionGuid == InVersionGuid, but I’m not sure whether such a situation can occur with two different caches, in which case this fix would not help.

For the first point, could you please share any ideas on how such a crash might happen? As I understand it, the problem is that at the moment of compiling a specific PSO from the bundled cache, a required shader is missing, even though there is a check for missing shaders when PSO precompile tasks are scheduled. This suggests that either the checks at the moment of scheduling or at the moment of compilation are giving incorrect results, or that the shader was available initially but was somehow deleted or moved between those two stages.

For the second point, I can either apply the fix I mentioned or simply disable PSO Cache masking for Android OpenGL when r.PSOPrecaching is disabled. So it’s not that critical, but I still wanted to let you know.

The callstacks:

1.

UE::Logging::Private::BasicFatalLog (StructuredLog.cpp:1104)

FIoRequest::GetResultOrDie (IoDispatcher.cpp:1232)

FIoStoreShaderCodeArchive::CreateShader (ShaderCodeArchive.cpp:2381)

FShaderLibraryInstance::GetOrCreateShader (ShaderCodeLibrary.cpp:1188)

FShaderLibrariesCollection::CreateShader (ShaderCodeLibrary.cpp:3234)

FShaderCodeLibrary::CreateVertexShader (ShaderCodeLibrary.cpp:3944)

FShaderPipelineCacheTask::Precompile (ShaderPipelineCache.cpp:1052)

FShaderPipelineCacheTask::PrecompilePipelineBatch (ShaderPipelineCache.cpp:1515)

FShaderPipelineCacheTask::Tick (ShaderPipelineCache.cpp:1913)

FShaderPipelineCache::Tick (ShaderPipelineCache.cpp:1883)

TickRenderingTickables (RenderingThread.cpp:200)

2.

FOpenGLProgramBinaryCache::OnShaderPipelineCacheOpened (OpenGLProgramBinaryFileCache.cpp:535)

Invoke<T> (Invoke.h:66)

UE::Core::Private::Tuple::TTupleBase<T>::ApplyAfter<T> (Tuple.h:317)

TBaseRawMethodDelegateInstance<T>::ExecuteIfSafe (DelegateInstancesImpl.h:533)

TMulticastDelegateBase<T>::Broadcast<T> (MulticastDelegateBase.h:258)

TMulticastDelegate<T>::Broadcast (DelegateSignatureImpl.inl:1079)

FShaderPipelineCacheTask::BeginPrecompilingPipelineCache (ShaderPipelineCache.cpp:2147)

FShaderPipelineCache::BeginNextPrecompileCacheTaskInternal (ShaderPipelineCache.cpp:1778)

FShaderPipelineCache::BeginNextPrecompileCacheTask (ShaderPipelineCache.cpp:1709)

FShaderPipelineCache::ApplyNewUsageMaskToAllTasks (ShaderPipelineCache.cpp:734)

FShaderPipelineCache::SetGameUsageMaskWithComparison (ShaderPipelineCache.cpp:690)

UPSOCacheHelpers::SetPSOCacheUsageMask (PSOCacheHelpers.cpp:152)

UPSOCacheServiceImpl::UpdatePSOCacheState (PSOCacheServiceImpl.cpp:80)

UPSOCacheServiceImpl::InitDependencies (PSOCacheServiceImpl.cpp:33)

Also, regarding the sample project, we can’t provide it at the moment as we’re currently focused on getting the bundled PSO cache working for the upcoming release.

[Attachment Removed]

13

Hi Alexander,

I don’t know exactly why you are getting that chunk-loading error, but as you said, it might be due to randomness in the pak/container mount ordering. The chunks mount asynchronously, and the ShaderLibraryStateChanged function triggers PSO precompilation immediately for each chunk as it mounts. If a given chunk A’s PSO cache references a shader whose group data lives in another chunk B’s container, and chunk B mounts before chunk A, the IoStore could fail with a “Not Found” error.

With that said, thanks for bringing up the other error to our attention. Unfortunately, without a repro we can run locally, we won’t be able to help you much further than logging a ticket. Once you can share a project, please let us know, and we can certainly revisit your issues.

Cheers,

Tim

[Attachment Removed]

14

Hi Tim,

Considering all these cases, do you think it’s worth continuing to use the Bundled PSO Cache with DLC, or should we drop it and rely entirely on PSO Precaching?

Do you know if other UE projects use it with DLC, for example, Fortnite? Or is it more of a legacy feature that isn’t really supported anymore?

Kind regards

[Attachment Removed]

15

Hi Alexander,

I am still trying to get an answer for you on using bundled PSOs with DLC. I can tell you, though, that we are all-in on PSO Precaching in Fortnite and that we have not used bundling for quite some time. However, the PSO Precaching system has undergone several overhauls since 5.5, so if you are willing to backport those changes into your build, it might be worth investigating. Would that be an acceptable option, or is that an unrealistic goal?

[Attachment Removed]

16

Hi Tim,

Thank you for the information. Yes, we are using both bundled PSOs and Precaching, so if you can share the most important improvements to PSO precaching released after 5.5, it would be really helpful.

[Attachment Removed]

17

Hi Alexander,

I attached a text file containing the most recent changes to PSO precaching since 5.5. Knowing you are on mobile, there are a few changes there that are likely most interesting to you. Please feel free to review those changes and let me know if you have any questions. Have you had a chance to start some pull requests for your proposed fixes to the PSO bundling crashes you have been encountering? I checked in with the dev team, and they would consider taking those changes, since PSO bundling is currently in maintenance mode.

[Attachment Removed]

18

Hello Tim,

Thank you for sharing this, it’s really helpful!

Yes, I created the PR where I restart the base PSO cache in case there is no chunk-specific one: https://github.com/EpicGames/UnrealEngine/pull/14540

However, it’s worth noting that with this approach, the crash I mentioned above (caused by a missing shader) seems to have a higher reproduction rate compared to using chunked PSO, so we ended up with using chunked PSO but only on mobiles where we didn’t observe such crashes in our internal tests, although most likely the reproduction rate is just lower there.

[Attachment Removed]

19

Ok, thanks for letting me know. I will make the code owner aware of that caveat. We could have someone in QA attempt to reproduce the crash on our side. Do you have any more questions about the PSO bundling changes right now? Otherwise, I will look to close out the ticket.

[Attachment Removed]

20

Hi Tim,

Thank you for your support.

Yes, I think we can close this ticket, we don’t have any further questions about bundled PSO on our side.

If any questions arise regarding the list of PSO precaching changes you shared, I will open a new ticket.

Kind regards.

[Attachment Removed]