Hello, I see that inserting the bundled PSO cache into the build eliminates the “Encountered new PSO” events from the CSV trace and logs. However, the PSO Misses metric in the CSV trace remains unchanged (not taking into account random fluctuations).
Could you please clarify whether this is expected behavior? What is the exact meaning of each metric, and which one should we focus on if our goal is to eliminate PSO compilation hitches?
You can find the CSV traces from the runs before and after the PSO cache was added to the build, as well as the corresponding logs.
Another issue we’ve observed is that the bundled PSO cache doesn’t eliminate all “Encountered new PSO” events. In the attached logs, you can see that during the second runs, some events appear with the same hashes as in the first runs. For some reason, these PSOs were either not included in the cache or were not properly precompiled from it.
[Image Removed]
Could you please provide some insights into this behavior? Have you encountered this before? Are there any mechanisms available to debug this in more detail?
Steps to Reproduce
- Run specific in-game scenario with PSO logging and CSV profiling enabled
- Check how many Encountered new PSO events and PSO Misses are present in the CSV trace
- Collect recorded PSOs, convert them, and include them in the next build
- Run the same scenario once again and check the same metrics in the CSV trace
Compare values from the first and second runs. For some reason, in the second run with PSO Cache, the PSO Misses number is still the same, while Encountered new PSO events are gone.
Hello,
I’ve double-checked and can confirm that the same problem exists on Windows (PCD3D_SM6) as well:
[Image Removed]Please find the same set of files as provided for Vulkan and GLES in the attached archive.
Hi Alexander,
PSOmisses refers to cache misses however, the meaning of the metrics can depend on what value you have set for r.PSOPrecaching. Can you confirm your setting?
Best regards
Hello Stephane,
Yes sure, we have r.PSOPrecaching=0
Kind regards
Hi Alexander,
- CSV PSO/PSOMisses and PSO/ComputePSOMisses track how many PSOs (graphics and compute, respectively) were missed because they were not in the runtime cache.
- A “miss” means one of the following:
- The PSO was not in the bundled cache.
- The PSO was in the bundled cache, but it has not been processed yet (i.e. the game did not wait for all PSOs in the bundled cache to be compiled).
- The PSO was in the bundled cache and was processed, but the runtime cache evicted it by the time it was needed. By default, the runtime cache evicts PSOs that have not been used in the last 60 seconds (r.pso.evictiontime). In this case the miss is not nearly as expensive as a miss of a PSO that was not in the cache, because the PSO has been cached by the GPU driver.
- The fact you’re getting “encountered new PSO” on the same hash that was seemingly seen on a previous run during collection is a bit strange. Are you using PSO bundled cache masks (SetGameUsageMaskWithComparison)? That’s a technique to only collect/load from the cache the PSOs that you need for the current scalability
In terms of debugging the fact you seem to be getting the same hash as a new PSO. You can use the
ShaderPipelineCacheTools commandlet to dump the contents of the bundled PSO cache (the .upipelinecache file) and verify the hash is indeed in there. Command below:
UnrealEditor-Cmd.exe NAME_OF_PROJECT -run=ShaderPipelineCacheTools dump bundled_cache_file.upipelinecacheBest regards.
Hi Stephane, thank you for the info - it definitely makes sense. Could you please elaborate further on the difference between these CSV PSO metrics and “Encountered new PSO” events? And what does it mean when we have a “PSO Miss” but no corresponding “Encountered new PSO”?
Regarding the second topic (with getting “Encountered new PSO” on the same hash in the second run): we don’t use SetGameUsageMaskWithComparison on Android due to the [Content removed] with Vulkan, but we do use it on other platforms. The tool you shared looks really useful, I will do my tests once again and check if this hash is in the file or not.
Kind regards.
Hi Alexander,
A PSO miss without a corresponding Encountered new PSO would indicate that the the PSO was in the bundled cache but it not been processed yet or was evicted. Refetching from bundled cache would involve a lesser penalty than the full Encountered new PSO round trip.
Let us know the results of the particular hash causing “Encountered new PSO” for a PSO that should have been in the bundled cache.
Best regards.
Hi Stephane,
For now I’m trying to deal with this difference between the number of “PSOMisses” and “Encountered new PSO”. I tried disabling PSO eviction by setting r.pso.evictiontime to 0, but it didn’t help - I still see only 5 entries of “Encountered new PSO” and 85 “PSOMisses”, so this difference is unlikely to be caused by PSO eviction.
Also, according to the logs, the app enqueues 138 tasks for precompiling and completes all of them in 0.04s (3.89s wall time since initial open), so it doesn’t look like the reason is that these PSOs haven’t been processed yet either.
2025-06-10 16:30:01: [ 0]LogRHI: Opened FPipelineCacheFile: ../../../Blitz/Content/PipelineCaches/Android/Blitz_SF_VULKAN_ES31_ANDROID.stable.upipelinecache (GUID: B9A8DB2541894AB217C858A0FBAB4B18) with 2840 entries. 2025-06-10 16:30:01: [ 0]LogRHI: Display: FPipelineCacheFile[Blitz] opened Blitz, filename Blitz, guid B9A8DB2541894AB217C858A0FBAB4B18. 2025-06-10 16:30:01: [ 0]LogRHI: FShaderPipelineCache::BeginNextPrecompileCacheTask() - Blitz begining compile. 2025-06-10 16:30:01: [ 0]LogVulkanRHI: FVulkanPipelineStateCacheManager::OnShaderPipelineCacheOpened Blitz 138 B9A8DB2541894AB217C858A0FBAB4B18 2025-06-10 16:30:01: [ 0]LogVulkanRHI: FVulkanPipelineStateCacheManager: /storage/emulated/0/Android/data/net.wargaming.medusa.test2/files/VulkanProgramBinaryCache/VulkanPSO_B9A8DB2541894AB217C858A0FBAB4B18.5143.6050002 does not exist. 2025-06-10 16:30:01: [ 0]LogRHI: Display: FShaderPipelineCache starting pipeline cache 'Blitz' and enqueued 138 tasks for precompile. (cache contains 2840, 2840 eligible, 2702 had missing shaders. 0 already compiled). BatchSize 5 and BatchTime 16.000000. ... 2025-06-10 16:30:05: [112]LogRHI: Warning: FShaderPipelineCache Blitz completed 138 tasks in 0.04s (3.89s wall time since intial open). 2025-06-10 16:30:05: [112]LogVulkanRHI: FVulkanPipelineStateCacheManager::OnShaderPipelineCachePrecompilationComplete 2025-06-10 16:30:05: [112]LogVulkanRHI: Display: FVulkanPipelineStateCacheManager: Saved device pipeline cache file '/storage/emulated/0/Android/data/net.wargaming.medusa.test2/files/VulkanProgramBinaryCache/VulkanPSO_B9A8DB2541894AB217C858A0FBAB4B18.5143.6050002', 957715 bytes 2025-06-10 16:30:05: [112]LogRHI: FShaderPipelineCache::BeginNextPrecompileCacheTask() - Finished, no jobs remaining.The only concern I have right now is about the rest of the cache (cache contains 2840, 2840 eligible, 2702 had missing shaders, 0 already compiled). Could these extra PSOMisses be related to the cache entries with missing shaders? What do these entries with missing shaders actually stand for?
Do you have any other ideas on how to find the reason and get rid of this difference between “PSOMisses” and “Encountered new PSO” events?
P.S. I’m attaching the full log of the run with r.pso.evictiontime 0
Hi Alexander,
For the missing shaders, the bundled cache logic checks that the shaders are still available in the cooked data (
bHasShaders &= FShaderCodeLibrary::ContainsShaderCode(Hash);
). Can you confirm PSO’s are collected against AND using the cache on the exact same version, without shader or cooked content changes in between?
Best regards.
Hello Stephane,
To be 100% sure I ran the new tests:
- Deleted all the .spc files from the build and launched the test
- Collected the .rec.upipelinecache file from this run
- Expanded it into a new .spc file and put it into the build (not changing anything else other than that)
- Launched the second run with the same scenario
And with that I’m still seeing the main issue: the number of PSOMisses in CSV profiler hasn’t reduced in the second run compared to the first run, even though at the same time the number of “Encountered a new PSO” has reduced dramatically.
Are there any insights you can share regarding what should be my steps to understand why it’s so?
Regarding the second topic - the same “Encountered a new PSO” entries in the second run, I was able to get a dump of .rec.upipelinecache and I see that not all the entries from the log are in the dump, I’m continuing investigating this, and I would appreciate any insights about this as well.
You can find all the files from both runs in the attached zip, and feel free to ask for any additional data you might need.
Kind regards,
Alexander
Hello Stephane,
It seems that I found the reason why not all the encountered PSOs are saved into the recorded cache. The reason is the openning of another PipelineFileCaches during mounting paks (see the logs below) and reopening UserPipelineCacheFile afterwards as a result, which causes the cleanup of all NewPSOs encountered previously (see FPipelineFileCacheManager::CloseUserPipelineFileCache).
Could you please clarify if it’s expected behaviour? And how to deal with it?
2025-06-20 09:51:45: [164]LogIoDispatcher: Display: Mounting container '/data/user/0/net.wargaming.medusa.test2/files/Dlc/PakCache/pakchunk10-Android_ASTC_0_P.utoc' in location slot 0 2025-06-20 09:51:45: [164]LogPakFile: Display: Mounted IoStore container "/data/user/0/net.wargaming.medusa.test2/files/Dlc/PakCache/pakchunk10-Android_ASTC_0_P.utoc" 2025-06-20 09:51:45: [164]LogDlcCdn: CdnDlcDownloader: successfully downloaded pak file pakchunk10-Android_ASTC_0_P.utoc. 2025-06-20 09:51:45: [165]LogDlcCdn: CdnDlcDownloader: starting download of pak file pakchunk103-Android_ASTC.pak from https://lt1-ms1-s3.wargaming.net/uedlc/15166808405_5144744/Android_ASTC/pakchunk103-Android_ASTC.pak. 2025-06-20 09:51:45: [165]LogShaderLibrary: Display: Using IoDispatcher for shader code library Blitz_Chunk10. Total 53538 unique shaders. 2025-06-20 09:51:45: [165]LogAnalyticsCollector: encountered struct in event which is not data group.is new data group declared and not collected? event=AnalyticsEventsProtoData_Analytics_DlcPakTracking_v1 field=Header type=AnalyticsEventsProtoData_Analytics_DlcPakTrackingHeader 2025-06-20 09:51:45: [165]LogAnalyticsCollector: encountered struct in event which is not data group.is new data group declared and not collected? event=AnalyticsEventsProtoData_Analytics_DlcPakTracking_v1 field=Details type=AnalyticsEventsProtoData_Analytics_DlcPakTrackingDetails 2025-06-20 09:51:45: [165]LogShaderLibrary: Display: Cooked Context: Using Shared Shader Library Blitz_Chunk10 2025-06-20 09:51:45: [165]LogStats: StoreEventImpl - 0.110 ms 2025-06-20 09:51:45: [165]LogRHI: Could not open FPipelineCacheFile: ../../../Blitz/Content/PipelineCaches/Android/Blitz_Chunk10_SF_VULKAN_ES31_ANDROID.stable.upipelinecache 2025-06-20 09:51:45: [165]LogRHI: Display: CloseUserPipelineFileCache (Opening: Blitz), NewPSOs.Num was 13
Hi Alexander,
Can you confirm if you are still seeing this last issue in 5.5?
Best regards.
Hi Stephane,
Right now we are still in the process of moving to 5.5, so I can’t check it right now, once we finish the transition, I will check it and share the results
Thank you Alexander, I’ll await your update.
Best regards.