UE 5.7.3 D3D12 SM6: PSO cache pipeline, compute PSO bundling, and late PSO hitches on PC

Development Rendering
, ,
1

Hi

We have a question regarding late PSO compilation on PC. During gameplay, we see 200-300 ms hitches related to “PSO too late” compilation. This is currently the last meaningful blocker before we lock down the PC version

We would like to check if our pipeline and configuration are correct for UE 5.7.3, D3D12 SM6.

Current CVar configuration

Config/DefaultEngine.ini

r.PSOPrecaching=1

r.PSOPrecache.Validation=2

r.PSOPrecache.GlobalShaders=1

r.PSOPrecache.ProxyCreationWhenPSOReady=1

r.ShaderPipelineCache.Enabled=1

r.ShaderPipelineCache.ExcludePrecachePSO=1

r.ShaderPipelineCache.StartupMode=0

r.ShaderPipelineCache.SaveUserCache=0

Platforms/Windows/Config/WindowsEngine.ini

Values:

r.ShaderPipelineCacheTools.IncludeComputePSODuringCook=0

r.ShaderPipelineCache.GameFileMaskEnabled=0

r.ShaderPipelineCache.Enabled=1

r.ShaderPipelineCache.StartupMode=1

r.ShaderPipelineCache.PrecompileBatchTime=0

r.ShaderPipelineCache.BatchSize=50

r.ShaderPipelineCache.BatchTime=16

r.ShaderPipelineCache.BackgroundBatchSize=32

r.ShaderPipelineCache.BackgroundBatchTime=8

Config/DefaultGame.ini

Packaging:

bUseIoStore=True

bShareMaterialShaderCode=True

bSharedMaterialNativeLibraries=True

bGenerateChunks=False

bChunkHardReferencesOnly=False

We are not using usage masks. The intent is to have one bundle that covers the whole game.

Questions

  1. Based on the pipeline and CVar configuration above, is anything missing or misconfigured for the recommended UE 5.7 setup on PC with D3D12 SM6?
  2. Is this the correct way to bundle compute PSOs on PC?

[Attachment Removed]

2

Steps to Reproduce
Collection step

We collect PSOs on QA workstations with this command:

PlayGround.exe -clearPSODriverCache -logPSO -log -newconsole

Each QA session produces one .rec.upipelinecache file here:

<Project>/Saved/CollectedPSOs/

We collect files from multiple testers and merge them.

After each cook, we also back up the stable shader info files:

<Project>/Saved/Cooked/Windows/PlayGround/Metadata/PipelineCaches/ShaderStableInfo-Global-PCD3D_SM6.shk

<Project>/Saved/Cooked/Windows/PlayGround/Metadata/PipelineCaches/ShaderStableInfo-PlayGround-PCD3D_SM6.shk

Expand step

We produce the .spc file with this command:

[Image Removed]

We commit this file to source control:

<Project>\Build\Windows\PipelineCaches\PlayGround_PCD3D_SM6.spc

Cook, stage, and archive step

On the build machine we run:

[Image Removed]

During cook, UE automatically invokes ShaderPipelineCacheTools build with the .spc and .shk files and produces the final .stable.upipelinecache, which is packaged into the IoStore containers.

Validation run

For validation we run:

PlayGround.exe -clearPSODriverCache -logPSO -log -newconsole

On the loading screen, we wait until:

FShaderPipelineCache::NumPrecompilesRemaining() == 0

Only after that do we show the world.

[Attachment Removed]

3

Hi,

With r.ShaderPipelineCache.ExcludePrecachePSO=1 the PSO bundle will only contain PSOs which are missing from the PSOPrecaching - which is probably fine. But this also means that PSOs which are TooLate won’t be in the bundle. I think I changed this in 5.8 as well to also add TooLate PSOs with CL: 51094323 (you don’t need the whole change for this but mostly the logic related to if (PSOPrecacheResult == EPSOPrecacheResult::Complete) in PipelineFileCache (but precache state needs to be passed along or you can just change the meaning of bWasPSOPrecached on the caller as well to make the change smaller. This CL also has another good improvement for bundles and PSO precaching working together by pushing the bundles PSOs in the PSO precache thread pool to have less contention between the 2 systems.

52204694 - does the reverse and adds the option to add all PSOs in the bundle which will make the bundle a lot bigger but might be fine if you have an upfront screen showing PSO compilation.

Do you have a log file with r.PSOPrecache.Validation=2 enabled and also perhaps an insight trace to better understand where the TooLate PSOs are coming from, because this might be fixeable in another way as well potentially but then I would need to know the type of PSO and where it comes from.

Cheers,

Kenzo

[Attachment Removed]

4

Hi,

I received some traces and this is what I can see in them:

  • Minor hitches coming from precached state, meaning it takes a long time to load something from the driver - keeping PSOs in memory is sadly enough the only real workaround for this and for 5.8 we have some improved solutions out of the box for this
  • Hitch in Frame 569: mostly because of delete RHI resources, but also a miss of 37 msec for WoundDeformer_OptimusNode_CustomComputeKernel - hard to say for what this compute kernal is used and why it’s missed. Is this a global shader or some niagara effect?
  • Really big hitches at frame 967: these are all coming from TooLate compute shaders of nanite which is not expected at all. We did make some fixes in 5.8 for correctly picking up the fallback bins for nanite which can perhaps be related. From the log file I only see a single TooLate for nanite coming from M_Ten_Trim_Wood_2_1_BODisolve and then a handful of too lates for Slate which I assume happen during loading or the first frame. The log does contain a quite a few misses on nanite raster so I assume these are the problems: can you check if cherry picking the NaniteRaster PSO precaching fixes solves the problem?
  • Also see a few misses on CableComponent - this is not something we test very often in house so if you have a sample with those assets then it should be pretty easy to fix those up

Or should I be checking for something else?

Kind regards,

Kenzo

[Attachment Removed]

5

Hi,

sorry for the late reply. 48715278 in UE5 main is the main one for correct handling of fallback materials - hope this one helps.

Use the refcounted PSO tracking requires quite a few follow up changes and might not be all that easy to integrate, but I can provide the list of all these CLs if you are interested. In the meantime using r.PSOPrecache.KeepInMemoryUntilUsed=2 and tweaking r.PSOPrecache.KeepInMemoryGraphicsMaxNum & r.PSOPrecache.KeepInMemoryComputeMaxNum should help with hitches coming from loading PSOs from the driver cache. Do you have these enabled already?

D3D12.PSOPrecache.KeepLowLevel & D3D12.PSO.KeepUsedPSOsInLowLevelCache can be used for debugging as well to verify that the driver related hitches are resolved from reading already compiled PSOs from the driver cache. This does require quite a bit of extra memory so we don’t suggest shipping with these unless you have a high memory min spec.

The deformer graph compute kernels clearly are not precached yet and we have a local change which should resolve that. I will try and get this in soon but it will only be available in 6.0 stream. You should be able to cherry pick it from there as well then.

Regarding the cable component - if you have provide empty project with the asset & material used which gives the miss in cooked project that would be really helpful to track down this miss.

Kind regards,

Kenzo

[Attachment Removed]

6

Hi Igor,

Kenzo is currently out of the office, but I have been looking into this issue in the meantime. Just to confirm, you are also hitting that assert at PipelineFileCache.cpp line 1424?

[Attachment Removed]

7

By the way, would it be possible for you to share the SPC files for 5.7 and 5.8 with me? It would be good to do some comparisons there

[Attachment Removed]

8

[Attachment Removed]

9

Thanks, Igor! Good eye on the file size differences, since that is actually a symptom of what is causing the cook assertions. For 5.8, we changed the shader hashing algorithm from SHA-1 to XXH64, which requires fewer bytes, making our packaged builds more compact. The problem is that we didn’t account for that change when reading the byte stream from the SPC file, but we have a fix in place now and are looking to release it very soon. I will keep you in the loop on when that lands

[Attachment Removed]

10

Hello again, I pushed the patch for this issue to UE6-Main in CL 55874754 yesterday. I am also going to work on integrating the change into the 5.8.2 hot fix, but let me know if you cherry-pick it early and run into any issues.

[Attachment Removed]

11

Great, that’s good to hear. I will close out this ticket then. If you find any issues, please raise a new one.

[Attachment Removed]

12

Hi, Kenzo,

Could you please point us to the 5.8 changelists for the following things (if at all possible):

>it takes a long time to load something from the driver - keeping PSOs in memory is sadly enough the only real workaround for this and for 5.8 we have some improved solutions out of the box

>The log does contain a quite a few misses on nanite raster so I assume these are the problems: can you check if cherry picking the NaniteRaster PSO precaching fixes solves the problem?

Meanwhile i’ve dug more info on WoundDeformer_OptimusNode_CustomComputeKernel & CableComponent:

Cable component - is a generic UE5 component/actor from native UE5 plugin, this one /Script/CableComponent.CableActor . Do you need an empty project with this cable setup with our material to look it up? As far as i can see here we do not change anything on it out of the box.

WoundDeformer_OptimusNode_CustomComputeKernel - this is a Deformer Graph which is used on Main Character and most enemies. It recieves damage data, like position or size, and draws a wounds mask into skeletal mesh vertex color. There is only one asset we use for this for the whole game, but it is used in many character BPs.

Thank you for helping us with this matter!

[Image Removed]

[Attachment Removed]

13

Hi Kenzo,

We upgraded our project to UE 5.8 because we expected several PSO‑related fixes in the new version. However, after migrating, we can’t cook the build at all — we’re running into an issue very similar to the one described in this thread:

[Content removed]

After moving to 5.8, we removed all 5.7‑specific resources (.upipelinecache, .shk, and .spc files), cooked a fresh build, and collected a new .upipelinecache from it. Then we generated a new .spc file using the .shk from the updated build (without any 5.7 leftovers), placed the .spc file into the Project directory — and the cooking process started failing immediately afterward.

Could you help us understand what might be causing this and how we can move forward?

Best regards,

Igor

[Attachment Removed]

14

Hi Tim

​

Yep, exactly at line 1424 in checkNoEntry(); I attached a zip file with the SPC files and noticed that the new one is smaller than the 5.7 version.

[Attachment Removed]

15

We’ve built the project with your CL — with it, the build finally compiled. In our initial quick tests, PSO performs significantly better than in 5.7, and the early results look promising. We’re now starting deep testing, and I’ll get back to you if any issues arise.

[Attachment Removed]