PSO Compilation and Out of Memory Crashes

Development Platform & Builds
, ,
1

Our error reporting backend has caught some errors over time and we are suspecting it is due to the PSO compilation on devices causing it. Most of the background processing is stuck there and the specs of the machine indicate the core to memory ratio is bad for some of these crashes assuming on thread could go up to 2 GB.

We have adjusted the number of threads used for it to limit the overall memory consumption and that seems to have reduced similar reports for now. But that ends up increasing loadtimes if we wait on PSOs or just pop in issues if we don’t . Is there a better way to cap the memory used and avoid over utilization of these threads that endup causing OOM crashes.

2

Hello,

Thank you for reaching out.

I’ve been assigned this issue, and we will be looking into the memory usage of PSO Precaching for you.

3

Hello,

Can you please answer a few questions for us?

  1. Are you able to reproduce this crash yourself, or are you seeing it happen only on player machines outside of development?
  2. Are you experiencing this issue on NVidia hardware? If so, have you tried disabling the CVar “r.PSOPrecache.KeepInMemoryUntilUsed”?
4

We see it only happening on player machines and reported in our error backend. Our live game is using 5.4 so we don’t really have that commandline there to check.

5

Just noticed these console commands in 5.6 . Would these help in addressing our issue.

r.PSOPrecache.KeepInMemoryGraphicsMaxNum

and

r.PSOPrecache.KeepInMemoryComputeMaxNum

6

Hello,

“r.PSOPrecache.KeepInMemoryGraphicsMaxNum” and “r.PSOPrecache.KeepInMemoryComputeMaxNum” are related to the CVar “r.PSOPrecache.KeepInMemoryUntilUsed”, and are only used on NVIDIA hardware.

These CVars were introduced by CL: 35904165 and was locked to NVIDIA hardware in CL: 36240864 on //UE5/Main. If you do not have these CLs, then adding them will be unlikely to help.

Are there any common hardware / driver configurations among the crash reports?

The following CLs from //UE5/Main might help you: 43263054, 41960822, 42109499. They all have to do with optimizing PSO counts, retention and memory.

Please let us know if this helps.