Possible race condition in PSO precache collection tasks

Development Rendering
, ,
1

Hello

During the playtests, we have encountered random crashes related to PSO cache collection. Those crashes occur on PCs with wide variety of hardware, however low spec ones have higher chances of reproduction.

I tried manually reproducing this crash on few low-spec PCs similar to the ones with high occurrence but it was unsuccessful.

Due to the nature of these playtests I was unable to gather any additional information, certain steps or conditions leading to the reproduction. The only thing that is certain - it randomly happens during level streaming. I have assumption that the Material or it’s ShaderMap becoming invalid right during the PSO collection task.

Would appreciate if someone has any additional information regarding this or similar issues, possible reproduction or fix suggestions.

2

Hi there,

thanks for including the callstack. To help debug this crash further, can you please try the following (please note that none of these are fixes and are just meant for debugging purposes):

  • Clear the driver cache before every playtest using the “-clearPSODriverCache” command-line argument. This will delete all the cached PSOs, which will result in longer load times as the PSOs will need to be rebuilt from scratch on every run. This may also result in more hitches during gameplay and potentially trigger the crashes more frequently.
  • Can you try setting r.PSOPrecaching to 0. This will disable PSO precaching to confirm if the crashes are PSO precaching related. This should only be disabled for debugging however (by default this should be enabled).
  • With r.PSOPrecaching enabled again, please try setting the r.PSOPrecache.ProxyCreationWhenPSOReady to 0 and check if this changes the frequency of the crashes
  • Please try setting r.PSOPrecache.ProxyCreationWhenPSOReady to 1 and r.PSOPrecache.ProxyCreationDelayStrategy to 1. This will render meshes which PSO is not ready with a default material. This may also result in less crashes=
  • You may also bring up the PSO stats by setting r.PSOPrecache.Validation to 2 and r.PSOPrecache.Validation.TrackMinimalPSOs to 1, then use the command “stat psoprecache”

You can find more information on these CVars in this link.

Please let me know how that goes.

Thanks,

Sam