Render thread stuck waiting in FVirtualTextureProducer::Release() and eventually causes game thread timeout after 120 seconds.

Development Platform & Builds
, ,
1

Hi there,

We’ve been seeing few instances recently that looks like a potential deadlock, where the render thread would stuck waiting for ProducePageTasks in FVirtualTextureProducer::Release() at the following line:

...	
FGraphEventArray ProducePageTasks;
VirtualTexture->GatherProducePageDataTasks(HandleToSelf, ProducePageTasks);
if (ProducePageTasks.Num())
{
   TRACE_CPUPROFILER_EVENT_SCOPE(FVirtualTextureProducer::Release_Wait);
   FTaskGraphInterface::Get().WaitUntilTasksComplete(ProducePageTasks, ENamedThreads::GetRenderThread_Local());
}
...

It would eventually lead to a game thread timeout after 120 seconds. The task dependency chain looks like I/o Read -> FCreateCodecTask -> FTranscodeTask. From inspecting the state of ProducePageTasks in a minidump, there are two tasks in ProducePageTasks, and both them have NumLock equal to 1 and their Prerequisites arrays are empty. I think NumLock being 1 means there are still one pending prerequisite and the Prerequisites array might be cleared due to the call to TryRetractAndExecute? In the meantime, all other worker threads are parked waiting for work. No tasks were getting processed anywhere. It feels like we’re hitting a deadlock in the task graph system where there are outstanding work but none of the worker are awaken to pickup the work?

For context, we are on 5.6.1, and I used an AI agent to help search for post 5.6.1 commits that seems to address this class of issues on GitHub’s release branch. It suggested this particular commit: https://github.com/EpicGames/UnrealEngine/commit/b27a59e67c2e. I’m going to cherry pick it and observe if it would makes these timeouts go away, and figured while I do that, I can open a thread here to gather some ideas. Does this issue look familiar and do you have any suggestions regarding next steps to further narrow it down? Thanks in advance for your time and support!

Best,

Min

[Attachment Removed]

2

Hello!

Can you share the Parallel Stacks when the deadload occurs? You can save a PNG of the UI using the disk icon that is the last icon on the top bar.

Martin

[Attachment Removed]

3

Here is the parallel stacks exported from VS. Also attached it just in case the embedded image is hard to see. Let me know if you need anything else. Thank you!

[Image Removed]

[Attachment Removed]

4

I searched for changes related to the release of VTs and came up with this list:

  • 45319541 - Changing more uses of RHICreateTexture to RHICmdList.CreateTexture.
  • 46261481 - Fix virtual texture upload issue where we could overrun the number of staging buffers addressable by the tile handle. When it happened it would corrupt the staging buffer free tile array and lead to memory exceptions when copying the tile data. Changed allocation of bits in tile handle to avoid this happening. Added check to catch unlikely occurrence in dev builds.
  • 46273235 - Fix 64-32bit downcast when reporting virtual texture upload staging memory.
  • 50360260 - Fix issue in virtual texture upload cache where tiles in PendingRelease were not being released quickly due to incorrect early out from the iteration loop. Also added some more virtual texture stats to show reasons why updates are throttled. Updated default value for r.VT.MaxUploadRequests to be more in line with r.VT.MaxUploadMemory.
  • 50447973 - Virtual Texturing: r.VT.MaxUploadMemory and r.VT.MaxUploadRequests can now be set to 0. A value of 0 disables their upload throttling. r.VT.MaxUploadRequests defaults to 0 so that by default we only throttle VT uploads by memory budget.
  • 51224527 - Fix virtual texture ensure for RDG external access mode. Could trigger if VT update/finalize happened more than once in the same RDG graph.
  • 51554338 - Removed legacy streaming virtual texture upload path.
  • 51735400 - Split PooledRenderTarget and SceneRenderingAllocator off into separate header and add explicit includes for RealtimeGPUProfiler to files with an implicit dependency.

Out of those, 50360260 seems to be the most promising in your case.

Martin

[Attachment Removed]

5

Thanks for sharing those changelists. Going to give 50360260 a try.

[Attachment Removed]