After upgrading our project from 5.4 to 5.5 we’ve started experiencing a somewhat frequent GPU hang (and then crash). We were able to reproduce the hang with rhivalidation enabled. I’ve attach the full log for reference.
The log contains a RHI Resource Transition Validation Errorlike this…
Attempted to access resource “VirtualTexture_PageTable” (0x000001AA41D43760) (Mip 0, Slice 0, Plane 0) from a hardware unit it is not currently accessible from. A resource transition is required.
…for each of the 12 mip levels associated with VirtualTexture_PageTable and then two similar validation errors for VirtualTexture_Physical with different addresses.
The log also contains GPU breadcrumb and Aftermath data.
The most similar we could find in the support forum is this one, though it’s not similar enough to assume it’s the same root cause: [VirtualTexture_PageTableAdaptiveIndirection RHI [Content removed]
Unfortunately, we’re at a bit of a loss as to how to further address this issue. Any idea what’s happening here or the best way to gather additional information?
Hi there,
Since this case’s priority was requested to be elevated, I wanted to get a preliminary response back on my investigation.
It appears that there have been fixes upstream for RHI validation errors in the virtual texture system. I am currently investigating the following commit:
https://github.com/epicgames/UnrealEngine/commit/785b6d242cafc0d79209ba4dd26102f3c3224a8b
To see if it fixes the RHI validation errors that I can replicate.
This commit is unfortunately quite a big refactor, and appears like it might also be dependent on several previous commits. So it might be difficult to backport this to 5.5.
I was not able to replicate your exact validation error. However, I can replicate similar ones by building a streaming virtual texture for an RVT. This triggers similar validation errors for me. Though the layout transitions are slightly different.
If you can create a minimal reproduction for your exact validation error, that might also be helpful.
Regards,
Lance Chaney
Another update, I’m still looking for the commit that fixes my reproducible validation errors. The previous commit I was investigating didn’t seem to be the one that fixed these. Unfortunately, there are quite a few potentially relevant commits, and the one that fixes my validation error, might not fix yours. So trying to figure out how we can reproduce your exact validation errors might be necessary, in order for me to be able to track down a fix for you.
Alternatively, you could try upgrading all the way to 5.6, if you’re willing to invest more time into doing this. I can confirm that I get no RHI validation errors on this engine version.
You may also want to get an Nvidia aftermath dump with full source code debugging. This way we can make sure that this RHI validation error is actually the cause of your GPU crash issues, and we aren’t just going down a massive rabbit hole.
To do this, set the following CVars in you <EnginePath>/Engine/Config/ConsoleVariables.ini:
r.GPUCrashDebugging.Aftermath.DumpShaderDebugInfo=1
r.Shaders.Optimize=0
r.Shaders.Symbols=1
r.Shaders.ExtraData=1
r.Shaders.SkipCompression=1
When you get a crash with these settings, an *.nv-gpudmp will be created at <ProjectDir>/Saved/Crashes/UECC-Windows-###/D3D12.###.nv-gpudmp
You can open this in Nvidia Nsight Graphics, available here: https://developer.nvidia.com/nsight-graphics
After opening the crash dump, set your shader search paths, in tools->options, as follows:
[Image Removed]You can then check the crash info tab to see exactly where you are getting the crash in shader code. Use the yellow arrow to jump to the next instruction that would have executed after the crash:
[Image Removed]
Regards,
Lance Chaney
Thank you for your efforts and the information you’ve provided, Lance. While we discuss the possibility of upgrading to 5.6, I will attempt to get an Aftermath dump with full source debugging as you’ve described so we can further narrow down the issue.
Cheers,
Jesse
These were the most relevant commits I found:
60ad6b8d7f263bb3671b7decf259cb994074b259 28/4/2025 <- Deprecate use of low level RHI Texture target when requesting virtual texture production. Virtual texture producers should work with the pooled render target to ensure that texture transition logic can stay robust.
4ffbb7daed740959b2d4d319dc1821f889d8ef95 28/4/2025 <- Mark page table textures as ForceImmediateFirstBarrier in the RDG graph so that render target view transitions for aren’t hoisted into RenderFinalize() or earlier where they will be incorrect for virtual texture sampling.
4624ab8417a4689e111b319ff56441167797056e 28/4/2025 <- Virtual texture upload cache now uses RDG to manage upload. … Change also removes the complexity from the physical pool texture transition logic.
D9342a45ebc17d759f37a6d94f3a01d65df40334 28/4/2025 <- Remove direct access to VT physical texture RHI. Should use PooledRenderTarget or RHI SRV only. This is part of ensuring correct transition usage.
Fe5eb741ce175ae719204663ccdfea5c22daac21 28/4/2025 <- Runtime virtual texture always uses PooledRenderTarget and no longer uses the low level RHI Texture. This simplifies transition handling.
f09cebf2b24cd05fb76ff3d7ff1743fe233805f0 28/4/2025 <- VirtualTextureAdapter now uses PooledRenderTarget instead of RHI Texture as target. Makes transition logic simpler.
fd1aa0630225a23bb07a4e8555177b961cab6726 28/4/2025 <- Fix incorrect transition for virtual texture feedback debug.
785b6d242cafc0d79209ba4dd26102f3c3224a8b 24/4/2025 <- Split virtual texture Finalize into two phases. … This seperation allows us to make transitions to page table and physical textures at the correct times.
There is also the adaptive virtual texture rhi validation fix that you mentioned here: 21aa53bf86f95e5b435278795e383dec88dcd7b9
However, this fix would only be relevant if you have adaptive page table enabled in your virtual texture project settings. From your validation error, it looks like you are using the standard page table.
Unfortunately, my reproduction case is actually fixed before all of these. I have been unable to track down the commit that fixes it so far. I think at this stage my reproduction isn’t close enough to your actual error that I should continue chasing this. I would need a better reproduction in order to proceed further. Could you try replicating your RVT setup in a blank project, to see if you can replicate the validation errors?
Regards,
Lance Chaney