Epic are looking into this with NVidia. Disabling Texture Streaming delays the issue but does not fix it - it’ll still happen eventually.
The issue does not occur in D3D11 it appears, it’s exclusive to D3D12. If you’re using 5.1 and require Nanite/VSM, you’ll likely have to wait it out.