Nanite and Raytracing using increasing amounts of VRAM in 5.6.1 after upgrading from 5.4.4

We have a project that we upgraded from 5.4.4 to 5.6.1

It ran at 60fps originally and in the upgrade to 5.6 we got a lot more quality from lumen and want to use megalights. the project still runs at 60fps but over time more and more vram is used as you fly around the scene. eventually it exhausts all memory and the frame rate tanks to 10fps or less.

Disabling nanite solves it. Disabling raytracing solves it. They cant be enabled together.

Toggling virtual shadow maps in console command resets memory usage, so you can toggle it off and memory usage resets and then begins to climb again. you can toggle it back on and memory usage resets and begins to climb again as you explore.

Turning all levels off in the level outliner resets memory usage as well in the same manner as above.

Deleting all lights doesn’t stop the problem from happening.

Looking at the Render Resource Viewer and comparing it to 5.4.4, two entries are very different.

NaniteRayTracing.IndexBuffer and NaniteRayTracing.VertexBuffer 256mb and 192mb in size. in 5.4.4 there are just 2 of these in the viewer. in 5.6.1 there is about 5 at the start and then they multiply into 100+ and eat up as much memory as possible.

Any ideas what is happening here?

1 Like

It’s related to r.raytracing.nanite.mode 1

When this is enabled those buffers mentioned above keep appearing. If you disable this, nanite shadows look awful but the memory issues goes away.

In 5.4.4 this came at a smaller cost of 448mb (the 256 + 192 for each buffer) in 5.6.1 the cost rises to about 20,000mb

2 Likes

Using a fallback error of 0 on all meshes costs less than using r.raytracing.nanite.mode 1

I don’t think that is the intended way for this to work? Documentation says the CVar should be much more performant compared to fallback error 0

Hi, facing the exact same issue here. r.raytracing.nanite.mode 0 makes the game look awful. Is there a better solution? Setting fallback error to 0 looks pretty bad too, especially for my use case.

I don’t think using fallback error 0 should be looking worse than nanite.mode 1. You may be having a different issue?

using fallback error 0 looks the same as nanite.mode 1 for me. but the cost in VRAM between the two methods is drastically different

I have a lot of nanite foliage in my world, and I need good quality shadows for that as it’s for Archviz. The solution I went with was writing a small plugin that monitors the VRAM usage and calls r.Shadows.Virtual.Enabled 1 every time the VRAM usage hits 90%, and that flushes the VRAM and makes the game run smoothly again.
The first time the game starts, the VRAM climb is super fast, but after the first “flush” the climb is pretty slow. I’d rather have a few small 1 second glitches in my software once in a while than just horrible visuals, which works out in my very rare use case.

Hopefully they fix this in a hotfix or 5.7

I still find this issue present in 5.7 and have not figured out why it uses so much more vram than 5.4

Yep, still an issue. My “hack” to flush VRAM every time it hits 90% is not ideal especially on 16GB VRAM cards as it happens way too often. It really sucks as I cant use HW ray tracing and have to resort to Lumen which makes my projects look not so great, despite the hardware easily being able to run it.

I hope someone at epic can fix this or at least tell us if we are doing something wrong.

I did speak to someone at EPIC who said they could not reproduce it their end but it was indeed present in my project(ue 5.6). However i have since installed 5.7 completely fresh and started a fresh project and still it happens. So i’m not sure what variables are changing my end to cause it to happen.

but so long as r.raytracing.nanite.mode 1 is active, the issue is present.

Hi! This issue has been fixed in UE5/Main (https://github.com/EpicGames/UnrealEngine/commit/f999f20e07893ef4840737e5916215ec9a722ca3) and will be included in an upcoming engine release.

oh amazing. I’ve not edited from git before, can i copy paste this file for a fix now or do i just need to wait for the update to avoid having to mess with the engine?

hoped it was going to be that 5.7.3 update but looks like it wasnt :frowning: currently getting better performance without nanite. less VRAM and for whatever reason better reflections and better fps.

Yup, i looked at the code and the fix is not yet merged into the release branch. Looks like just one line has changed in one file, so i’ve made that change in the release branch and I’m trying to build the engine from source to see if it fixes the issue.