Hey folks!
Since upgrading to UE 5.7, we have been experiencing some issues with landscape painting. There is often a very noticeable delay between painting and the change becoming visible on the landscape (see attached video), which makes it difficult to paint accurately.
Are there any settings we can tweak to eliminate this delay?
I already played around with some of the virtual texture related console variables which looked like they might be connected to this (r.VT.MaxContinuousUpdatesPerFrameInEditor, r.VT.MaxUploadsPerFrameInEditor, r.VT.MaxTilesProducedPerFrame), but none of them seemed to solve the issue.
Any advice would be appreciated!
Cheers,
Dave
Video Showcasing the Issue.mp4(44.4 MB)
Hi David,
It’s possible that the issue is with RVT, indeed, but first off, can you activate landscape debug mode and check if this is at least reactive when you paint a given layer? If not, the problem might be with the landscape tools. I wouldn’t think that’s the case, but better make sure.
Could you also send us a video of the bug with the CVar r.VT.Residency.Show 1 set? That should show the amount of pages being generated by virtual texturing.
I also wonder if anti-aliasing could come into play here. Can you disable Temporal AA (show flags menu -> Advanced sub-menu) and make sure that no kind of DLSS or DLAA plugin is used?
If your project supports D3D11 (-d3d11 commandline argument) and/or vulkan (-vulkan commandline argument), does the behavior reproduce with it?
One other useful thing to try, since it takes a while for the render to catch up would be to try to take a GPU dump (DumpGPU in the console) when the catchup occurs. Or even capture it with render doc : (start the editor with -attachrenderdoc and then press Alt-F12 to trigger a capture, which should leave a .rdc file in the Saved/RenderDocCaptures folder of your project) and send it to us. That should help diagnose the issue.
Thanks,
Jonathan
Thanks for the additional details. It definitely points to a RVT issue, then. Out of curiosity, how many Runtime Virtual Texture components do you have in your world?
And does boosting r.VT.MaxContinuousUpdatesPerFrame, r.VT.MaxUploadsPerFrame, r.VT.MaxTilesProducedPerFrame to an insane value like 99999 fix anything in the standalone cooked game?
Ok thanks, that makes a bit more sense that setting r.VT.MaxUploadsPerFrameInEditor to 99999 would have an effect in the editor, it was the one thing that seemed impossible from your initial description.
We can’t think of any change that could have led to such a regression in UE5.7 so I think that our best bet is to investigate the regression range. What version of UE were you using before this started happening? Is there a way for you to go back in time and try to see when this started occurring?
One thing that could help with the investigation would be to visualize the area of RVT that gets invalidated when you paint (or at runtime, athough I am not sure whether the visual logger is available there). If it’s too large, then it’s possible that too many pages get invalidated and that ends up clogging the RVT update. In order to do this, you need to start the visual logger (one way to do this is to type “vislog” in the console). Then activate the logging of RVT invalidations using r.VT.RVT.VisLogManualInvalidations 1. Then press the Start button in the visual logger window, to start recording. And then use the paint brush (or Teleport, if you manage to run this in the standalone game, save the visual log file, and open it afterwards). If you select the Virtual Texture Component track in the visual logger, you should be able to see a box in the viewport, showing the invalidation areas during the brush stroke (you can also use Auto-Scroll in order to see the invalidated area as you paint). Note that the box might be located under the landscape so you may have to hide the landscape temporarily to see it. The important thing is : the box should have the size of the brush, pretty much. If it’s too large, then it means that a very large area of landscape gets invalidated, which means, more RVT pages to update at once…
Note that this CVar (r.VT.RVT.VisLogManualInvalidations) has been introduced in UE5.6, so if the regression appeared between 5.6 and 5.7, it would be interesting to do the same test in the UE5.6 version, and see if the invalidation area is smaller there.
That being said, it’s unlikely that the invalidated area would be the problem, since you say that it also happens in the cooked game. But it’s better to check.
Let us know what you find.
Cheers,
Jonathan
Here’s what it should look like :
Thanks for the quick reply!
> What version of UE were you using before this started happening?
Unfortunately, we upgraded from UE 5.2 more or less straight to 5.7, so that’s a big range to cover.
> Is there a way for you to go back in time and try to see when this started occurring?
I did try to make sure to get at least the editor into a semi working state for each minor engine version, so I should be able to roll back to the versions in between and check where it broke. Will just take some time to compile the entire engine for every version, I’ll update you once I’m done.
As you suspected, the visual logger doesn’t show anything out of the ordinary. The invalidated areas seem to match the painted areas pretty accurately.
Alright, so I ran some tests and here’s what I found:
- The issue appeared somewhere between 5.2.1 and 5.4.3 (unfortunately we don’t have a working in-between state for 5.3 so I can’t be more precise than that)
- When comparing the behavior between 5.2.1 and the later versions, I noticed that in 5.2 the landscape immediately showed an updated low-res version and then gradually updated to higher resolutions later on. The delay before the high-res tiles are rendered seems similar to the delay we are experiencing in the newer versions, but we are now missing the low-res intermediate step.
- I have attached a video of the behavior in 5.2.1 for comparison.
editor_5.2.1.mp4(22.6 MB)
Interesting. We notice that your RVT page size is actually quite small. That could lead to a lot of strain on RVT if something in the material artificially samples very fine mips. One way to validate this would be :
- In UE5.7 : set the CVar r.VT.Borders to half your RVT asset’s “Page Table Size” (i.e. r.VT.Borders 128 if Page Table Size is 256)
- Prior to UE5.7 : use the CVar r.vt.rvt.mipcolors instead
This colors pages by mip. The number is the “border size” so setting it at half the page size will cover the page.
Then capture the behavior before and after the regression (and post them here). You should see “lighter” colors (white/yellow) for mip 0/1 and then some other rainbow-style colors as you move down the mip chain. If there’s something in the material that leads to more fine mips being sampled, you should see a majority of light colors on screen and you can then start to dig into it and see where that might come from.
Let us know what you find
Cheers,
Jonathan
It should look something like this :
Hey Jonathan, thanks for the suggestions!
I did as you suggested in UE 5.7 and attached a video of that as well as some screenshots of our current RVT settings, not sure if they make any sense.
Unfortunately, it seems like r.vt.rvt.mipcolors does not exist (yet?) in UE 5.2 and I haven’t found another way to visualize the mip levels there, so I can’t give you a good “before” video for comparison.
While looking into ways to visualize the mip levels however, I stumbled upon a very curious setting in our terrain material:
It looks like all our RVT samples were using a mip bias of -2, which… doesn’t seem ideal. I tried setting that to 0 and it drastically improves the rendering performance and virtual texture pool usage.
This in itself doesn’t reduce the delay while painting, but I can now increase the value of r.VT.MaxUploadsPerFrameInEditor without tanking performance, which allows us to paint without a noticeable delay.
However I’d still be interested in figuring out why the intermediate low-res updates of the RVT no longer seem to happen in 5.7.
virtual_textures.zip(33.6 MB)
Indeed, the -2 bias certainly didn’t help! One thing I forgot to ask you : when you’re testing, say UE5.2.1 vs. UE5.4.3, is the content identical (in other words, did you use a mip bias of -2 in the latter but not the former)? I wonder if the regression could come from such a content issue.
But our best theory for what you’re seeing is that between UE5.3 and 5.4, the update behavior was changed, with the introduction of r.VT.RVT.DirtyPagesKeptMappedFrames, which it doesn’t unmap pages (forcing us back to lower res pages briefly) and instead queues the existing pages to be reproduced. You could try setting that to 0, just to see if that leads to the same behavior as UE5.2.1, i.e. that invalidated pages become immediately blurry.
Let us know what you find. Videos with/without r.VT.RVT.DirtyPagesKeptMappedFrames and with/without increasing the upload budget (r.VT.MaxUploadsPerFrameInEditor) would be useful.
Thanks,
Jonathan
Thanks for the reply!
So I followed all of your suggestions and attached the results.
- Landscape debug mode shows that the painting happens instantly, it’s only the virtual texture that takes a long time to update.
- r.VT.Residency.Show shows that the virtual texture seems to be pretty much full to the limit, even when only a very small chunk of the landscape is on screen (I tried increasing the size of the virtual texture pool but it seems there is a hard limit that doesn’t allow it to go above 512MB?)
- Disabling Temporal AA and DLSS did not cause any noticeable changes.
- Our project does not support D3D11, but the issue is also reproduceable with Vulkan, so it does not seem to be API-specific.
The previous upload seems to have failed, let’s try again.
Here’s the RenderDoc capture:
editor_renderdoc.rar(882 MB)
Here’s a video of the landscape layer debug view:
editor_layer_debug.mp4(31 MB)
And here’s a video showing the virtual texture residency:
editor_vt_residency.mp4(28.4 MB)
We have also noticed that we have a very similar issue in the standalone cooked build. Here it’s very noticeable after teleporting that the virtual texture will take several seconds before it even starts updating.
Could this be related?
Here’s a video showcasing the issue in the cooked build:
game_vt_residency.mp4(45 MB)
And here’s a matching RenderDoc capture.
game_renderdoc.rar(875 MB)
We have three RuntimeVirtualTextureVolumes, each spanning our entire world. Though only the one used for the landscape material causes any issues, the other two are barely used (and use different texture pools).
I tried playing with the mentioned console variables in the cooked build, and specifically increasing r.VT.MaxUploadsPerFrame seems to solve the issue. So we’ll probably just try increasing this value during the loading screen.
In the editor, setting r.VT.MaxUploadsPerFrameInEditor seems to also kind of “solve” the issue (though I could have sworn I had tried that before without any effect), meaning the landscape will indeed update within one frame. BUT it absolutely tanks the framerate while drawing (from about 100 down to below 10 fps on an R9 9950X3D/RTX 4090). So this definitely isn’t an ideal solution.
