Shadows on mesh terrain

[chrisaverage]

Hi, I have another issue with (I think) mesh terrain. There are weird shadow blobs on it that shouldn’t be there. It kinda looks like some garbage in the VSMs and it looks like moving along with the cascades, but I’m having trouble with identifying where it comes from or how to clear it.

This happens when I open the map in editor, play in PIE and also in a packaged build.

If I walk away far enough or fly the camera far enough it fixes itself (the shadows are gone).
If I move an object across them it fixes itself in sort of a block by block pattern.
If I edit the mesh terrain in any way it fixes itself.
If I rebuild the terrain it fixes itself.
If I change the view mode to one of the debug ones it fixes itself.
If I hide and show that piece of terrain mesh it fixes itself.
When I’m in PIE and press F8 twice (detach and reattach the player controller) it fixes itself.
There’s a bunch of other things that fixes it when I touch them, e.g. change some settings, use any of the build options etc.

The problem is it only fixes itself temporarily i.e. if I reload the map, restart editor or the packaged game it’s back in the same place. It wouldn’t be that big of an issue in editor, but it’s also in the packaged game.

I tried force rebuilding the mesh terrain, deleting the generated actors (preview and the compiled renderdata actors), resaving everything, editing the terrain, nuking the Saved and Intermediate folders from the game and localappdata directories, nothing helps. After I reopen the map the shadows are back, so it seems to regenerate them from something, but I don’t know what.

I’m using UE 5.8.1, Nanite, Lumen and VSMs.

I’d greatly appreciate some creative ideas how to identify the cause and get rid of it. Debugging it proves difficult, since most things/settings I touch fix it temporarily and I’m running out of ideas.

When it’s ok:

When it’s not ok in game:

When it’s not ok in editor:

[chrisaverage]

The plot thickens. This is what happens when I turn on Ray Traced Shadows.
The shadows don’t seem to follow any terrain features. Just a bunch of random blobs.

It’s completely borked

[chrisaverage]

Right, this seems to happen when I enable Nanite in Static Mesh Transformer options. Looks like for some reason these broken VSMs get cached when the mesh loads.

I obviously don’t want to disable nanite or VSMs, but as an ugly workaround I can disable VSM caching (r.Shadow.Virtual.Cache 0) before I load the terrain and then reenable it when the meshes are loaded and that seems to work. I might need to dig deeper into the loading of these meshes to fully figure out what’s happening, but that seems like a tall order.

[Deamonche]

Hey there! Nanite can be acting odd with animations, shadows and complicated shaders. I had jumping trees,uyp and down, because of an wind animations. Have you tried different materials on your terrain? Does your terrain really need Nanite enabled? Here is a really good article on the topic:

https://medium.com/@sarah.hyperdense/understanding-nanite-when-to-use-it-and-when-to-skip-it-b7fcfadc3058

[chrisaverage]

It’s a static terrain mesh. There are no animations and the material is (for now) just the simple default grid, as seen on the screenshots.

This particular piece is simple test, but I want to try the more complicated features of mesh terrain like overhangs, caves and more advanced non-destructive modifiers. I know there are issues, but Nanite really helps with visible pop in of traditional LODs. Currently I’m tripping on the simplest cases though. Seems the mesh terrain module is VERY experimental.

I’m not really looking for alternatives. I’m not shipping tomorrow and I know there are tried and tested ways to do that. I can use standard LODs, Landscape or even just static meshes if I needed to, but I’d rather understand and fix the problem.

[MostHost_LA]

I doubt you can.

Seems like a core bug. You can pull the engine source, build, and try to fix epic’s c-rap. But why?

Trust me, no one anywhere will care and epic won’t even allow the idea of a pull request

It is a good exercise. But that’s really all you get out of it (and some hair loss).

[chrisaverage]

Yeah, I’ve reported my share of bugs to Epic that were very much ignored, trust me. I do keep my copy of the source and have fixed a bunch of issues in the engine. I know a couple people at Epic that work on various features and they often talk about the layers of red tape they have to go through to do anything. I’ve heard enough stories that I don’t even try to bother them directly if I have issues. They couldn’t help me even if they wanted to and it’s often the case that if someone doesn’t work on particular feature they have no say whatsoever. It’s a huge code base and a corpo structure after all.

That being said I’m not doing it for someone to care. If they do great, but I’m doing it because if the tech does 95% of what I need and I can fix the 5% myself then that’s a win for me - saving a lot of time, getting what I want, learning a ton and, if I can, sharing the knowledge after to save someone else some time.

Sure, maintaining changes to the engine is a chore I avoid if I can, but who ever said gamedev would be easy right? A lot can be fixed/worked around without actual source changes and I’m not afraid of hair loss anymore. I’ve been doing this so long I have very little left anyway

[Petrockets (Alex Peterson)]

Hi there!

As you noted, there are a few known issues where a Nanite landscape or mesh terrain streams in and the VSM cache renders with the lower resolution mesh before streaming is complete and so you get incorrect shadows because the VSM mesh and the display mesh are different. Turning off VSM caching with r.shadow.virtual.cache=0 is a valid workaround, and in most cases we’ve seen, disabling the cache momentarily while the geometry is streaming in, and then can re-enable the cache is fine.

Another route that has worked with Nanite landscape in some cases is to use r.Nanite.VSMInvalidateOnLODDelta=1 and adjust r.Shadow.Virtual.DeferredInvalidationBudget to throttle the invalidations and avoid hitches. I haven’t tested this with Mesh Terrain, but it should work, though this is also experimental.

CL#53832027 (bcdecd) [VSM] Added throttle for the invalidation triggered from r.Nanite.VSMInvalidateOnLODDelta such that it can be turned on without causing large performance regressions. The budget is set via r.Shadow.Virtual.DeferredInvalidationBudget (defaults to infinite).

1. static TAutoConsoleVariable<int32> CVarVSMDeferredInvalidationBudget(
2. TEXT("r.Shadow.Virtual.DeferredInvalidationBudget"),
3. -1,
4. TEXT("Maximum number of VSM physical pages whose deferred-invalidation extended flag may be consumed per frame.\n")
5. TEXT("Excess pages keep the flag set and retry next frame, so the total work is preserved but spread across frames.\n")
6. TEXT("Today's only source of this flag is the streaming-LOD-delta channel from Nanite (see r.Nanite.VSMInvalidateOnLODDelta).\n")
7. TEXT(" 0 disables consumption entirely.\n")
8. TEXT(" -1 (default) is unbounded (pre-throttle behavior, may spike)."),
9. ECVF_RenderThreadSafe | ECVF_Scalability
10. );

If you do find a fix, we do look at PRs. The priority for these experimental features are not as high currently, but we are working on them.

[chrisaverage]

Hi Alex,

This seems like a far better solution, thank you very much. I was still monkeying around the streaming code looking how to achieve something exactly like that, so I’m happy that it’s already in. Saves me a lot of work.

I just tested it and it works well in my case. I still see a bit of wobble in the geometry itself on load, as it loads lower res first, but that’s just a usual loading stuff problem I can cover up with art and such. The important part for now is it’s transient and doesn’t leave invalid state at the end.

As for the hitching - I can see that potentially causing issues, but I’ll leave the budgeting tests to see if it’s a real problem for me until I have more actual final assets. Thanks again!