Using VSM and Distance Field Shadows on the same scene results in missing shadows

Development Rendering
, ,
1

I wish to report an issue with how VSMs and Distance Field Shadows interact. We’re working on a large scale open world game and wish to utilize distance field shadowing for a major part of our screen. We also want to use VSM’s in the same area and benefit from strengths of both approaches.

The current issue we’re facing, is that following the documentation, we’re using “Dynamic Shadow Distance” property of the directional light, to define the boundary where VSM’s end and where we’d like to rely only on distance fields. The thinking behind this was that we don’t need high detail shadows at the distance, and so we could save some performance.

The problem with this approach is, that Distance Fields Shadows also use this same property, to define “at what distance do we start”. We don’t want Distance Field Shadows to be gone at closer distances, we want them to be visible at any distance. The problem is, that in our case, the distance field shadows are very noticeable and make a major part of the shadowing, adding more detail and depth to the scene than the VSM’s alone do. Unfortunately that cannot be done, since with a single property we either have:

  • VSMs at distance 100K AND Distance Field Shadows beyond 100K - no Distance Field Shadows below 100K, undesirable
  • VSMs at distance of only 1K AND Distance Field Shadows beyond 1K - VSMs range way too short, undesirable

So with the goal of figuring out how to work this one out, I’ve settled for using “r.Shadow.Virtual.Clipmap.LastLevel”. Even the documentation says, that this indirectly determines the max range of VSM, which is exactly what I want. Following that idea:

  • I set “r.Shadow.Virtual.Clipmap.LastLevel” to a lower level, around 15-17, at a range where I want VSMs to end
  • I set “Dynamic Shadow Distance” to 0, to have Distance Field Shadows visible at all ranges, even close up

Unfortunately, that’s when I hit the wall with the bug, which I am reporting here. Our scene consists mostly of nanite meshes, but we do have some non-nanite ones. This is a smaller issue however and we could probably even ignore it. The main problem is that many HLODs are nanite-enabled and they loose the shadow, which is very visible and unacceptable. They never cast distance field shadow, at all. Non nanite HLODs are okay.

Please assist with figuring out some approach or alternatives here. Either a place in code where I can split the usage of this property to 2 separate ones, or some bugfix to the killing of Distance Field Shadows due to VSM clipmap setup, which I believe should not happen.

Screens from the provided repro steps, showing what we’re experiencing, as attachements.

[Attachment Removed]

2

Steps to Reproduce

No custom changes in engine required, clean 5.5.

  1. Use the Open World Template map
  2. Place a high nanite static mesh on the scene, for example a tree.
  3. Place a non-nanite static mesh on the scene, also high one, or use the same asset as in #2, but click “disallow nanite” in actor properties (that’s what I did).
  4. Notice that both of those assets are casting shadows on the landscape.
  5. Use the “r.Shadow.Virtual.Clipmap.LastLevel” CVar to reduce VSM range, so that you can see where the VSM shadow ends. I usually use the range between 13 - 15. This might require adjusting the camera position to achieve similar result as on the screens I present.
  6. Enable “Distance Field Shadowing” bool on the directional light and set the radius to be a large one (2^22 ideally, to make sure to cover entire default VSM region)
  7. Set the DistanceField Trace Distance to a high value (100K) so that it does not impact and influence the test
  8. To allow Distance Fields to cast shadow close enough for the test to work, set “Dynamic Shadow Dystance” to 0.
  9. Notice that the assets behave differently. Non Nanite asset has DF shadow overlaid on top of the VSM one, while the nanite one has only the VSM shadow. Distance field does not cover the range outside the VSM.
  10. Set “r.Shadow.Virtual.Clipmap.LastLevel” back to default 22.
  11. Notice that the Nanite enabled static mesh now correctly casts Distance Field shadows after the boundary

I can provide the repro if necessary, but it is simple enough with the above steps.

[Attachment Removed]

3

Hello,

Thank you for reaching out.

I’ve been assigned this issue, and we will be looking into this shadow artifact for you.

[Attachment Removed]

4

Hello,

Thank you for reporting this. I can confirm this shadow issue can be reproduced as described in the latest CL, and have opened a bug report:

https://issues.unrealengine.com/issue/UE\-364657

The tracker will be visible after it is approved internally at Epic Games and is publicly accessible.

If you have any further questions, please let us know.

[Attachment Removed]

5

Hello,

Thank you for sharing your solution.

The bug report links back to this ticket, so all comments here can be referenced.

Can we go ahead and close this ticket now?

[Attachment Removed]

6

Thank you. I will go ahead and close this ticket.

[Attachment Removed]

7

Hi,

Actually I do. I’m reporting a few more issues I’ve found around this topic, which prevent me from using even workarounds.

First is related to the fact that when I do use the “Dynamic Shadow Distance”, the shadows are actually getting corrupted, which also breaks visuals: [Content removed]

Second relates to the fact, that the distance field shadows are never actually cast, at all, for nearby objects, even without messing with the clipmap cvars: [Content removed]

I’d like to ask for some support on the issues, is it possible for someone familiar with the codebase to provide any hints as to what could cause it, or where to look for solutions? I can get back to the issue in a few weeks, but I definitely cannot wait for 5.8 for a fix (and that’s optimistic timeline), so I need something faster.

EDIT: I’ve also reported a third one, regarding WPO, adding here to keep links: [Content removed]

[Attachment Removed]

8

Hello,

Thank you for creating separate tickets for the other bugs.

To address this yourself, you could look into the shader “FCullObjectsForShadowCS”, and it’s parameter “bDrawNaniteMeshes” from DistanceFieldShadowing.cpp.

For engine modifications like this, we can offer information about existing functionality, and we leave the details of the implementation to you.

Please let us know if this helps.

[Attachment Removed]

9

Hey,

I’ve finally had the time to dig deeper here. Yes, this helps. A simple fix to always set this bool to true (actually not this bool directly, but the caller above) results in a great visual gain in the mid and far range, but as expected - worse visual look in the close views. VSMs are far more detailed, especially on complex meshes. So, not ideal.

Ideally, we’d get the self shadow from distance fields on all objects, even the close ones, and trigger non self shadow only for objects outside the last VSM clipmap. So, VSMs cast shadows as usual, no change here, but distance field casts self shadow (branches and leaves onto other branches of the same tree) only on close, and full shadow (on other objects, landscape) when far. If you have any idea, or can ask for any ideas, on how to do that one - then I’m all ears. For now I suppose I’ll keep this noted down for the future.

What I did next is modifying the shader source, to only cull nanite instances which will actually have VSM shadows. Those outside the far clipmap will get distance field shadows. I believe that this is a pretty good fix for the actual issue. I decided to make it flexible and set the distance with another cvar (added in the file), instead of directly using the VSM clipmap.

The main thing to do is use the “ViewDist2” already calculated in the shader to compare against the wished cull distance squared. Also, move the “LoadDFObjectData” later, since it is not needed until the next ‘if’ condition.

The problem with this solution is that at the boundary where the range is set, the mesh shadow pops, resulting in visual glitches when moving. Culling is on/off, there’s no blend, no smooth interpolation.

So the last iteration was to skip that shader code modification entirely, simply set “bCullingForDirectShadowing” to false with a CVar to change the mode (and allow all nanite meshes into the distance field), and then manipulate the “Dynamic Shadow Distance” property of the directional light, to ideally match the ranges. This however produces a missing line of shadows, as on the screen:

[Image Removed]

This can be controlled by property “Transition Fraction”, but it’s not enough even with the max available to set (0.3). So I ended up adding some offset to account for that. A good offset turned out to be -512, so reduce the “Dynamic Shadow Distance” so that the distance field shadow starts sooner, than the max VSM clipmap range stops. And keep the transition to 0.3. This gives ideal solution. Smooth blend, smooth self shadow:

[Image Removed]

Thank you for your help. Consider adding this info to the reported issue, because I believe this is how it should look in the first place. That bool, “bCullingForDirectShadowing” is always set to true in the codebase, so the condition never makes sense. Seems like unfinished code.

[Attachment Removed]

10

Yes.

[Attachment Removed]