Cannot use Virtual Shadow Maps in non-Nanite projects

Development Rendering
, ,
1

Hello,

In our project we use Virtual Shadow Maps but we do not use Nanite. Since updating to Unreal 5.6 Virtual Shadow Maps require Nanite to be enabled at the project level to work (i.e. r.Nanite.ProjectEnabled=True). This wasn’t the case in Unreal 5.5. This is unfortunate as this change risks bloating our game packages with unneeded Nanite data.

This change in behaviour seems to originate from CL 41759390.

There’s a thread about this issue here: https://forums.unrealengine.com/t/in\-5\-6\-vsm\-not\-working\-unless\-nanite\-is\-enabled/2652638\.

Is it possible to revert to the old behaviour?

Thanks

2

Steps to Reproduce

  • Open a non-Nanite project in Unreal 5.6 (r.Nanite.ProjectEnabled=False)
  • Open a level setup for dynamic shadowing (e.g. one movable directional light and some static meshes casting and receiving shadows)
  • Enable VSMs via the console: r.Shadow.Virtual.Enable 1
  • Notice shadows within the dynamic shadow distance are either not rendered or rendered with CSMs. Notice that VSM visualization modes also do not work (e.g. View Mode => Virtual Shadow Map => Virtual Page).
3

Hello,

After looking into this it does appear that this change was unintentional and you can probably revert the change - or part of it - locally and continue as usual. However, can you provide more information about why you decided to use VSM without Nanite in your project? Where did you see bloat in your game package if none of your content used Nanite - was it just for shaders? And if so, why do you prefer VSM over CSM? Is it for quality?

4

Thanks, we’ll try reverting parts of the change.

Unfortunately we didn’t take the time to measure the bloat in our game packages. The documentation mentioned that we would be saving on shaders by disabling Nanite and that was good enough for us.

Regarding our preference for VSMs over CSMs, your guess is good: it is for quality reasons.

5

Thanks for the added details - can you also share why you chose not to use Nanite? Was there a measurable performance or quality loss or was it not supported on your target platforms?

6

I’m currently consulting with the parties involved in our evaluation of Nanite, I’ll get back to you when I have a bit more information on the topic.

I took some measurements regarding the Nanite-caused package bloat: disabling Nanite at the project level reduced the size of our pak files by ~45MB: size of Game/Content/Paks folder went down from 2.26GB to 2.22GB (win64, Test config).

7

Regarding Nanite, we chose not to use it because we had poorer or equal gpu performance with it in the scenarios we tested (when compared with traditional mesh LODs). Our meshes are often simple (i.e. low triangle density, large triangles) which probably explains our results. We also need traditional LODs anyway as some of our target platforms do not support Nanite.

8

I can confirm that reverting some of the changes in RenderUtils.cpp did the trick. Here’s our patch in case it helps:

==== //UE5/Release-5.6/Engine/Source/Runtime/RenderCore/Private/RenderUtils.cpp#3 (text) ====
 
[Content removed]13 @@
 bool DoesPlatformSupportNanite(EShaderPlatform Platform, bool bCheckForProjectSetting)
 {
 	// Nanite allowed for this project
-	static FShaderPlatformCachedIniValue<int32> CVarPlatformNaniteProjectEnabled(TEXT("r.Nanite.ProjectEnabled"));
-	bool bIsNaniteProjectEnable = CVarPlatformNaniteProjectEnabled.Get(Platform) != 0;
-
-	if (UNLIKELY(!bIsNaniteProjectEnable))
+	if (bCheckForProjectSetting)
 	{
-		return false;
+		const bool bNaniteSupported = GNaniteProjectEnabled != 0;
+		if (UNLIKELY(!bNaniteSupported))
+		{
+			return false;
+		}
 	}
 	
 	// Make sure the current platform has DDPI definitions.
[Content removed]7 @@
 {
 	static const auto EnableVirtualSMCVar = IConsoleManager::Get().FindConsoleVariable(TEXT("r.Shadow.Virtual.Enable"));
 	const bool bVirtualShadowMapsEnabled = EnableVirtualSMCVar ? (EnableVirtualSMCVar->GetInt() != 0) : false;
-	return bVirtualShadowMapsEnabled && DoesRuntimeSupportNanite(ShaderPlatform, true /* check for atomics */, true /* check project setting */);
+	return bVirtualShadowMapsEnabled && DoesRuntimeSupportNanite(ShaderPlatform, true /* check for atomics */, false /* check project setting */);
 }
 
 bool UseVirtualShadowMaps(EShaderPlatform ShaderPlatform, const FStaticFeatureLevel FeatureLevel)
[Content removed]13 @@
 
 bool DoesPlatformSupportVirtualShadowMaps(EShaderPlatform Platform)
 {
-	return DoesPlatformSupportNanite(Platform);
+	return DoesPlatformSupportNanite(Platform, false /* check project setting */);
 }
 
 bool DoesPlatformSupportNonNaniteVirtualShadowMaps(EShaderPlatform ShaderPlatform)
 {
 	static const auto EnableCVar = IConsoleManager::Get().FindConsoleVariable(TEXT("r.Shadow.Virtual.NonNaniteVSM"));
-	return EnableCVar->GetInt() != 0 && DoesPlatformSupportNanite(ShaderPlatform);
+	return EnableCVar->GetInt() != 0 && DoesPlatformSupportNanite(ShaderPlatform, false /* check project setting */);
 }
 
 bool UseNonNaniteVirtualShadowMaps(EShaderPlatform ShaderPlatform, const FStaticFeatureLevel FeatureLevel)