Frequent crashes with Nanite Tessellation in UE 5.4.2 (Assertion failed: !RasterizerPass.ClusterComputeShader.IsNull())

Development Rendering
,
1

Hi everyone :slight_smile: ,

Since upgrading to UE 5.4.2 and enabling tessellation in my project, as well as converting my landscape to Nanite, I’ve been experiencing frequent crashes. The crash details are as follows:

Runnable thread Foreground Worker #1 crashed.
=== Critical ===
Assertion failed: !RasterizerPass.ClusterComputeShader.IsNull() [File:D:\build\++UE5\Sync\Engine\Source\Runtime\Renderer\Private\Nanite\NaniteCullRaster.cpp] [Line: 3905] 
[Callstack] 0x00007ffb38427601 UnrealEditor-Renderer.dll!`FPixelShaderUtils::AddRasterizeToRectsPass<FNaniteMarkStencilPS,FNaniteMarkStencilRectsParameters>'::`2'::<lambda_1>::operator()() [D:\build\++UE5\Sync\Engine\Source\Runtime\Renderer\Private\Nanite\NaniteCullRaster.cpp:3905]
[Callstack] 0x00007ffb3842cab0 UnrealEditor-Renderer.dll!`FPixelShaderUtils::AddRasterizeToRectsPass<FNaniteMarkStencilPS,FNaniteMarkStencilRectsParameters>'::`2'::<lambda_1>::operator()() [D:\build\++UE5\Sync\Engine\Source\Runtime\Renderer\Private\Nanite\NaniteCullRaster.cpp:4051]
[Callstack] 0x00007ffb38428ab2 UnrealEditor-Renderer.dll!`FPixelShaderUtils::AddRasterizeToRectsPass<FNaniteMarkStencilPS,FNaniteMarkStencilRectsParameters>'::`2'::<lambda_1>::operator()() [D:\build\++UE5\Sync\Engine\Source\Runtime\Renderer\Private\Nanite\NaniteCullRaster.cpp:4124]
[Callstack] 0x00007ffb38479669 UnrealEditor-Renderer.dll!FRHICommand<FRHICommandPollOcclusionQueries,FRHICommandPollOcclusionQueriesString2138>::ExecuteAndDestruct() [D:\build\++UE5\Sync\Engine\Source\Runtime\Core\Public\Tasks\TaskPrivate.h:855]
[Callstack] 0x00007ffb37ccae3e UnrealEditor-Renderer.dll!UE::Tasks::Private::FTaskBase::TryExecuteTask() [D:\build\++UE5\Sync\Engine\Source\Runtime\Core\Public\Tasks\TaskPrivate.h:504]
[Callstack] 0x00007ffb37c84804 UnrealEditor-Renderer.dll!LowLevelTasks::TTaskDelegate<LowLevelTasks::FTask * __ptr64 __cdecl(bool),48>::TTaskDelegateImpl<`LowLevelTasks::FTask::Init<`UE::Tasks::Private::FTaskBase::Init'::`2'::<lambda_1> >'::`13'::<lambda_1>,0>::CallAndMove() [D:\build\++UE5\Sync\Engine\Source\Runtime\Core\Public\Async\Fundamental\TaskDelegate.h:171]
[Callstack] 0x00007ffba2a124c5 UnrealEditor-Core.dll!LowLevelTasks::FTask::ExecuteTask() [D:\build\++UE5\Sync\Engine\Source\Runtime\Core\Public\Async\Fundamental\Task.h:627]
[Callstack] 0x00007ffba2a121ff UnrealEditor-Core.dll!LowLevelTasks::FScheduler::ExecuteTask() [D:\build\++UE5\Sync\Engine\Source\Runtime\Core\Private\Async\Fundamental\Scheduler.cpp:155]
[Callstack] 0x00007ffba2a3a155 UnrealEditor-Core.dll!LowLevelTasks::FScheduler::WorkerMain() [D:\build\++UE5\Sync\Engine\Source\Runtime\Core\Private\Async\Fundamental\Scheduler.cpp:397]
[Callstack] 0x00007ffba2a023d0 UnrealEditor-Core.dll!UE::Core::Private::Function::TFunctionRefCaller<`LowLevelTasks::FScheduler::CreateWorker'::`2'::<lambda_1>,void __cdecl(void)>::Call() [D:\build\++UE5\Sync\Engine\Source\Runtime\Core\Public\Templates\Function.h:406]
[Callstack] 0x00007ffba2c07c93 UnrealEditor-Core.dll!FThreadImpl::Run() [D:\build\++UE5\Sync\Engine\Source\Runtime\Core\Private\HAL\Thread.cpp:69]
[Callstack] 0x00007ffba306c93d UnrealEditor-Core.dll!FRunnableThreadWin::Run() [D:\build\++UE5\Sync\Engine\Source\Runtime\Core\Private\Windows\WindowsRunnableThread.cpp:149]
[Callstack] 0x00007ffba30602b7 UnrealEditor-Core.dll!FRunnableThreadWin::GuardedRun() [D:\build\++UE5\Sync\Engine\Source\Runtime\Core\Private\Windows\WindowsRunnableThread.cpp:79]
[Callstack] 0x00007ffc5ec7257d KERNEL32.DLL!UnknownFunction []

Turning off Nanite tessellation resolves the issue, but I would prefer not to disable this awesome feature (I’ve missed it since UE4).

The crashes occur when I open my world partition map (happens the moment the map opens) or when I run the packaged game (again, right when the level opens). It does not crash every time, but rather occasionally, and without any visible pattern.

I’ve found that tweaking the following values can somewhat mitigate the frequency of crashes, but I haven’t been able to eliminate them completely:

r.Nanite.MaxVisibleClusters=10485760 ; [default: 4194304]
r.Nanite.MaxCandidateClusters=16777216 ; [default: 16777216] Maximum number of Nanite clusters before cluster culling.
r.Nanite.MaxNodes=10485760 ; [default: 2097152]

EDIT 1: See my second post for a more detailed breakdown of conducted tests.

One of my observations is that setting r.Nanite.DicingRate to 1 in stat NaniteStreaming results in the fewest Page Requests, but it significantly increases the number of clusters. On the other hand, when I set it to 4, the visuals look fine and there are fewer clusters, but the number of Page Requests goes up. How should I interpret this?

Lastly, is it normal for the Nanite Triangles debug view to be this dense, or could there be something wrong with it?

1719562481-EZB9ju1p53
1719562481-EZB9ju1p531815×779 483 KB

Has anyone else experienced this issue or found a solution? Any help or suggestions would be greatly appreciated!

Thank you in advance!
Thomas Winged

1 Like
2

Hi everyone :blush:,

Previously, I attempted to adjust various parameters to reduce the crashes, but I couldn’t eliminate them entirely. So, I decided to spend more time conducting a detailed analysis. Below, I present the results of the tests I conducted:

Initially, the tests showed some improvement, but it turned out to be a statistical error, as subsequent runs consistently resulted in crashes. Unfortunately, none of the commands improved the situation in a reliable manner.

Detailed Breakdown of Tests Conducted to Enable and Stabilize Nanite Tessellation

Test 1: Enabling Basic Nanite Tessellation

Objective: To activate Nanite tessellation and evaluate stability.

r.Nanite.AllowTessellation=1 ; [default: 0] Enables Nanite displacement
r.Nanite.Tessellation=1 ; [default: 0] Activates Nanite tessellation

Results: 4 out of 5 tests resulted in crashes.
Consensus: Nanite tessellation is active but highly unstable, indicating potential issues with the experimental feature’s initial implementation.

Test 2: Disabling Tessellation

Objective: To determine if disabling tessellation improves stability.

r.Nanite.AllowTessellation=0 ; [default: 0]
r.Nanite.Tessellation=0 ; [default: 0]

Results: 0 out of 5 tests resulted in crashes.
Consensus: Stability achieved without tessellation, confirming that the crashes are related to the tessellation feature.

Test 3: Increasing Nanite Streaming Memory

Objective: To test if increasing the streaming memory pool size reduces crashes.

r.Nanite.AllowTessellation=1 ; [default: 0]
r.Nanite.Tessellation=1 ; [default: 0]
r.Nanite.Streaming.StreamingPoolSize=1024 ; [default: 512] Increases streaming pool size to 1024 MB.

Results: 2 out of 5 tests resulted in crashes.
Consensus: Increased memory appears to reduce crash frequency, suggesting that memory allocation might be a factor in stability.

Test 4: Further Increasing Streaming Memory

Objective: To test the effects of further increasing the streaming memory pool size.

r.Nanite.AllowTessellation=1 ; [default: 0]
r.Nanite.Tessellation=1 ; [default: 0]
r.Nanite.Streaming.StreamingPoolSize=2048 ; [default: 512]

Critical Error:

Streaming pool size (2048MB) must be smaller than the maximum allocation supported by the graphics hardware (2048MB).

Consensus: The maximum supported allocation size was reached, causing a critical error. This suggests a hardware limitation.

Test 5: Intermediate Streaming Memory Size

Objective: To find a balanced streaming memory pool size that might improve stability.

r.Nanite.AllowTessellation=1 ; [default: 0]
r.Nanite.Tessellation=1 ; [default: 0]
r.Nanite.Streaming.StreamingPoolSize=1536 ; [default: 512]

Results: 2 out of 5 tests resulted in crashes.
Consensus: No significant improvement, potentially due to statistical variance.

Test 6: Disabling Specific Nanite Features

Objective: To identify specific Nanite features that might be causing instability.

r.Nanite.AllowTessellation=1
r.Nanite.Tessellation=1
r.Nanite.AsyncRasterization=0 ; [default: 1]
r.Nanite.MeshShaderRasterization=0 ; [default: 1]
r.Nanite.PrimShaderRasterization=0 ; [default: 1]
r.Nanite.Streaming.AsyncCompute=0 ; [default: 1]
r.Nanite.ComputeRasterization=0 ; [default: 1]

Results: 3 out of 5 tests resulted in crashes. Landscape disappeared.
Notes: Crashes persist, and landscape issues are likely due to enabled tessellation.

Test 7: Disabling Programmable Rasterization

Objective: To test if disabling programmable rasterization improves stability.

r.Nanite.AllowTessellation=1
r.Nanite.Tessellation=1
r.Nanite.ProgrammableRaster=0 ; [default: 1]

Results: 0 out of 5 tests resulted in crashes.
Notes: Crashes stopped, but landscape has no masked alpha on masked materials.

Test 8: Disabling Async Rasterization

Objective: To check if disabling async rasterization helps with stability while keeping programmable rasterization enabled.

r.Nanite.AllowTessellation=1
r.Nanite.Tessellation=1
r.Nanite.AsyncRasterization=0 ; [default: 1]

Results: 3 out of 5 tests resulted in crashes.
Notes: Crash frequency remained unchanged.

Test 9: Forcing Shader Bundle Dispatch Emulation

Objective: To evaluate the impact of forcing shader bundle dispatch emulation on stability.

r.Nanite.AllowTessellation=1
r.Nanite.Tessellation=1
r.Nanite.Bundle.Emulation=1 ; [default: 0]

Results: 3 out of 5 tests resulted in crashes.
Notes: Crash frequency remained unchanged.

Test 10: Reducing Nanite Clusters

Objective: To test if reducing the count of Nanite clusters by controlling tessellation dicing and distance density improves stability.

r.Nanite.AllowTessellation=1
r.Nanite.Tessellation=1
r.Nanite.MaxPixelsPerEdge=2 ; [default: 1]
r.Nanite.DicingRate=8 ; [default: 2]

Results: 3 out of 5 tests resulted in crashes.
Notes: Crash frequency remained unchanged.

Test 11: Increasing Nanite Clusters

Objective: To test if increasing the count of Nanite clusters by controlling tessellation dicing affects stability.

r.Nanite.AllowTessellation=1
r.Nanite.Tessellation=1
r.Nanite.DicingRate=1 ; [default: 2]

Results: 3 out of 5 tests resulted in crashes.
Notes: Crash frequency remained unchanged.

Test 12: Using Epic’s Proposed Values

Objective: To evaluate the stability of Epic’s proposed settings for Nanite, found in ElectricDreamsEnv project.

r.Nanite.AllowTessellation=1 ; [default: 0]
r.Nanite.Tessellation=1 ; [default: 0]
r.Nanite.MaxVisibleClusters=10485760 ; [default: 4194304]
r.Nanite.MaxCandidateClusters=16777216 ; [default: 16777216]
r.Nanite.MaxNodes=10485760 ; [default: 2097152]

Results: 4 out of 5 tests resulted in crashes.
Consensus: No significant improvement in stability, indicating that these values might not be optimal for all hardware configurations.

Test 13: Lowering Values for Stability

Objective: To test if lowering specific Nanite parameters improves stability.

r.Nanite.AllowTessellation=1 ; [default: 0]
r.Nanite.Tessellation=1 ; [default: 0]
r.Nanite.MaxVisibleClusters=131072; [default: 4194304]
r.Nanite.MaxCandidateClusters=4194304; [default: 16777216]
r.Nanite.MaxNodes=65536; [default: 2097152]

Results: 1 out of 5 tests resulted in crashes.
Consensus: Better stability achieved, indicating that lowering these parameters can significantly reduce crashes while maintaining functionality. It still may be potentially due to statistical variance.
EDIT 1: While initial tests showed improved stability, further testing revealed consistent crashes, indicating that the improvements were likely due to statistical variance.

Conclusion

Lowering specific Nanite parameters improved stability maintaining tessellation functionality. This suggests that fine-tuning these settings is crucial for achieving optimal performance and stability when using the experimental Nanite tessellation feature in Unreal Engine 5.4.

EDIT 1: Although initial results suggested that lowering specific Nanite parameters improved stability while maintaining tessellation functionality, further testing disproved this, revealing the improvement was a statistical anomaly. Regular crashes resumed, showing that these adjustments did not provide a reliable solution.

Further testing and adjustments are still necessary as more updates and insights become available.

I hope this helps others who are facing similar issues. If you have any additional suggestions or solutions, please share them!

Thank you in advance!
Thomas Winged

2 Likes
3

Hi everyone :blush:,

After extensive testing and troubleshooting, I finally found a solution that works, at least on the test map I used to isolate the problem. Here’s a detailed breakdown of my findings and solution:

Steps Taken and Findings:

  1. Initial Hypothesis: Landscape Tessellation

    • Initially, I removed all meshes from the map and kept only the landscape with tessellation enabled. Surprisingly, the crashes stopped. This indicated that the issue might not be with the landscape tessellation itself but rather with specific meshes or materials.

  2. Isolating the Problematic Meshes

    • I reverted the changes and started removing actors and meshes from the scene one by one until the crashes stopped. This process helped isolate the problematic elements.

  3. Identifying the Culprit

    • After minimizing the scene, I noticed certain trees with a shader error: “[SM6] Expression is part of a cycle. Please make sure the material graph is acyclic.”
      1720159545-plPKVJahQP
      1720159545-plPKVJahQP1416×515 243 KB

      1720159592-69Uqs68u0g
      1720159592-69Uqs68u0g1416×515 287 KB
    • Upon opening the material, I found the culprit causing the issue. The funny thing is that nobody in the project touched this material for ages.
      1720159656-kDg0EOPFCt
    • I fixed the error in the shader, and the scene stopped crashing.
      1720159705-88qDWdZ7CJ
      1720159705-88qDWdZ7CJ1412×513 232 KB
    • To ensure the fix was reliable, I reopened and closed the editor 10 times without any crashes, ruling out statistical anomalies.

  4. Validating the Solution

    • I reverted to the original scene setup before removing elements and applied the same shader fix. The crashes ceased entirely, and further testing (10x editor open/close cycles) confirmed stability.

Conclusion:

It appears that a shader error on the tree leaves, combined with enabled landscape tessellation, was causing the frequent crashes. While this solution worked for my test map, it may not be a universal fix for all projects. The takeaway here is to maintain clean and error-free materials in your project.

Automating Material Error Detection:

To prevent similar issues in the future, I wrote a quick Python script to recompile all materials in the project. This script helps identify any materials with errors, which can then be fixed to enhance overall project stability.

import unreal

asset_registry = unreal.AssetRegistryHelpers.get_asset_registry()
material_class_path = unreal.TopLevelAssetPath("/Script/Engine.Material")
assets_data = asset_registry.get_assets_by_path("/Game/", recursive=True, include_only_on_disk_assets=False)

for asset_data in assets_data:
    if asset_data.asset_class_path.package_name == material_class_path.package_name and asset_data.asset_class_path.asset_name == material_class_path.asset_name:
        material = unreal.EditorAssetLibrary.load_asset(asset_data.package_name)
        if material:
            unreal.MaterialEditingLibrary.recompile_material(material)

Running this script might take some time for large projects, but it’s worth it to catch and fix any material errors that could cause crashes.

I hope this helps anyone facing similar issues. If you have any further suggestions or solutions, please share them!

Thank you in advance!
Thomas Winged

2 Likes
4

This is an amazingly helpful writeup, thanks Winged!

Could you provide some more guidance on how to execute your python script for someone that has not done one?

1 Like
5

Sure, I’d be happy to help! Here’s a basic guide on how to execute a Python script within the Unreal Engine. You can find more detailed information in this documentation: Unreal Engine Scripting the Unreal Editor using Python.

  1. Save the Script: First, save the Python script to a file with a .py extension. For example, you might save it as my_script.py.

  2. Open Unreal Engine: Launch Unreal Engine and open your project.

  3. Enable Python Support in Unreal Engine:

    • Go to Edit → Plugins.
    • Search for “Python” in the search bar.
    • Enable the “Editor Scripting Utilities” and “Python Editor Script Plugin”.
    • You might need to restart Unreal Engine for the changes to take effect.

  4. Execute the Script:

    • Open the Window → Developer Tools → Output Log to see any output or errors from your script.
    • Go to File → Execute Python Script.
    • In the file dialog that appears, navigate to your saved script (my_script.py) and select it.

If you have any specific questions or run into issues, feel free to ask!

2 Likes
6

You are, a legend! <3

Script worked wonders. A warning to those using it though on large projects (I had ~1500 materials), it will ask you to re-save them all even if they are ‘OK’. It took about ~30m to parse them all.

1 Like
7

I’m curious if you have version control enabled. In my situation, I don’t see any of these dialog windows; all my materials are compiled and waiting in Save Pending.

8

Yes, I have perforce VC enabled

9

Try disabling version control while scanning materials. This way, it won’t ask for permission to check out the asset every time.

10

It’s not a big deal; annoying to have to resave all material assets but happy to have all the materials ‘up-to-date’, so to speak :slight_smile:

11

I encountered this issue in 5.4 and found the cause was that a material would somehow have tessellation enabled on its shader permutation when it shouldn’t. This would then fail to find a corresponding shader variant since the material didn’t have tessellation/displacement enabled. Since it should only load fixed materials in the function where the crash occurs I simply disabled tessellation on the shader permutation before it tries to find the shader (since it should never be on) and that seems to have fixed things.

1 Like
12

ive got this same issue in ue 5.4.3

for me it crashes the first 2 times i launch it… then finally works on the third launch. Did this on 4 different computers.

The problems began when i started messing with DefaultEngine.ini

Originally i thought my problems had to do with one of hte following… but seems like i was wrong… Tessellation never gave me issues until i started changing things up so i dont think it has anything to do with tessellation.

;r.Lumen.SampleFog=0
;r.VolumetricFog.Emissive=0
;r.OptimizedWPO=1
;r.OptimizedWPO.AffectNonNaniteShaderSelection=1
;r.Nanite.RasterIndirectionMultiplier=1.1

here are my settings for the actual file…

r.ReflectionMethod=1
r.GenerateMeshDistanceFields=True
r.DynamicGlobalIlluminationMethod=1
r.Mobile.EnableNoPrecomputedLightingCSMShader=1
r.DefaultFeature.AutoExposure.ExtendDefaultLuminanceRange=True
r.DefaultFeature.AutoExposure.ExtendDefaultLuminanceRange=True

r.Shadow.Virtual.Enable=1

r.DefaultFeature.LocalExposure.HighlightContrastScale=0.8
r.DefaultFeature.LocalExposure.ShadowContrastScale=0.8

r.SkinCache.CompileShaders=True

r.RayTracing=True

r.VirtualTextures=True
r.VirtualTexturedLightmaps=False
r.VT.AnisotropicFiltering=True
bEnableVirtualTextureOpacityMask=False
;(Default 16) A higher feedback factor increases the rate at which the system gathers this feedback, potentially improving texture streaming responsiveness at the cost of higher processing demands.
r.vt.FeedbackFactor=16

r.EarlyZPass=3

r.Lumen.HardwareRayTracing=False
r.Lumen.HardwareRayTracing.LightingMode=0
r.Lumen.Reflections.HardwareRayTracing.Translucent.Refraction.EnableForProject=False
r.Lumen.TraceMeshSDFs=0
;Leave this off for performance… setting this to 1 may make GI look better but at a cost.
r.Lumen.TraceMeshSDFs=0

r.DefaultFeature.AutoExposure=True

r.DefaultFeature.AmbientOcclusionStaticFraction=False

r.Nanite.AllowSplineMeshes=1
r.MeshStreaming=True

r.AllowStaticLighting=False

r.Nanite.AllowTessellation=1

r.Nanite.Tessellation=1

r.SeparateTranslucencyScreenPercentage=66

r.Shadow.CSMCaching=True
r.MinScreenRadiusForCSMDepth=0.03

r.Mobile.EnableStaticAndCSMShadowReceivers=False

r.DefaultFeature.AutoExposure.ExtendDefaultLuminanceRange=True

r.GPUSkin.Support16BitBoneIndex=True
;False is default for r.GPUSkin.UnlimitedBoneInfluences… makes limit of bone influence per vertex 4… which should be enough.
r.GPUSkin.UnlimitedBoneInfluences=False

SkeletalMesh.UseExperimentalChunking=1

;r.Lumen.SampleFog=0
;default is 1… i dont see a difference honestly… supposedly its so emissive materials effect volumetric fog
;r.VolumetricFog.Emissive=0
;not sure if this command even works anymore… but i did also find r.OptimizedWPO.AffectNonNaniteShaderSelection which may be worth while for non nanite meshes.
r.OptimizedWPO=1
r.OptimizedWPO.AffectNonNaniteShaderSelection=1

r.Nanite.RasterIndirectionMultiplier=1.1

r.SupportStationarySkylight=False

r.Shadow.Virtual.UseFarShadowCulling

"

Assertion failed: !RasterizerPass.ClusterComputeShader.IsNull() [File:D:\build++UE5\Sync\Engine\Source\Runtime\Renderer\Private\Nanite\NaniteCullRaster.cpp] [Line: 3915]

0x00007ffc7e287611 UnrealEditor-Renderer.dll!UnknownFunction
0x00007ffc7e28cac0 UnrealEditor-Renderer.dll!UnknownFunction
0x00007ffc7e288ac2 UnrealEditor-Renderer.dll!UnknownFunction
0x00007ffc7e2d9679 UnrealEditor-Renderer.dll!UnknownFunction
0x00007ffc7db2ae3e UnrealEditor-Renderer.dll!UnknownFunction
0x00007ffc7dae4804 UnrealEditor-Renderer.dll!UnknownFunction
0x00007ffc9cfb24c5 UnrealEditor-Core.dll!UnknownFunction
0x00007ffc9cfb21ff UnrealEditor-Core.dll!UnknownFunction
0x00007ffc9cfda155 UnrealEditor-Core.dll!UnknownFunction
0x00007ffc9cfa23d0 UnrealEditor-Core.dll!UnknownFunction
0x00007ffc9d1a7c93 UnrealEditor-Core.dll!UnknownFunction
0x00007ffc9d60c93d UnrealEditor-Core.dll!UnknownFunction
0x00007ffc9d6002b7 UnrealEditor-Core.dll!UnknownFunction
0x00007ffd8372257d KERNEL32.DLL!UnknownFunction

Crash in runnable thread Background Worker #5
Crash in runnable thread Foreground Worker #0
"

13

Used some of this for some debug, many thanks!

14

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.