Frequent crashes with Nanite Tessellation in UE 5.4.2 (Assertion failed: !RasterizerPass.ClusterComputeShader.IsNull())

Hi everyone :slight_smile: ,

Since upgrading to UE 5.4.2 and enabling tessellation in my project, as well as converting my landscape to Nanite, I’ve been experiencing frequent crashes. The crash details are as follows:

Runnable thread Foreground Worker #1 crashed.
=== Critical ===
Assertion failed: !RasterizerPass.ClusterComputeShader.IsNull() [File:D:\build\++UE5\Sync\Engine\Source\Runtime\Renderer\Private\Nanite\NaniteCullRaster.cpp] [Line: 3905] 
[Callstack] 0x00007ffb38427601 UnrealEditor-Renderer.dll!`FPixelShaderUtils::AddRasterizeToRectsPass<FNaniteMarkStencilPS,FNaniteMarkStencilRectsParameters>'::`2'::<lambda_1>::operator()() [D:\build\++UE5\Sync\Engine\Source\Runtime\Renderer\Private\Nanite\NaniteCullRaster.cpp:3905]
[Callstack] 0x00007ffb3842cab0 UnrealEditor-Renderer.dll!`FPixelShaderUtils::AddRasterizeToRectsPass<FNaniteMarkStencilPS,FNaniteMarkStencilRectsParameters>'::`2'::<lambda_1>::operator()() [D:\build\++UE5\Sync\Engine\Source\Runtime\Renderer\Private\Nanite\NaniteCullRaster.cpp:4051]
[Callstack] 0x00007ffb38428ab2 UnrealEditor-Renderer.dll!`FPixelShaderUtils::AddRasterizeToRectsPass<FNaniteMarkStencilPS,FNaniteMarkStencilRectsParameters>'::`2'::<lambda_1>::operator()() [D:\build\++UE5\Sync\Engine\Source\Runtime\Renderer\Private\Nanite\NaniteCullRaster.cpp:4124]
[Callstack] 0x00007ffb38479669 UnrealEditor-Renderer.dll!FRHICommand<FRHICommandPollOcclusionQueries,FRHICommandPollOcclusionQueriesString2138>::ExecuteAndDestruct() [D:\build\++UE5\Sync\Engine\Source\Runtime\Core\Public\Tasks\TaskPrivate.h:855]
[Callstack] 0x00007ffb37ccae3e UnrealEditor-Renderer.dll!UE::Tasks::Private::FTaskBase::TryExecuteTask() [D:\build\++UE5\Sync\Engine\Source\Runtime\Core\Public\Tasks\TaskPrivate.h:504]
[Callstack] 0x00007ffb37c84804 UnrealEditor-Renderer.dll!LowLevelTasks::TTaskDelegate<LowLevelTasks::FTask * __ptr64 __cdecl(bool),48>::TTaskDelegateImpl<`LowLevelTasks::FTask::Init<`UE::Tasks::Private::FTaskBase::Init'::`2'::<lambda_1> >'::`13'::<lambda_1>,0>::CallAndMove() [D:\build\++UE5\Sync\Engine\Source\Runtime\Core\Public\Async\Fundamental\TaskDelegate.h:171]
[Callstack] 0x00007ffba2a124c5 UnrealEditor-Core.dll!LowLevelTasks::FTask::ExecuteTask() [D:\build\++UE5\Sync\Engine\Source\Runtime\Core\Public\Async\Fundamental\Task.h:627]
[Callstack] 0x00007ffba2a121ff UnrealEditor-Core.dll!LowLevelTasks::FScheduler::ExecuteTask() [D:\build\++UE5\Sync\Engine\Source\Runtime\Core\Private\Async\Fundamental\Scheduler.cpp:155]
[Callstack] 0x00007ffba2a3a155 UnrealEditor-Core.dll!LowLevelTasks::FScheduler::WorkerMain() [D:\build\++UE5\Sync\Engine\Source\Runtime\Core\Private\Async\Fundamental\Scheduler.cpp:397]
[Callstack] 0x00007ffba2a023d0 UnrealEditor-Core.dll!UE::Core::Private::Function::TFunctionRefCaller<`LowLevelTasks::FScheduler::CreateWorker'::`2'::<lambda_1>,void __cdecl(void)>::Call() [D:\build\++UE5\Sync\Engine\Source\Runtime\Core\Public\Templates\Function.h:406]
[Callstack] 0x00007ffba2c07c93 UnrealEditor-Core.dll!FThreadImpl::Run() [D:\build\++UE5\Sync\Engine\Source\Runtime\Core\Private\HAL\Thread.cpp:69]
[Callstack] 0x00007ffba306c93d UnrealEditor-Core.dll!FRunnableThreadWin::Run() [D:\build\++UE5\Sync\Engine\Source\Runtime\Core\Private\Windows\WindowsRunnableThread.cpp:149]
[Callstack] 0x00007ffba30602b7 UnrealEditor-Core.dll!FRunnableThreadWin::GuardedRun() [D:\build\++UE5\Sync\Engine\Source\Runtime\Core\Private\Windows\WindowsRunnableThread.cpp:79]
[Callstack] 0x00007ffc5ec7257d KERNEL32.DLL!UnknownFunction []

Turning off Nanite tessellation resolves the issue, but I would prefer not to disable this awesome feature (I’ve missed it since UE4).

The crashes occur when I open my world partition map (happens the moment the map opens) or when I run the packaged game (again, right when the level opens). It does not crash every time, but rather occasionally, and without any visible pattern.

I’ve found that tweaking the following values can somewhat mitigate the frequency of crashes, but I haven’t been able to eliminate them completely:

r.Nanite.MaxVisibleClusters=10485760 ; [default: 4194304]
r.Nanite.MaxCandidateClusters=16777216 ; [default: 16777216] Maximum number of Nanite clusters before cluster culling.
r.Nanite.MaxNodes=10485760 ; [default: 2097152]

EDIT 1: See my second post for a more detailed breakdown of conducted tests.

One of my observations is that setting r.Nanite.DicingRate to 1 in stat NaniteStreaming results in the fewest Page Requests, but it significantly increases the number of clusters. On the other hand, when I set it to 4, the visuals look fine and there are fewer clusters, but the number of Page Requests goes up. How should I interpret this?

Lastly, is it normal for the Nanite Triangles debug view to be this dense, or could there be something wrong with it?

Has anyone else experienced this issue or found a solution? Any help or suggestions would be greatly appreciated!

Thank you in advance!
Thomas Winged

Hi everyone :blush:,

Previously, I attempted to adjust various parameters to reduce the crashes, but I couldn’t eliminate them entirely. So, I decided to spend more time conducting a detailed analysis. Below, I present the results of the tests I conducted:

Initially, the tests showed some improvement, but it turned out to be a statistical error, as subsequent runs consistently resulted in crashes. Unfortunately, none of the commands improved the situation in a reliable manner.

Detailed Breakdown of Tests Conducted to Enable and Stabilize Nanite Tessellation

Test 1: Enabling Basic Nanite Tessellation

Objective: To activate Nanite tessellation and evaluate stability.

r.Nanite.AllowTessellation=1 ; [default: 0] Enables Nanite displacement
r.Nanite.Tessellation=1 ; [default: 0] Activates Nanite tessellation

Results: 4 out of 5 tests resulted in crashes.
Consensus: Nanite tessellation is active but highly unstable, indicating potential issues with the experimental feature’s initial implementation.

Test 2: Disabling Tessellation

Objective: To determine if disabling tessellation improves stability.

r.Nanite.AllowTessellation=0 ; [default: 0]
r.Nanite.Tessellation=0 ; [default: 0]

Results: 0 out of 5 tests resulted in crashes.
Consensus: Stability achieved without tessellation, confirming that the crashes are related to the tessellation feature.

Test 3: Increasing Nanite Streaming Memory

Objective: To test if increasing the streaming memory pool size reduces crashes.

r.Nanite.AllowTessellation=1 ; [default: 0]
r.Nanite.Tessellation=1 ; [default: 0]
r.Nanite.Streaming.StreamingPoolSize=1024 ; [default: 512] Increases streaming pool size to 1024 MB.

Results: 2 out of 5 tests resulted in crashes.
Consensus: Increased memory appears to reduce crash frequency, suggesting that memory allocation might be a factor in stability.

Test 4: Further Increasing Streaming Memory

Objective: To test the effects of further increasing the streaming memory pool size.

r.Nanite.AllowTessellation=1 ; [default: 0]
r.Nanite.Tessellation=1 ; [default: 0]
r.Nanite.Streaming.StreamingPoolSize=2048 ; [default: 512]

Critical Error:

Streaming pool size (2048MB) must be smaller than the maximum allocation supported by the graphics hardware (2048MB).

Consensus: The maximum supported allocation size was reached, causing a critical error. This suggests a hardware limitation.

Test 5: Intermediate Streaming Memory Size

Objective: To find a balanced streaming memory pool size that might improve stability.

r.Nanite.AllowTessellation=1 ; [default: 0]
r.Nanite.Tessellation=1 ; [default: 0]
r.Nanite.Streaming.StreamingPoolSize=1536 ; [default: 512]

Results: 2 out of 5 tests resulted in crashes.
Consensus: No significant improvement, potentially due to statistical variance.

Test 6: Disabling Specific Nanite Features

Objective: To identify specific Nanite features that might be causing instability.

r.Nanite.AllowTessellation=1
r.Nanite.Tessellation=1
r.Nanite.AsyncRasterization=0 ; [default: 1]
r.Nanite.MeshShaderRasterization=0 ; [default: 1]
r.Nanite.PrimShaderRasterization=0 ; [default: 1]
r.Nanite.Streaming.AsyncCompute=0 ; [default: 1]
r.Nanite.ComputeRasterization=0 ; [default: 1]

Results: 3 out of 5 tests resulted in crashes. Landscape disappeared.
Notes: Crashes persist, and landscape issues are likely due to enabled tessellation.

Test 7: Disabling Programmable Rasterization

Objective: To test if disabling programmable rasterization improves stability.

r.Nanite.AllowTessellation=1
r.Nanite.Tessellation=1
r.Nanite.ProgrammableRaster=0 ; [default: 1]

Results: 0 out of 5 tests resulted in crashes.
Notes: Crashes stopped, but landscape has no masked alpha on masked materials.

Test 8: Disabling Async Rasterization

Objective: To check if disabling async rasterization helps with stability while keeping programmable rasterization enabled.

r.Nanite.AllowTessellation=1
r.Nanite.Tessellation=1
r.Nanite.AsyncRasterization=0 ; [default: 1]

Results: 3 out of 5 tests resulted in crashes.
Notes: Crash frequency remained unchanged.

Test 9: Forcing Shader Bundle Dispatch Emulation

Objective: To evaluate the impact of forcing shader bundle dispatch emulation on stability.

r.Nanite.AllowTessellation=1
r.Nanite.Tessellation=1
r.Nanite.Bundle.Emulation=1 ; [default: 0]

Results: 3 out of 5 tests resulted in crashes.
Notes: Crash frequency remained unchanged.

Test 10: Reducing Nanite Clusters

Objective: To test if reducing the count of Nanite clusters by controlling tessellation dicing and distance density improves stability.

r.Nanite.AllowTessellation=1
r.Nanite.Tessellation=1
r.Nanite.MaxPixelsPerEdge=2 ; [default: 1]
r.Nanite.DicingRate=8 ; [default: 2]

Results: 3 out of 5 tests resulted in crashes.
Notes: Crash frequency remained unchanged.

Test 11: Increasing Nanite Clusters

Objective: To test if increasing the count of Nanite clusters by controlling tessellation dicing affects stability.

r.Nanite.AllowTessellation=1
r.Nanite.Tessellation=1
r.Nanite.DicingRate=1 ; [default: 2]

Results: 3 out of 5 tests resulted in crashes.
Notes: Crash frequency remained unchanged.

Test 12: Using Epic’s Proposed Values

Objective: To evaluate the stability of Epic’s proposed settings for Nanite, found in ElectricDreamsEnv project.

r.Nanite.AllowTessellation=1 ; [default: 0]
r.Nanite.Tessellation=1 ; [default: 0]
r.Nanite.MaxVisibleClusters=10485760 ; [default: 4194304]
r.Nanite.MaxCandidateClusters=16777216 ; [default: 16777216]
r.Nanite.MaxNodes=10485760 ; [default: 2097152]

Results: 4 out of 5 tests resulted in crashes.
Consensus: No significant improvement in stability, indicating that these values might not be optimal for all hardware configurations.

Test 13: Lowering Values for Stability

Objective: To test if lowering specific Nanite parameters improves stability.

r.Nanite.AllowTessellation=1 ; [default: 0]
r.Nanite.Tessellation=1 ; [default: 0]
r.Nanite.MaxVisibleClusters=131072; [default: 4194304]
r.Nanite.MaxCandidateClusters=4194304; [default: 16777216]
r.Nanite.MaxNodes=65536; [default: 2097152]

Results: 1 out of 5 tests resulted in crashes.
Consensus: Better stability achieved, indicating that lowering these parameters can significantly reduce crashes while maintaining functionality. It still may be potentially due to statistical variance.
EDIT 1: While initial tests showed improved stability, further testing revealed consistent crashes, indicating that the improvements were likely due to statistical variance.

Conclusion

Lowering specific Nanite parameters improved stability maintaining tessellation functionality. This suggests that fine-tuning these settings is crucial for achieving optimal performance and stability when using the experimental Nanite tessellation feature in Unreal Engine 5.4.

EDIT 1: Although initial results suggested that lowering specific Nanite parameters improved stability while maintaining tessellation functionality, further testing disproved this, revealing the improvement was a statistical anomaly. Regular crashes resumed, showing that these adjustments did not provide a reliable solution.

Further testing and adjustments are still necessary as more updates and insights become available.

I hope this helps others who are facing similar issues. If you have any additional suggestions or solutions, please share them!

Thank you in advance!
Thomas Winged

Hi everyone :blush:,

After extensive testing and troubleshooting, I finally found a solution that works, at least on the test map I used to isolate the problem. Here’s a detailed breakdown of my findings and solution:

Steps Taken and Findings:

  1. Initial Hypothesis: Landscape Tessellation

    • Initially, I removed all meshes from the map and kept only the landscape with tessellation enabled. Surprisingly, the crashes stopped. This indicated that the issue might not be with the landscape tessellation itself but rather with specific meshes or materials.
  2. Isolating the Problematic Meshes

    • I reverted the changes and started removing actors and meshes from the scene one by one until the crashes stopped. This process helped isolate the problematic elements.
  3. Identifying the Culprit

    • After minimizing the scene, I noticed certain trees with a shader error: “[SM6] Expression is part of a cycle. Please make sure the material graph is acyclic.”

    • Upon opening the material, I found the culprit causing the issue. The funny thing is that nobody in the project touched this material for ages.
      1720159656-kDg0EOPFCt
    • I fixed the error in the shader, and the scene stopped crashing.
    • To ensure the fix was reliable, I reopened and closed the editor 10 times without any crashes, ruling out statistical anomalies.
  4. Validating the Solution

    • I reverted to the original scene setup before removing elements and applied the same shader fix. The crashes ceased entirely, and further testing (10x editor open/close cycles) confirmed stability.

Conclusion:

It appears that a shader error on the tree leaves, combined with enabled landscape tessellation, was causing the frequent crashes. While this solution worked for my test map, it may not be a universal fix for all projects. The takeaway here is to maintain clean and error-free materials in your project.

Automating Material Error Detection:

To prevent similar issues in the future, I wrote a quick Python script to recompile all materials in the project. This script helps identify any materials with errors, which can then be fixed to enhance overall project stability.

import unreal

asset_registry = unreal.AssetRegistryHelpers.get_asset_registry()
material_class_path = unreal.TopLevelAssetPath("/Script/Engine.Material")
assets_data = asset_registry.get_assets_by_path("/Game/", recursive=True, include_only_on_disk_assets=False)

for asset_data in assets_data:
    if asset_data.asset_class_path.package_name == material_class_path.package_name and asset_data.asset_class_path.asset_name == material_class_path.asset_name:
        material = unreal.EditorAssetLibrary.load_asset(asset_data.package_name)
        if material:
            unreal.MaterialEditingLibrary.recompile_material(material)

Running this script might take some time for large projects, but it’s worth it to catch and fix any material errors that could cause crashes.

I hope this helps anyone facing similar issues. If you have any further suggestions or solutions, please share them!

Thank you in advance!
Thomas Winged

1 Like

This is an amazingly helpful writeup, thanks Winged!

Could you provide some more guidance on how to execute your python script for someone that has not done one?

Sure, I’d be happy to help! Here’s a basic guide on how to execute a Python script within the Unreal Engine. You can find more detailed information in this documentation: Unreal Engine Scripting the Unreal Editor using Python.

  1. Save the Script: First, save the Python script to a file with a .py extension. For example, you might save it as my_script.py.

  2. Open Unreal Engine: Launch Unreal Engine and open your project.

  3. Enable Python Support in Unreal Engine:

    • Go to EditPlugins.
    • Search for “Python” in the search bar.
    • Enable the “Editor Scripting Utilities” and “Python Editor Script Plugin”.
    • You might need to restart Unreal Engine for the changes to take effect.
  4. Execute the Script:

    • Open the WindowDeveloper ToolsOutput Log to see any output or errors from your script.
    • Go to FileExecute Python Script.
    • In the file dialog that appears, navigate to your saved script (my_script.py) and select it.

If you have any specific questions or run into issues, feel free to ask!

1 Like

You are, a legend! <3

Script worked wonders. A warning to those using it though on large projects (I had ~1500 materials), it will ask you to re-save them all even if they are ‘OK’. It took about ~30m to parse them all.