UE5.2 crash on V100 GPU due to nanite

Environment:

  • UE5.2.1

  • Ubuntu20.04

  • GPU: Tesla V100, Driver: 525.125.06

My packaged game will crash when startup. I suspect that the problem is caused by nanite because when I disabled all nanite asset, the issue disappeared.

Error info:

[2023.08.04-02.36.25:603][179]LogRHI: Timed out while waiting for GPU to catch up on occlusion results. (0.5 s)
[2023.08.04-02.36.26:103][179]LogRHI: Timed out while waiting for GPU to catch up on occlusion results. (0.5 s)
[2023.08.04-02.36.26:603][179]LogRHI: Timed out while waiting for GPU to catch up on occlusion results. (0.5 s)
[2023.08.04-02.36.27:103][179]LogRHI: Timed out while waiting for GPU to catch up on occlusion results. (0.5 s)
[2023.08.04-02.36.27:603][179]LogRHI: Timed out while waiting for GPU to catch up on occlusion results. (0.5 s)
[2023.08.04-02.36.28:103][179]LogRHI: Timed out while waiting for GPU to catch up on occlusion results. (0.5 s)
[2023.08.04-02.36.28:603][179]LogRHI: Timed out while waiting for GPU to catch up on occlusion results. (0.5 s)
Assertion failed: bSuccess [File:./Runtime/VulkanRHI/Private/VulkanCommandBuffer.cpp] [Line: 503] 

[2023.08.04-02.36.28:649][179]LogCore: Error: appError called: Assertion failed: bSuccess [File:./Runtime/VulkanRHI/Private/VulkanCommandBuffer.cpp] [Line: 503] 


Signal 11 caught.
Malloc Size=262146 LargeMemoryPoolOffset=262162 
CommonUnixCrashHandler: Signal=11
[2023.08.04-02.36.28:669][179]LogCore: === Critical error: ===
Unhandled Exception: SIGSEGV: invalid attempt to write memory at address 0x0000000000000003

[2023.08.04-02.36.28:670][179]LogCore: Assertion failed: bSuccess [File:./Runtime/VulkanRHI/Private/VulkanCommandBuffer.cpp] [Line: 503] 

0x0000000009afa890 SolarEngineUE5!FVulkanCommandBufferManager::WaitForCmdBuffer(FVulkanCmdBuffer*, float) [/home/caros/SSD/Documents/UnrealEngine5.2.1/UnrealEngine-release/Engine/Source/./Runtime/VulkanRHI/Private/VulkanCommandBuffer.cpp:503]
0x0000000009bc258f SolarEngineUE5!FVulkanViewport::WaitForFrameEventCompletion() [/home/caros/SSD/Documents/UnrealEngine5.2.1/UnrealEngine-release/Engine/Source/./Runtime/VulkanRHI/Private/VulkanViewport.cpp:275]
0x0000000009bc5aad SolarEngineUE5!FVulkanViewport::Present(FVulkanCommandListContext*, FVulkanCmdBuffer*, FVulkanQueue*, FVulkanQueue*, bool) [/home/caros/SSD/Documents/UnrealEngine5.2.1/UnrealEngine-release/Engine/Source/./Runtime/VulkanRHI/Private/VulkanViewport.cpp:965]
0x0000000009b868b0 SolarEngineUE5!FVulkanCommandListContext::RHIEndDrawingViewport(FRHIViewport*, bool, bool) [/home/caros/SSD/Documents/UnrealEngine5.2.1/UnrealEngine-release/Engine/Source/./Runtime/VulkanRHI/Private/VulkanRHI.cpp:1007]
0x0000000009cf9c51 SolarEngineUE5!FRHICommand<FRHICommandEndDrawingViewport, FRHICommandEndDrawingViewportString2071>::ExecuteAndDestruct(FRHICommandListBase&, FRHICommandListDebugContext&) [/home/caros/SSD/Documents/UnrealEngine5.2.1/UnrealEngine-release/Engine/Source/Runtime/RHI/Public/RHICommandList.h:1105]
0x0000000009cb4b6e SolarEngineUE5!FRHICommandListBase::Execute(TRHIPipelineArray<IRHIComputeContext*>&, FRHICommandListBase::FPersistentState::FGPUStats*) [/home/caros/SSD/Documents/UnrealEngine5.2.1/UnrealEngine-release/Engine/Source/./Runtime/RHI/Private/RHICommandList.cpp:592]
0x0000000009d11c5e SolarEngineUE5!UE::Core::Private::Function::TFunctionRefCaller<FRHICommandListImmediate::ExecuteAndReset()::$_124, void ()>::Call(void*) [/home/caros/SSD/Documents/UnrealEngine5.2.1/UnrealEngine-release/Engine/Source/Runtime/Core/Public/Templates/Function.h:474]
0x0000000004508fda SolarEngineUE5!TGraphTask<TFunctionGraphTaskImpl<void (), (ESubsequentsMode::Type)0> >::ExecuteTask(TArray<FBaseGraphTask*, TSizedDefaultAllocator<32> >&, ENamedThreads::Type, bool) [/home/caros/SSD/Documents/UnrealEngine5.2.1/UnrealEngine-release/Engine/Source/Runtime/Core/Public/Async/TaskGraphInterfaces.h:1310]
0x000000000674eb95 SolarEngineUE5!FNamedTaskThread::ProcessTasksNamedThread(int, bool) [/home/caros/SSD/Documents/UnrealEngine5.2.1/UnrealEngine-release/Engine/Source/./Runtime/Core/Private/Async/TaskGraph.cpp:758]
0x000000000674d1f0 SolarEngineUE5!FNamedTaskThread::ProcessTasksUntilQuit(int) [/home/caros/SSD/Documents/UnrealEngine5.2.1/UnrealEngine-release/Engine/Source/./Runtime/Core/Private/Async/TaskGraph.cpp:648]
0x000000000674c0e9 SolarEngineUE5!FTaskGraphCompatibilityImplementation::ProcessThreadUntilRequestReturn(ENamedThreads::Type) [/home/caros/SSD/Documents/UnrealEngine5.2.1/UnrealEngine-release/Engine/Source/./Runtime/Core/Private/Async/TaskGraph.cpp:2069]
0x0000000009dfe5cc SolarEngineUE5!FRHIThread::Run() [/home/caros/SSD/Documents/UnrealEngine5.2.1/UnrealEngine-release/Engine/Source/./Runtime/RenderCore/Private/RenderingThread.cpp:333]
0x0000000006853066 SolarEngineUE5!FRunnableThreadPThread::Run() [/home/caros/SSD/Documents/UnrealEngine5.2.1/UnrealEngine-release/Engine/Source/./Runtime/Core/Private/HAL/PThreadRunnableThread.cpp:25]
0x000000000680ea08 SolarEngineUE5!FRunnableThreadPThread::_ThreadProc(void*) [/home/caros/SSD/Documents/UnrealEngine5.2.1/UnrealEngine-release/Engine/Source/Runtime/Core/Private/HAL/PThreadRunnableThread.h:187]
0x00007fb7f5e21609 libpthread.so.0!UnknownFunction(0x8608)
0x00007fb7f028f133 libc.so.6!clone(+0x42)
2 Likes

I have the same problem, and i get a VK_ERROR_DEVICE_LOST

and use -gpucrashdebugging, i got

[VK_NV_device_diagnostic_checkpoints] 0: Stage VK_PIPELINE_STAGE_TOP_OF_PIPE_BIT Classify Materials
[VK_NV_device_diagnostic_checkpoints] 1: Stage VK_PIPELINE_STAGE_BOTTOM_OF_PIPE_BIT Finalize Materials

I have the same problem, and i get a VK_ERROR_DEVICE_LOST

and use -gpucrashdebugging, i got

[2023.08.17-12.10.18:438][  6]LogVulkanRHI: Error: [VK_NV_device_diagnostic_checkpoints] 0: Stage VK_PIPELINE_STAGE_TOP_OF_PIPE_BIT (0x00000001), Finalize Materials (CRC 0x6ea6b15a)
[2023.08.17-12.10.18:438][  6]LogVulkanRHI: Error: [VK_NV_device_diagnostic_checkpoints] 1: Stage VK_PIPELINE_STAGE_BOTTOM_OF_PIPE_BIT (0x00002000), Classify Materials (CRC 0x2d9c4290)

Did you guys ever fix this? Im experiencing this on Nvidia A10