UE5.2 crash on V100 GPU due to nanite

Environment:

  • UE5.2.1

  • Ubuntu20.04

  • GPU: Tesla V100, Driver: 525.125.06

My packaged game will crash when startup. I suspect that the problem is caused by nanite because when I disabled all nanite asset, the issue disappeared.

Error info:

[2023.08.04-02.36.25:603][179]LogRHI: Timed out while waiting for GPU to catch up on occlusion results. (0.5 s)
[2023.08.04-02.36.26:103][179]LogRHI: Timed out while waiting for GPU to catch up on occlusion results. (0.5 s)
[2023.08.04-02.36.26:603][179]LogRHI: Timed out while waiting for GPU to catch up on occlusion results. (0.5 s)
[2023.08.04-02.36.27:103][179]LogRHI: Timed out while waiting for GPU to catch up on occlusion results. (0.5 s)
[2023.08.04-02.36.27:603][179]LogRHI: Timed out while waiting for GPU to catch up on occlusion results. (0.5 s)
[2023.08.04-02.36.28:103][179]LogRHI: Timed out while waiting for GPU to catch up on occlusion results. (0.5 s)
[2023.08.04-02.36.28:603][179]LogRHI: Timed out while waiting for GPU to catch up on occlusion results. (0.5 s)
Assertion failed: bSuccess [File:./Runtime/VulkanRHI/Private/VulkanCommandBuffer.cpp] [Line: 503] 

[2023.08.04-02.36.28:649][179]LogCore: Error: appError called: Assertion failed: bSuccess [File:./Runtime/VulkanRHI/Private/VulkanCommandBuffer.cpp] [Line: 503] 


Signal 11 caught.
Malloc Size=262146 LargeMemoryPoolOffset=262162 
CommonUnixCrashHandler: Signal=11
[2023.08.04-02.36.28:669][179]LogCore: === Critical error: ===
Unhandled Exception: SIGSEGV: invalid attempt to write memory at address 0x0000000000000003

[2023.08.04-02.36.28:670][179]LogCore: Assertion failed: bSuccess [File:./Runtime/VulkanRHI/Private/VulkanCommandBuffer.cpp] [Line: 503] 

0x0000000009afa890 SolarEngineUE5!FVulkanCommandBufferManager::WaitForCmdBuffer(FVulkanCmdBuffer*, float) [/home/caros/SSD/Documents/UnrealEngine5.2.1/UnrealEngine-release/Engine/Source/./Runtime/VulkanRHI/Private/VulkanCommandBuffer.cpp:503]
0x0000000009bc258f SolarEngineUE5!FVulkanViewport::WaitForFrameEventCompletion() [/home/caros/SSD/Documents/UnrealEngine5.2.1/UnrealEngine-release/Engine/Source/./Runtime/VulkanRHI/Private/VulkanViewport.cpp:275]
0x0000000009bc5aad SolarEngineUE5!FVulkanViewport::Present(FVulkanCommandListContext*, FVulkanCmdBuffer*, FVulkanQueue*, FVulkanQueue*, bool) [/home/caros/SSD/Documents/UnrealEngine5.2.1/UnrealEngine-release/Engine/Source/./Runtime/VulkanRHI/Private/VulkanViewport.cpp:965]
0x0000000009b868b0 SolarEngineUE5!FVulkanCommandListContext::RHIEndDrawingViewport(FRHIViewport*, bool, bool) [/home/caros/SSD/Documents/UnrealEngine5.2.1/UnrealEngine-release/Engine/Source/./Runtime/VulkanRHI/Private/VulkanRHI.cpp:1007]
0x0000000009cf9c51 SolarEngineUE5!FRHICommand<FRHICommandEndDrawingViewport, FRHICommandEndDrawingViewportString2071>::ExecuteAndDestruct(FRHICommandListBase&, FRHICommandListDebugContext&) [/home/caros/SSD/Documents/UnrealEngine5.2.1/UnrealEngine-release/Engine/Source/Runtime/RHI/Public/RHICommandList.h:1105]
0x0000000009cb4b6e SolarEngineUE5!FRHICommandListBase::Execute(TRHIPipelineArray<IRHIComputeContext*>&, FRHICommandListBase::FPersistentState::FGPUStats*) [/home/caros/SSD/Documents/UnrealEngine5.2.1/UnrealEngine-release/Engine/Source/./Runtime/RHI/Private/RHICommandList.cpp:592]
0x0000000009d11c5e SolarEngineUE5!UE::Core::Private::Function::TFunctionRefCaller<FRHICommandListImmediate::ExecuteAndReset()::$_124, void ()>::Call(void*) [/home/caros/SSD/Documents/UnrealEngine5.2.1/UnrealEngine-release/Engine/Source/Runtime/Core/Public/Templates/Function.h:474]
0x0000000004508fda SolarEngineUE5!TGraphTask<TFunctionGraphTaskImpl<void (), (ESubsequentsMode::Type)0> >::ExecuteTask(TArray<FBaseGraphTask*, TSizedDefaultAllocator<32> >&, ENamedThreads::Type, bool) [/home/caros/SSD/Documents/UnrealEngine5.2.1/UnrealEngine-release/Engine/Source/Runtime/Core/Public/Async/TaskGraphInterfaces.h:1310]
0x000000000674eb95 SolarEngineUE5!FNamedTaskThread::ProcessTasksNamedThread(int, bool) [/home/caros/SSD/Documents/UnrealEngine5.2.1/UnrealEngine-release/Engine/Source/./Runtime/Core/Private/Async/TaskGraph.cpp:758]
0x000000000674d1f0 SolarEngineUE5!FNamedTaskThread::ProcessTasksUntilQuit(int) [/home/caros/SSD/Documents/UnrealEngine5.2.1/UnrealEngine-release/Engine/Source/./Runtime/Core/Private/Async/TaskGraph.cpp:648]
0x000000000674c0e9 SolarEngineUE5!FTaskGraphCompatibilityImplementation::ProcessThreadUntilRequestReturn(ENamedThreads::Type) [/home/caros/SSD/Documents/UnrealEngine5.2.1/UnrealEngine-release/Engine/Source/./Runtime/Core/Private/Async/TaskGraph.cpp:2069]
0x0000000009dfe5cc SolarEngineUE5!FRHIThread::Run() [/home/caros/SSD/Documents/UnrealEngine5.2.1/UnrealEngine-release/Engine/Source/./Runtime/RenderCore/Private/RenderingThread.cpp:333]
0x0000000006853066 SolarEngineUE5!FRunnableThreadPThread::Run() [/home/caros/SSD/Documents/UnrealEngine5.2.1/UnrealEngine-release/Engine/Source/./Runtime/Core/Private/HAL/PThreadRunnableThread.cpp:25]
0x000000000680ea08 SolarEngineUE5!FRunnableThreadPThread::_ThreadProc(void*) [/home/caros/SSD/Documents/UnrealEngine5.2.1/UnrealEngine-release/Engine/Source/Runtime/Core/Private/HAL/PThreadRunnableThread.h:187]
0x00007fb7f5e21609 libpthread.so.0!UnknownFunction(0x8608)
0x00007fb7f028f133 libc.so.6!clone(+0x42)
3 Likes

I have the same problem, and i get a VK_ERROR_DEVICE_LOST

and use -gpucrashdebugging, i got

[VK_NV_device_diagnostic_checkpoints] 0: Stage VK_PIPELINE_STAGE_TOP_OF_PIPE_BIT Classify Materials
[VK_NV_device_diagnostic_checkpoints] 1: Stage VK_PIPELINE_STAGE_BOTTOM_OF_PIPE_BIT Finalize Materials

I have the same problem, and i get a VK_ERROR_DEVICE_LOST

and use -gpucrashdebugging, i got

[2023.08.17-12.10.18:438][  6]LogVulkanRHI: Error: [VK_NV_device_diagnostic_checkpoints] 0: Stage VK_PIPELINE_STAGE_TOP_OF_PIPE_BIT (0x00000001), Finalize Materials (CRC 0x6ea6b15a)
[2023.08.17-12.10.18:438][  6]LogVulkanRHI: Error: [VK_NV_device_diagnostic_checkpoints] 1: Stage VK_PIPELINE_STAGE_BOTTOM_OF_PIPE_BIT (0x00002000), Classify Materials (CRC 0x2d9c4290)

Did you guys ever fix this? Im experiencing this on Nvidia A10

Have you fixed it? Any help is appreciated here.

Hi guys.
Tell me, did you find a solution? Ue5.4 Ubuntu 22.04 Nvidia V100