Vulkan crash

Sr.Vlad.m · September 24, 2022, 9:33pm

Hi, so our game works fine in the editor but crash on the mobile after package, can someone help me identify the problem? Iam stuck more then 2 weeks on that…

here is part of the log:

D/UE4: [2022.09.24-20.51.15:769][276]LogVulkanRHI: Error: Result failed, VkResult=-4
D/UE4: at D:/Build/++UE4/Sync/Engine/Source/Runtime/VulkanRHI/Private/VulkanMemory.cpp:4789
D/UE4: with error VK_ERROR_DEVICE_LOST
D/UE4: Fatal error: [File:D:/Build/++UE4/Sync/Engine/Source/Runtime/VulkanRHI/Private/VulkanUtil.cpp] [Line: 958]
D/UE4: Result failed, VkResult=-4
D/UE4: at D:/Build/++UE4/Sync/Engine/Source/Runtime/VulkanRHI/Private/VulkanMemory.cpp:4789
D/UE4: with error VK_ERROR_DEVICE_LOST
D/UE4: [2022.09.24-20.51.15:836][276]Fatal error: [File:D:/Build/++UE4/Sync/Engine/Source/Runtime/VulkanRHI/Private/VulkanUtil.cpp] [Line: 958]
D/UE4: Result failed, VkResult=-4
D/UE4: at D:/Build/++UE4/Sync/Engine/Source/Runtime/VulkanRHI/Private/VulkanMemory.cpp:4789
D/UE4: with error VK_ERROR_DEVICE_LOST
D/UE4: 0x00000000A98A11F0 libUE4.so(0x00000000076D61F0) Unknown []
D/UE4: 0x00000000A98757B0 libUE4.so(0x00000000076AA7B0)!VulkanRHI::VerifyVulkanResult(VkResult, char const*, char const*, unsigned int) []
D/UE4: 0x00000000A9823DA0 libUE4.so(0x0000000007658DA0)!VulkanRHI::FFenceManager::WaitForFence(VulkanRHI::FFence*, unsigned long long) []
D/UE4: 0x00000000A97DB9F8 libUE4.so(0x00000000076109F8)!FVulkanCommandBufferManager::WaitForCmdBuffer(FVulkanCmdBuffer*, float) []
D/UE4: 0x00000000A984BA20 libUE4.so(0x0000000007680A20)!FVulkanDynamicRHI::RHIReadSurfaceFloatData(FRHITexture*, FIntRect, TArray<FFloat16Color, TSizedDefaultAllocator<32> >&, ECubeFace, int, int) []
D/UE4: 0x00000000AB0C4808 libUE4.so(0x0000000008EF9808)!FDynamicRHI::RHIReadSurfaceFloatData_RenderThread(FRHICommandListImmediate&, FRHITexture*, FIntRect, TArray<FFloat16Color, TSizedDefaultAllocator<32> >&, ECubeFace, int, int) []
D/UE4: 0x00000000AC48BCE0 libUE4.so(0x000000000A2C0CE0)!ComputeSingleAverageBrightnessFromCubemap(FRHICommandListImmediate&, ERHIFeatureLevel::Type, int, FSceneRenderTargetItem&) []
D/UE4: 0x00000000AC259B4C libUE4.so(0x000000000A08EB4C)!MobileReflectionEnvironmentCapture::ComputeAverageBrightness(FRHICommandListImmediate&, ERHIFeatureLevel::Type, int, float&) []
D/UE4: 0x00000000AC520078 libUE4.so(0x000000000A355078) Unknown []
D/UE4: 0x00000000A9926E44 libUE4.so(0x000000000775BE44)!FNamedTaskThread::ProcessTasksNamedThread(int, bool) []
D/UE4: 0x00000000A99251B8 libUE4.so(0x000000000775A1B8)!FNamedTaskThread::ProcessTasksUntilQuit(int) []
D/UE4: 0x00000000AB16CE64 libUE4.so(0x0000000008FA1E64)!RenderingThreadMain(FEvent*) []
D/UE4: 0x00000000AB18510C libUE4.so(0x0000000008FBA10C)!FRenderingThread::Run() []
D/UE4: 0x00000000A9A4589C libUE4.so(0x000000000787A89C)!FRunnableThreadPThread::Run() []
D/UE4: 0x00000000A991E644 libUE4.so(0x0000000007753644)!FRunnableThreadPThread::_ThreadProc(void*) []
D/UE4: 0x00000000F1FCCA78 libc.so(0x0000000000080A78) Unknown []
D/UE4: 0x00000000F1F85A36 libc.so(0x0000000000039A36) Unknown []
D/UE4: [2022.09.24-20.51.15:836][276]LogAndroid: Error: === Critical error: ===
D/UE4: [2022.09.24-20.51.15:836][276]LogAndroid: Error:
D/UE4: [2022.09.24-20.51.15:836][276]LogAndroid: Error: Fatal error: [File:D:/Build/++UE4/Sync/Engine/Source/Runtime/VulkanRHI/Private/VulkanUtil.cpp] [Line: 958]
D/UE4: [2022.09.24-20.51.15:836][276]LogAndroid: Error: Result failed, VkResult=-4
D/UE4: [2022.09.24-20.51.15:836][276]LogAndroid: Error: at D:/Build/++UE4/Sync/Engine/Source/Runtime/VulkanRHI/Private/VulkanMemory.cpp:4789
D/UE4: [2022.09.24-20.51.15:836][276]LogAndroid: Error: with error VK_ERROR_DEVICE_LOST
D/UE4: [2022.09.24-20.51.15:836][276]LogAndroid: Error: [Callstack] 0x00000000A98A11F0 libUE4.so(0x00000000076D61F0) Unknown []
D/UE4: [2022.09.24-20.51.15:836][276]LogAndroid: Error: [Callstack] 0x00000000A98757B0 libUE4.so(0x00000000076AA7B0)!VulkanRHI::VerifyVulkanResult(VkResult, char const*, char const*, unsigned int) []
D/UE4: [2022.09.24-20.51.15:836][276]LogAndroid: Error: [Callstack] 0x00000000A9823DA0 libUE4.so(0x0000000007658DA0)!VulkanRHI::FFenceManager::WaitForFence(VulkanRHI::FFence*, unsigned long long) []
D/UE4: [2022.09.24-20.51.15:836][276]LogAndroid: Error: [Callstack] 0x00000000A97DB9F8 libUE4.so(0x00000000076109F8)!FVulkanCommandBufferManager::WaitForCmdBuffer(FVulkanCmdBuffer*, float) []
D/UE4: [2022.09.24-20.51.15:836][276]LogAndroid: Error: [Callstack] 0x00000000A984BA20 libUE4.so(0x0000000007680A20)!FVulkanDynamicRHI::RHIReadSurfaceFloatData(FRHITexture*, FIntRect, TArray<FFloat16Color, TSizedDefaultAllocator<32> >&, ECubeFace, int, int) []
D/UE4: [2022.09.24-20.51.15:837][276]LogAndroid: Error: [Callstack] 0x00000000AB0C4808 libUE4.so(0x0000000008EF9808)!FDynamicRHI::RHIReadSurfaceFloatData_RenderThread(FRHICommandListImmediate&, FRHITexture*, FIntRect, TArray<FFloat16Color, TSizedDefaultAllocator<32> >&, ECubeFace, int, int) []
D/UE4: [2022.09.24-20.51.15:837][276]LogAndroid: Error: [Callstack] 0x00000000AC48BCE0 libUE4.so(0x000000000A2C0CE0)!ComputeSingleAverageBrightnessFromCubemap(FRHICommandListImmediate&, ERHIFeatureLevel::Type, int, FSceneRenderTargetItem&) []
D/UE4: [2022.09.24-20.51.15:837][276]LogAndroid: Error: [Callstack] 0x00000000AC259B4C libUE4.so(0x000000000A08EB4C)!MobileReflectionEnvironmentCapture::ComputeAverageBrightness(FRHICommandListImmediate&, ERHIFeatureLevel::Type, int, float&) []
D/UE4: [2022.09.24-20.51.15:837][276]LogAndroid: Error: [Callstack] 0x00000000AC520078 libUE4.so(0x000000000A355078) Unknown []
D/UE4: [2022.09.24-20.51.15:837][276]LogAndroid: Error: [Callstack] 0x00000000A9926E44 libUE4.so(0x000000000775BE44)!FNamedTaskThread::ProcessTasksNamedThread(int, bool) []
D/UE4: [2022.09.24-20.51.15:837][276]LogAndroid: Error: [Callstack] 0x00000000A99251B8 libUE4.so(0x000000000775A1B8)!FNamedTaskThread::ProcessTasksUntilQuit(int) []
D/UE4: [2022.09.24-20.51.15:837][276]LogAndroid: Error: [Callstack] 0x00000000AB16CE64 libUE4.so(0x0000000008FA1E64)!RenderingThreadMain(FEvent*) []
D/UE4: [2022.09.24-20.51.15:837][276]LogAndroid: Error: [Callstack] 0x00000000AB18510C libUE4.so(0x0000000008FBA10C)!FRenderingThread::Run() []
D/UE4: [2022.09.24-20.51.15:837][276]LogAndroid: Error: [Callstack] 0x00000000A9A4589C libUE4.so(0x000000000787A89C)!FRunnableThreadPThread::Run() []
D/UE4: [2022.09.24-20.51.15:837][276]LogAndroid: Error: [Callstack] 0x00000000A991E644 libUE4.so(0x0000000007753644)!FRunnableThreadPThread::_ThreadProc(void*) []
D/UE4: [2022.09.24-20.51.15:837][276]LogAndroid: Error: [Callstack] 0x00000000F1FCCA78 libc.so(0x0000000000080A78) Unknown []
D/UE4: [2022.09.24-20.51.15:837][276]LogAndroid: Error: [Callstack] 0x00000000F1F85A36 libc.so(0x0000000000039A36) Unknown []
D/UE4: [2022.09.24-20.51.15:837][276]LogAndroid: Error:
D/UE4: [2022.09.24-20.51.15:837][276]LogAndroid: Error:
D/UE4: [2022.09.24-20.51.15:837][276]LogAndroid: Error:
D/UE4: [2022.09.24-20.51.15:837][276]LogAndroid: Error:
D/UE4: [2022.09.24-20.51.15:848][276]LogExit: Executing StaticShutdownAfterError
D/UE4: [2022.09.24-20.51.15:848][276]LogAndroid: FAndroidMisc::RequestExit(1)
I/ArcaneDefender: System.exit called, status: 0
I/AndroidRuntime: VM exiting with result code 0, cleanup skipped.

SupportiveEntity · September 26, 2022, 12:44am

Hey there @Sr.Vlad.m! Looks like an error happened during some memory cleanup. I’d seen this issue before a while back, but it usually comes down to a few factors.

Either the device itself ran out of memory during an operation, or the defrag script failed in another area. There used to be a workaround if it’s not your device memory.

Thread with the workaround: VK_ERROR_DEVICE_LOST - #2 by Acheta_Games

Hope this helps!

Sr.Vlad.m · September 26, 2022, 6:36am

Thank for your answers, It crush different devices (S10 and S20 Note)

I have seen that thread and tried it out but without any success unfortunately…

SupportiveEntity · September 26, 2022, 7:22am

Hmmm, normally I’d recommend some profiling to verify if for some reason the memory script is spiking it for no reason, but an S20 note shouldn’t be bottoming out as the game launches for most cases.

Does it crash immediately or after a few frames of rendering? Do you ever get to see anything?

Sr.Vlad.m · September 28, 2022, 9:39am

the game starts and the main map loads after playing the level or 2 when you try to get back to main map the game crash. I can provide screen recording if it can be helpful(in direct massage)

Memory report from S10:
Main_Map1-Android-28-13.17.33.memreport (282.0 KB)

SupportiveEntity · September 29, 2022, 4:14am

From the memory dump it doesn’t look like it it’s maxing out on memory, so signs point at an issue with Vulkan’s memory handler. My usual workaround for these things in the meantime is swapping users back to DX11 and verify if it still fails that way, but considering you’re targeting mobile that’s less of a plan.

This only happens during the packaged version correct?

Sr.Vlad.m · October 1, 2022, 6:53pm

Yes it only occurs on packaged version.
is there any other solution? maybe different version of vulkan?

this issue bothering us more then 2 weeks already and whole development is stuck…

dorusoftware · October 1, 2022, 8:21pm

I don’t think vulkan + ue5.0.3 is reliable. probably best to wait for ue5.1.0 maybe they will fix it.

SupportiveEntity · October 1, 2022, 11:37pm

From looking through the issues tracker I couldn’t find a report that matches this description exactly. I’ve been seeing issues with people trying to utilize some of the new rendering tools and Vulkan, but not back on 4.27. All of the workarounds I have available aren’t viable for mobile devices.

So I think it’s probably best to report it as a bug and keep an eye out for workarounds.

Sr.Vlad.m · October 3, 2022, 1:09pm

we found that Niagara system were causing this strange error, after we replaced it the crashes stopped.

I don’t know why the crashes were happing only on the mobile after package but Iam glad we found a way to overcome it.

SupportiveEntity · October 4, 2022, 3:39am

Thanks for the update! I’ve been seeing some consistent issues with Niagara breaking some things with the other renderers. If I may ask, did you remove the entire system or just tone it down?

Sr.Vlad.m · October 4, 2022, 7:12am

for now we completely replaced the system so we could keep working but we will try to identify the issue further maybe the exact module that caused it, I will update the topic in that case.

WinkyZhu · May 7, 2023, 12:49pm

You mentioned that you replaced all Niagara system, how did you do that? Replace with particle system or just make another niagara emitter to replace the old one? We are facing the same issue. Some niagara caused the crash, so we just disabled them. But sometimes it crashed when trying to load a level. The game crashed before the first frame was rendered. We checked the log, it says vk device lost.

SupportiveEntity · May 7, 2023, 7:51pm

Hey there @WinkyZhu! So the VK device lost error can be caused by a myriad of issues, generally any rendering errors could cause the device to fail. In many cases like this user’s have completely deleted offending systems and recreated them. For some reason this often fixes the issues. I’d believed it to originally be a caching error since that was the case, but even wiping the caches changed nothing in most user’s cases.

Sr.Vlad.m · July 26, 2023, 11:20am

just replaced with other niagara system