Server Crash in GatherNavigationDataGeometry after integrating 5.5

Development Programming & Scripting
, ,
1

This question was created in reference to: [Rare crash inside [Content removed]

We recently integrated 5.5.4 into our project. Since then we very randomly see the same crash as in the thread above. I’m mainly wondering if you have any guidance on it or if some active work is being done about it? Looking at the mentioned memory adress it seems to be an unhandled nullptr.

Kind regards,

Anton

Callstack:

[2025.06.09-11.29.19:226][920]LogCore: === Critical error: ===

Unhandled Exception: SIGSEGV: invalid attempt to read memory at address 0x0000000000000000

[2025.06.09-11.29.19:226][920]LogCore: 0x00007170757a07cd libc.so.6!UnknownFunction(0x1a07cd)

0x000000000a99fc4f SquidServer!ConstructItems<FAreaNavModifier, FAreaNavModifier, int> [D:/p4ws/Jenkins2-BP2-SWINU-020-001/Engine/Source/Runtime/Core/Public/GenericPlatform/GenericPlatformMemory.h:593]

0x000000000a9826b5 SquidServer!FCompositeNavModifier::FCompositeNavModifier(FCompositeNavModifier const&) [D:/p4ws/Jenkins2-BP2-SWINU-020-001/Engine/Source/Runtime/Core/Public/Containers/Array.h:3210]

0x000000000a981dbd SquidServer!FRecastTileGenerator::GatherNavigationDataGeometry(TSharedRef<FNavigationRelevantData, (ESPMode)1> const&, UNavigationSystemV1&, FNavDataConfig const&, bool) [D:/p4ws/Jenkins2-BP2-SWINU-020-001/Engine/Source/./Runtime/NavigationSystem/Private/NavMesh/RecastNavMeshGenerator.cpp:2166]

0x000000000a9824f2 SquidServer!FRecastTileGenerator::GatherGeometry(FRecastNavMeshGenerator const&, bool) [D:/p4ws/Jenkins2-BP2-SWINU-020-001/Engine/Source/./Runtime/NavigationSystem/Private/NavMesh/RecastNavMeshGenerator.cpp:2099]

0x000000000a980c39 SquidServer!FRecastTileGenerator::Setup(FRecastNavMeshGenerator const&, TArray<UE::Math::TBox<double>, TSizedDefaultAllocator<32> > const&) [D:/p4ws/Jenkins2-BP2-SWINU-020-001/Engine/Source/./Runtime/NavigationSystem/Private/NavMesh/RecastNavMeshGenerator.cpp:1827]

0x000000000a9969bd SquidServer!FRecastNavMeshGenerator::CreateTileGenerator(UE::Math::TIntPoint<int> const&, TArray<UE::Math::TBox<double>, TSizedDefaultAllocator<32> > const&, double) [D:/p4ws/Jenkins2-BP2-SWINU-020-001/Engine/Source/Runtime/NavigationSystem/Public/NavMesh/RecastNavMeshGenerator.h:1050]

0x000000000a996c31 SquidServer!FRecastNavMeshGenerator::ProcessTileTasksAsyncAndGetUpdatedTiles(int) [D:/p4ws/Jenkins2-BP2-SWINU-020-001/Engine/Source/./Runtime/NavigationSystem/Private/NavMesh/RecastNavMeshGenerator.cpp:6746]

0x000000000a9904c9 SquidServer!FRecastNavMeshGenerator::ProcessTileTasksAndGetUpdatedTiles(int) [D:/p4ws/Jenkins2-BP2-SWINU-020-001/Engine/Source/./Runtime/NavigationSystem/Private/NavMesh/RecastNavMeshGenerator.cpp:7171]

0x000000000a99071a SquidServer!FRecastNavMeshGenerator::TickAsyncBuild(float) [D:/p4ws/Jenkins2-BP2-SWINU-020-001/Engine/Source/./Runtime/NavigationSystem/Private/NavMesh/RecastNavMeshGenerator.cpp:5514]

0x000000000a8f2a91 SquidServer!UNavigationSystemV1::Tick(float) [D:/p4ws/Jenkins2-BP2-SWINU-020-001/Engine/Source/./Runtime/NavigationSystem/Private/NavigationSystem.cpp:1718]

0x000000000b632f9f SquidServer!UWorld::Tick(ELevelTick, float) [D:/p4ws/Jenkins2-BP2-SWINU-020-001/Engine/Source/./Runtime/Engine/Private/LevelTick.cpp:1428]

0x000000000b3bd458 SquidServer!UGameEngine::Tick(float, bool) [D:/p4ws/Jenkins2-BP2-SWINU-020-001/Engine/Source/./Runtime/Engine/Private/GameEngine.cpp:1869]

0x000000000c4cd32b SquidServer!FEngineLoop::Tick() [D:/p4ws/Jenkins2-BP2-SWINU-020-001/Engine/Source/./Runtime/Launch/Private/LaunchEngineLoop.cpp:5889]

0x000000000c4cee4a SquidServer!GuardedMain(char16_t const*) [D:/p4ws/Jenkins2-BP2-SWINU-020-001/Engine/Source/./Runtime/Launch/Private/Launch.cpp:69]

0x000000000c43f185 SquidServer!CommonUnixMain(int, char**, int (*)(char16_t const*), void (*)()) [D:/p4ws/Jenkins2-BP2-SWINU-020-001/Engine/Source/./Runtime/Unix/UnixCommonStartup/Private/UnixCommonStartup.cpp:323]

0x0000717075629d90 libc.so.6!UnknownFunction(0x29d8f)

0x0000717075629e40 libc.so.6!__libc_start_main(+0x7f)

0x000000000642902a SquidServer!_start(+0x29)

2

Steps to Reproduce

3

Also saw it being mentioned in: [Content removed]

4

To give some more context we run a DynamicModifiersOnly NavMesh so it can update in runtime but it’s not fully dynamic. For instance the most likely relevant actors would be our destructibles which uses MetaAreas so that the bigger NPCs can walk through them, while smaller NPCs have to navigate around them.

We also have DoFullyAsyncNavDataGathering set to true and it also builds asynchronously.

5

Hi Anton,

Does this happen in PIE with server in a separate process or only in a cooked build? Can you send us your full navigation settings? Do you have any steps that cause this to repro more often? Is it a fail that happens randomly in a session or is it when the actor is loaded and data gathered? Are you using World Partition streaming for unloading the actor?

It does seem to be stemming from a section that was modified to help track down rare crashes. I have reached out to the dev who added the comment to see what info might be most helpful in further diagnosing this.

-James

6

Discussing this on the team, we believe that a small change may fix this, and are curious if you could test this for us in your setup since you can reproduce it in your project.

The change is to move from using

const UObject* ObjectOwner = WeakOwnerPtr.Get();to

const UObject* ObjectOwner = WeakOwnerPtr.Pin();inside FCompositeNavModifier::GetInstantiatedMetaModifier. With gathering being async, it could potentially GC the ObjectOwner ptr while that function is running. Knowing that this does fix the issue would be extremely helpful for ensure this is truly solved or needs more investigation.

-James

7

Apologies, it should also change the type of ptr to be

const TStrongObjectPtr<const UObject> ObjectOwner = WeakOwnerPtr.Pin();There are a couple other places that need updated to get the ptr from the TStrongObjectPtr as well.

8

Sorry for the late response, has been out on vacation.

It did seem to show up more when we did some changes relating to NavLinks so not sure if it somehow resulted in corrupted memory due to incompatible NavMesh versions somehow. We use WorldGen / commandlets to rebuild NavMesh so maybe the NavLink actors was in a newer version while the NavMesh had yet to be rebuilt, not sure.

But it only happened in cooked builds.

In one of our tutorial maps it happened quite reliably the first time the NavMesh got rebuilt after entering the map.

9

Thank you for the extra info. The team was discussing this prior to the Epic summer break, and now that we are back, we will start looking at this again. If you do run into any more steps for reproducing this or additional info, we would appreciate it if you could share that with us.

10

Hello!

I’m Anton’s colleague, and I’ve been continuing the investigation into the same issue he previously reported. I’ve collected more occurrences of the crash, now with more detailed logs and consistent stack traces. Unfortunately, we still haven’t been able to reproduce it deterministically, it appears to occur randomly during runtime.

Callstack 1:

0x0000000006279c76 SquidServer!FMallocBinnedCommonBase::FBundle::PopHead [Engine/Source/Runtime/Core/Public/HAL/MallocBinnedCommon.h:260] 0x0000000006279c76 SquidServer!FMallocBinnedCommonBase::FFreeBlockList::PopFromFront [Engine/Source/Runtime/Core/Public/HAL/MallocBinnedCommon.h:301] 0x0000000006279c76 SquidServer!TMallocBinnedCommon<T>::FPerThreadFreeBlockLists::Malloc [Engine/Source/Runtime/Core/Public/HAL/MallocBinnedCommon.h:520] 0x0000000006279c76 SquidServer!FMallocBinned2::ReallocInline [Engine/Source/Runtime/Core/Public/HAL/MallocBinned2.h:405] 0x0000000006279c76 SquidServer!FMallocBinned2::Realloc [Engine/Source/Runtime/Core/Public/HAL/MallocBinned2.h:364] 0x00000000061ed2d2 SquidServer!TSizedHeapAllocator<T>::ForAnyElementType::ResizeAllocation [Engine/Source/Runtime/Core/Public/Containers/ContainerAllocationPolicies.h:746] 0x00000000076c9aff SquidServer!TArray<T>::AllocatorResizeAllocation [Engine/Source/Runtime/Core/Public/Containers/Array.h:3095] 0x00000000076c9aff SquidServer!TArray<T>::ResizeForCopy [Engine/Source/Runtime/Core/Public/Containers/Array.h:3183] 0x000000000a6e49d4 SquidServer!TArray<T>::CopyToEmpty<T> [Engine/Source/Runtime/Core/Public/Containers/Array.h:3209] 0x000000000a6e49d4 SquidServer!TArray<T>::TArray [Engine/Source/Runtime/Core/Public/Containers/Array.h:454] 0x000000000a6e49d4 SquidServer!FAreaNavModifier::FAreaNavModifier [Engine/Source/Runtime/NavigationSystem/Public/NavMesh/NavModifier.h:96] 0x000000000a6e49d4 SquidServer!ConstructItems<FAreaNavModifier, FAreaNavModifier, int> [Engine/Source/Runtime/Core/Public/Templates/MemoryOps.h:144] 0x000000000aabbd17 SquidServer!TArray<T>::CopyToEmpty<T> [Engine/Source/Runtime/Core/Public/Containers/Array.h:3210] 0x000000000aabbd17 SquidServer!TArray<T>::operator= [Engine/Source/Runtime/Core/Public/Containers/Array.h:512] 0x000000000aabbd17 SquidServer!FCompositeNavModifier::operator= [Engine/Source/Runtime/NavigationSystem/Public/NavMesh/NavModifier.h:274] 0x000000000aaba8da SquidServer!FCompositeNavModifier::GetInstantiatedMetaModifier [Engine/Source/Runtime/NavigationSystem/Private/NavMesh/NavigationModifier.cpp:840] 0x00000000061cb8fd SquidServer!FMRSWRecursiveAccessDetector::~FMRSWRecursiveAccessDetector [Engine/Source/Runtime/Core/Public/Concurrency/MTAccessDetector.h:480] 0x00000000061cb8fd SquidServer!TDelegateAccessHandlerBase<T>::~TDelegateAccessHandlerBase [Engine/Source/Runtime/Core/Public/UObject/Delegates/DelegateAccessHandler.h:106] 0x00000000061cbbe0 SquidServer!TArray<T>::RemoveAtSwap [Engine/Source/Runtime/Core/Public/Containers/Array.h:1886] 0x00000000061cbbe0 SquidServer!FMRSWRecursiveAccessDetector::ReleaseWriteAccess [Engine/Source/Runtime/Core/Public/Concurrency/MTAccessDetector.h:645] 0x00000000061cbbe0 SquidServer!TDelegateAccessHandlerBase<T>::FWriteAccessScope::~FWriteAccessScope [Engine/Source/Runtime/Core/Public/UObject/Delegates/DelegateAccessHandler.h:159] 0x000000000a64b9ce SquidServer!TDelegateBase<T>::~TDelegateBase [Engine/Source/Runtime/Core/Public/UObject/Delegates/DelegateBase.h:179] 0x000000000a64b9ce SquidServer!FCompositeNavModifier::~FCompositeNavModifier [Engine/Source/Runtime/NavigationSystem/Public/NavMesh/NavModifier.h:274] 0x000000000a6c6b8b SquidServer!FRecastTileGenerator::GatherNavigationDataGeometry [Engine/Source/Runtime/NavigationSystem/Private/NavMesh/RecastNavMeshGenerator.cpp:2165] 0x00007170757a07cd libc.so.6!__futex_abstimed_wait_cancelable64 [futex-internal.c:139] 0x000000000a6c60ce SquidServer!FRecastTileGenerator::GatherGeometryFromSources [Engine/Source/Runtime/NavigationSystem/Private/NavMesh/RecastNavMeshGenerator.cpp:1999] 0x000000000a6c60ce SquidServer!FRecastTileGenerator::DoWork [Engine/Source/Runtime/NavigationSystem/Private/NavMesh/RecastNavMeshGenerator.cpp:1935] 0x000000000623bd8e SquidServer!FAsyncTaskBase::DoWork [Engine/Source/Runtime/Core/Public/Async/AsyncWork.h:288] 0x000000000623bd8e SquidServer!FAsyncTaskBase::DoThreadedWork [Engine/Source/Runtime/Core/Public/Async/AsyncWork.h:312] 0x000000000c224ed8 SquidServer!FQueuedLowLevelThreadPool::AddQueuedWork::lambda::operator() [Engine/Source/Runtime/Core/Public/HAL/QueuedThreadPoolWrapper.h:484]Callstack 2:

0x000000000699be36 SquidServer!FMallocBinnedCommonBase::FBundle::PopHead [Engine/Source/Runtime/Core/Public/HAL/MallocBinnedCommon.h:260] 0x000000000699be36 SquidServer!FMallocBinnedCommonBase::FFreeBlockList::PopFromFront [Engine/Source/Runtime/Core/Public/HAL/MallocBinnedCommon.h:301] 0x000000000699be36 SquidServer!TMallocBinnedCommon<T>::FPerThreadFreeBlockLists::Malloc [Engine/Source/Runtime/Core/Public/HAL/MallocBinnedCommon.h:520] 0x000000000699be36 SquidServer!FMallocBinned2::ReallocInline [Engine/Source/Runtime/Core/Public/HAL/MallocBinned2.h:405] 0x000000000699be36 SquidServer!FMallocBinned2::Realloc [Engine/Source/Runtime/Core/Public/HAL/MallocBinned2.h:364] 0x00000000069f04df SquidServer!FMallocPoisonProxy::Realloc [Engine/Source/Runtime/Core/Public/HAL/MallocPoisonProxy.h:64] 0x00000000069c33e4 SquidServer!FMemory::Realloc [Engine/Source/Runtime/Core/Public/Misc/FMemory.inl] 0x000000000687b452 SquidServer!TSizedHeapAllocator<T>::ForAnyElementType::ResizeAllocation [Engine/Source/Runtime/Core/Public/Containers/ContainerAllocationPolicies.h:746] 0x0000000006e242af SquidServer!TArray<T>::AllocatorResizeAllocation [Engine/Source/Runtime/Core/Public/Containers/Array.h:3095] 0x0000000006e242af SquidServer!TArray<T>::ResizeForCopy [Engine/Source/Runtime/Core/Public/Containers/Array.h:3183] 0x000000000b58ec34 SquidServer!TArray<T>::CopyToEmpty<T> [Engine/Source/Runtime/Core/Public/Containers/Array.h:3209] 0x000000000b58ec34 SquidServer!TArray<T>::TArray [Engine/Source/Runtime/Core/Public/Containers/Array.h:454] 0x000000000b58ec34 SquidServer!FAreaNavModifier::FAreaNavModifier [Engine/Source/Runtime/NavigationSystem/Public/NavMesh/NavModifier.h:96] 0x000000000b58ec34 SquidServer!ConstructItems<FAreaNavModifier, FAreaNavModifier, int> [Engine/Source/Runtime/Core/Public/Templates/MemoryOps.h:144] 0x000000000b56b8a0 SquidServer!TArray<T>::CopyToEmpty<T> [Engine/Source/Runtime/Core/Public/Containers/Array.h:3210] 0x000000000b56b8a0 SquidServer!TArray<T>::operator= [Engine/Source/Runtime/Core/Public/Containers/Array.h:512] 0x000000000b56b8a0 SquidServer!FRecastTileGenerator::AppendModifier [Engine/Source/Runtime/NavigationSystem/Private/NavMesh/RecastNavMeshGenerator.cpp:2489] 0x000000000b9d88b0 SquidServer!AutoRTFM::Open<T> [Engine/Source/Runtime/Core/Public/Stats/AutoRTFM.h:677] 0x000000000b9d88b0 SquidServer!FScopeCycleCounter::~FScopeCycleCounter [Engine/Source/Runtime/Core/Public/Stats/Stats.h:85] 0x000000000b9d88b0 SquidServer!FCompositeNavModifier::GetInstantiatedMetaModifier [Engine/Source/Runtime/NavigationSystem/Private/NavMesh/NavigationModifier.cpp:923]

11

Callstack 1 looks like it may be fixed by pinning the weak ptr. Would it be possible to test that change?

For Callstack 2, I am not seeing it line up with 5.5 source. I also see it using AutoRTFM which could be causing other problems. I know we have fixed a few cases in other areas to support AutoRTFM, but I do not know that we have done the same in the navigation system.