Linux Server Crash: Ensure condition failed (vehicle physics)

I’ve been running some soak tests lately to stabilize my vehicle combat game. Here’s the scenario:

  1. Launch cloud-hosted dedicated server
  2. Connect and start playing
  3. Introduce 7 bots (for a total of 8 players - this definitely puts some stress on the network/cpu)
  4. After 2.5 hours, I get kicked from the server and get the following output in the server log

It is, of course, difficult to say exactly what’s causing this, since the bots are playing capture the flag, shooting eachother, ramming eachother, etc. I haven’t been able to reproduce this in local soak tests even while simulating latency.

This is the first chunk of the crash output (which I believe is the most useful):

[2020.03.12-23.40.31:060][333]LogActor: Warning: [Server] Registering with Master Server: <redacted>
[2020.03.12-23.40.31:061][333]LogActor: Warning: [Server] Sending Request...
[2020.03.12-23.40.32:079][335]LogActor: Warning: [Server] Success
[2020.03.12-23.40.32:079][335]LogActor: Warning: [Server] Response:[200]: 
[2020.03.12-23.42.11:455][436]LogStats: FPlatformStackWalk::StackWalkAndDump -  0.073 s
[2020.03.12-23.42.11:456][436]LogOutputDevice: Error: === Handled ensure: ===
[2020.03.12-23.42.11:456][436]LogOutputDevice: Error: 
[2020.03.12-23.42.11:456][436]LogOutputDevice: Error: Ensure condition failed: NewTransform.IsValid() [File:C:\Users\hughe\code\UnrealEngine\Engine\Source\Runt
ime\Engine\Private\Components\SceneComponent.cpp] [Line: 611]
[2020.03.12-23.42.11:456][436]LogOutputDevice: Error: 
[2020.03.12-23.42.11:456][436]LogOutputDevice: Error: Stack: 
[2020.03.12-23.42.11:456][436]LogOutputDevice: Error: [Callstack] 0x00000000044a6b1f KalimpongServer!FDebug::OptionallyLogFormattedEnsureMessageReturningFalseI
mpl(bool, char const*, char const*, int, wchar_t const*, ...) [C:/Users/hughe/code/UnrealEngine/Engine/Source/Runtime/Core/Private/Misc/AssertionMacros.cpp:439
]
[2020.03.12-23.42.11:456][436]LogOutputDevice: Error: [Callstack] 0x000000000616a37b KalimpongServer!USceneComponent::UpdateComponentToWorldWithParent(USceneCo
mponent*, FName, EUpdateTransformFlags, FQuat const&, ETeleportType) [C:/Users/hughe/code/UnrealEngine/Engine/Source/Runtime/Core/Public/Misc/AssertionMacros.h
:94]
[2020.03.12-23.42.11:456][436]LogOutputDevice: Error: [Callstack] 0x000000000616cfcd KalimpongServer!USceneComponent::UpdateChildTransforms(EUpdateTransformFla
gs, ETeleportType) [C:/Users/hughe/code/UnrealEngine/Engine/Source/Runtime/Engine/Classes/Components/SceneComponent.h:904]
[2020.03.12-23.42.11:456][436]LogOutputDevice: Error: [Callstack] 0x000000000670dac2 KalimpongServer!USkeletalMeshComponent::FinalizeAnimationUpdate() [C:/User
s/hughe/code/UnrealEngine/Engine/Source/Runtime/Engine/Private/PhysicsEngine/PhysAnim.cpp:439]
[2020.03.12-23.42.11:462][436]LogOutputDevice: Error: [Callstack] 0x000000000687ace3 KalimpongServer!FSkeletalMeshComponentEndPhysicsTickFunction::ExecuteTick(
float, ELevelTick, ENamedThreads::Type, TRefCountPtr<FGraphEvent> const&) [C:/Users/hughe/code/UnrealEngine/Engine/Source/Runtime/Engine/Private/SkeletalMeshCo
mponentPhysics.cpp:2838]
[2020.03.12-23.42.11:462][436]LogOutputDevice: Error: [Callstack] 0x0000000006ad98b9 KalimpongServer!FTickFunctionTask::DoTask(ENamedThreads::Type, TRefCountPt
r<FGraphEvent> const&) [C:/Users/hughe/code/UnrealEngine/Engine/Source/Runtime/Engine/Private/TickTaskManager.cpp:270]
[2020.03.12-23.42.11:462][436]LogOutputDevice: Error: [Callstack] 0x0000000006ad955c KalimpongServer!TGraphTask<FTickFunctionTask>::ExecuteTask(TArray<FBaseGra
phTask*, FDefaultAllocator>&, ENamedThreads::Type) [C:/Users/hughe/code/UnrealEngine/Engine/Source/Runtime/Core/Public/Async/TaskGraphInterfaces.h:829]
[2020.03.12-23.42.11:462][436]LogOutputDevice: Error: [Callstack] 0x00000000043793a4 KalimpongServer!FNamedTaskThread::ProcessTasksNamedThread(int, bool) [C:/U
sers/hughe/code/UnrealEngine/Engine/Source/Runtime/Core/Public/Async/TaskGraphInterfaces.h:498]
[2020.03.12-23.42.11:462][436]LogOutputDevice: Error: [Callstack] 0x00000000043787b3 KalimpongServer!FNamedTaskThread::ProcessTasksUntilQuit(int) [C:/Users/hug
he/code/UnrealEngine/Engine/Source/Runtime/Core/Private/Async/TaskGraph.cpp:573]
[2020.03.12-23.42.11:462][436]LogOutputDevice: Error: [Callstack] 0x0000000004376d1c KalimpongServer!FTaskGraphImplementation::WaitUntilTasksComplete(TArray<TR
efCountPtr<FGraphEvent>, TInlineAllocator<4u, FDefaultAllocator> > const&, ENamedThreads::Type) [C:/Users/hughe/code/UnrealEngine/Engine/Source/Runtime/Core/Pr
ivate/Async/TaskGraph.cpp:1356]
[2020.03.12-23.42.11:462][436]LogOutputDevice: Error: [Callstack] 0x0000000006ad6c8b KalimpongServer!FTickTaskSequencer::ReleaseTickGroup(ETickingGroup, bool) 
[C:/Users/hughe/code/UnrealEngine/Engine/Source/Runtime/Engine/Private/TickTaskManager.cpp:540]
[2020.03.12-23.42.11:462][436]LogOutputDevice: Error: [Callstack] 0x0000000006ad2abe KalimpongServer!FTickTaskManager::RunTickGroup(ETickingGroup, bool) [C:/Us
ers/hughe/code/UnrealEngine/Engine/Source/Runtime/Engine/Private/TickTaskManager.cpp:1454]
[2020.03.12-23.42.11:462][436]LogOutputDevice: Error: [Callstack] 0x000000000647660d KalimpongServer!UWorld::Tick(ELevelTick, float) [C:/Users/hughe/code/Unrea
lEngine/Engine/Source/Runtime/Engine/Private/LevelTick.cpp:779]
[2020.03.12-23.42.11:463][436]LogOutputDevice: Error: [Callstack] 0x00000000062f7253 KalimpongServer!UGameEngine::Tick(float, bool) [C:/Users/hughe/code/Unreal
Engine/Engine/Source/Runtime/Engine/Private/GameEngine.cpp:1307]
[2020.03.12-23.42.11:463][436]LogOutputDevice: Error: [Callstack] 0x0000000003da5118 KalimpongServer!FEngineLoop::Tick() [C:/Users/hughe/code/UnrealEngine/Engi
ne/Source/Runtime/Launch/Private/LaunchEngineLoop.cpp:3489]
[2020.03.12-23.42.11:463][436]LogOutputDevice: Error: [Callstack] 0x0000000003da871a KalimpongServer!GuardedMain(wchar_t const*) [C:/Users/hughe/code/UnrealEng
ine/Engine/Source/Runtime/Launch/Private/Launch.cpp:62]
[2020.03.12-23.42.11:463][436]LogOutputDevice: Error: [Callstack] 0x0000000006ee093e KalimpongServer!CommonUnixMain(int, char**, int (*)(wchar_t const*)) [C:/U
sers/hughe/code/UnrealEngine/Engine/Source/Runtime/Unix/UnixCommonStartup/Private/UnixCommonStartup.cpp:245]
[2020.03.12-23.42.11:463][436]LogOutputDevice: Error: [Callstack] 0x00007f67e4526b97 libc.so.6!__libc_start_main(+0xe6)
[2020.03.12-23.42.11:463][436]LogOutputDevice: Error: [Callstack] 0x0000000003d9d029 KalimpongServer!_start()
Ensure condition failed: !NewTransform.ContainsNaN() [File:C:\Users\hughe\code\UnrealEngine\Engine\Source\Runtime\Engine\Private\PhysicsEngine\BodyInstance.cpp
] [Line: 2781] 
SetBodyTransform contains NaN (/Game/Developer/Maps/BigTree/BigTree.BigTree:PersistentLevel.KPTruck_482.HitBox)
-nan,-nan,-nan|31.423342,23.381357,174.203445|1.000000,1.000000,1.000000
[2020.03.12-23.42.11:463][436]LogOutputDevice: Error: 
[2020.03.12-23.42.11:472][436]LogStats:                SubmitErrorReport -  0.000 s
[2020.03.12-23.42.11:643][436]LogCore: Warning: Unable to statfs('/home/piinecone/LinuxServer/Kalimpong/Saved/Crashes/ensureinfo-Kalimpong-pid-6-0E0984C708D7C6
DEFC0C223630CCAE5A/Kalimpong.log'): errno=2 (No such file or directory)
[2020.03.12-23.42.11:645][436]LogStats:                    SendNewReport -  0.172 s
[2020.03.12-23.42.11:645][436]LogStats:             FDebug::EnsureFailed -  0.265 s

This is immediately followed by:

[2020.03.12-23.42.11:727][436]LogStats: FPlatformStackWalk::StackWalkAndDump -  0.080 s
[2020.03.12-23.42.11:727][436]LogOutputDevice: Error: === Handled ensure: ===
[2020.03.12-23.42.11:727][436]LogOutputDevice: Error: 
[2020.03.12-23.42.11:727][436]LogOutputDevice: Error: Ensure condition failed: !NewTransform.ContainsNaN() [File:C:\Users\hughe\code\UnrealEngine\Engine\Source
\Runtime\Engine\Private\PhysicsEngine\BodyInstance.cpp] [Line: 2781]
[2020.03.12-23.42.11:727][436]LogOutputDevice: Error: SetBodyTransform contains NaN (/Game/Developer/Maps/BigTree/BigTree.BigTree:PersistentLevel.KPTruck_482.H
itBox)
[2020.03.12-23.42.11:727][436]LogOutputDevice: Error: -nan,-nan,-nan|31.423342,23.381357,174.203445|1.000000,1.000000,1.000000
[2020.03.12-23.42.11:727][436]LogOutputDevice: Error: Stack: 
[2020.03.12-23.42.11:727][436]LogOutputDevice: Error: [Callstack] 0x00000000044a6b1f KalimpongServer!FDebug::OptionallyLogFormattedEnsureMessageReturningFalseI
mpl(bool, char const*, char const*, int, wchar_t const*, ...) [C:/Users/hughe/code/UnrealEngine/Engine/Source/Runtime/Core/Private/Misc/AssertionMacros.cpp:439
]
[2020.03.12-23.42.11:727][436]LogOutputDevice: Error: [Callstack] 0x00000000066d2dd8 KalimpongServer!FBodyInstance::SetBodyTransform(FTransform const&, ETelepo
rtType) [C:/Users/hughe/code/UnrealEngine/Engine/Source/Runtime/Core/Public/Misc/AssertionMacros.h:94]
[2020.03.12-23.42.11:727][436]LogOutputDevice: Error: [Callstack] 0x000000000613b120 KalimpongServer!UPrimitiveComponent::OnUpdateTransform(EUpdateTransformFla
gs, ETeleportType) [C:/Users/hughe/code/UnrealEngine/Engine/Source/Runtime/Engine/Private/Components/PrimitiveComponent.cpp:780]
[2020.03.12-23.42.11:727][436]LogOutputDevice: Error: [Callstack] 0x000000000616a845 KalimpongServer!USceneComponent::PropagateTransformUpdate(bool, EUpdateTra
nsformFlags, ETeleportType) [C:/Users/hughe/code/UnrealEngine/Engine/Source/Runtime/Engine/Private/Components/SceneComponent.cpp:726]
[2020.03.12-23.42.11:727][436]LogOutputDevice: Error: [Callstack] 0x000000000616a408 KalimpongServer!USceneComponent::UpdateComponentToWorldWithParent(USceneCo
mponent*, FName, EUpdateTransformFlags, FQuat const&, ETeleportType) [C:/Users/hughe/code/UnrealEngine/Engine/Source/Runtime/Engine/Private/Components/SceneCom
ponent.cpp:628]
[2020.03.12-23.42.11:727][436]LogOutputDevice: Error: [Callstack] 0x000000000616cfcd KalimpongServer!USceneComponent::UpdateChildTransforms(EUpdateTransformFla
gs, ETeleportType) [C:/Users/hughe/code/UnrealEngine/Engine/Source/Runtime/Engine/Classes/Components/SceneComponent.h:904]
[2020.03.12-23.42.11:728][436]LogOutputDevice: Error: [Callstack] 0x000000000670dac2 KalimpongServer!USkeletalMeshComponent::FinalizeAnimationUpdate() [C:/User
s/hughe/code/UnrealEngine/Engine/Source/Runtime/Engine/Private/PhysicsEngine/PhysAnim.cpp:439]
[2020.03.12-23.42.11:728][436]LogOutputDevice: Error: [Callstack] 0x000000000687ace3 KalimpongServer!FSkeletalMeshComponentEndPhysicsTickFunction::ExecuteTick(
float, ELevelTick, ENamedThreads::Type, TRefCountPtr<FGraphEvent> const&) [C:/Users/hughe/code/UnrealEngine/Engine/Source/Runtime/Engine/Private/SkeletalMeshCo
mponentPhysics.cpp:2838]
[2020.03.12-23.42.11:728][436]LogOutputDevice: Error: [Callstack] 0x0000000006ad98b9 KalimpongServer!FTickFunctionTask::DoTask(ENamedThreads::Type, TRefCountPt
r<FGraphEvent> const&) [C:/Users/hughe/code/UnrealEngine/Engine/Source/Runtime/Engine/Private/TickTaskManager.cpp:270]
[2020.03.12-23.42.11:734][436]LogOutputDevice: Error: [Callstack] 0x0000000006ad955c KalimpongServer!TGraphTask<FTickFunctionTask>::ExecuteTask(TArray<FBaseGra
phTask*, FDefaultAllocator>&, ENamedThreads::Type) [C:/Users/hughe/code/UnrealEngine/Engine/Source/Runtime/Core/Public/Async/TaskGraphInterfaces.h:829]
[2020.03.12-23.42.11:734][436]LogOutputDevice: Error: [Callstack] 0x00000000043793a4 KalimpongServer!FNamedTaskThread::ProcessTasksNamedThread(int, bool) [C:/U
sers/hughe/code/UnrealEngine/Engine/Source/Runtime/Core/Public/Async/TaskGraphInterfaces.h:498]
[2020.03.12-23.42.11:734][436]LogOutputDevice: Error: [Callstack] 0x00000000043787b3 KalimpongServer!FNamedTaskThread::ProcessTasksUntilQuit(int) [C:/Users/hug
he/code/UnrealEngine/Engine/Source/Runtime/Core/Private/Async/TaskGraph.cpp:573]
[2020.03.12-23.42.11:734][436]LogOutputDevice: Error: [Callstack] 0x0000000004376d1c KalimpongServer!FTaskGraphImplementation::WaitUntilTasksComplete(TArray<TR
efCountPtr<FGraphEvent>, TInlineAllocator<4u, FDefaultAllocator> > const&, ENamedThreads::Type) [C:/Users/hughe/code/UnrealEngine/Engine/Source/Runtime/Core/Pr
ivate/Async/TaskGraph.cpp:1356]
[2020.03.12-23.42.11:734][436]LogOutputDevice: Error: [Callstack] 0x0000000006ad6c8b KalimpongServer!FTickTaskSequencer::ReleaseTickGroup(ETickingGroup, bool) 
[C:/Users/hughe/code/UnrealEngine/Engine/Source/Runtime/Engine/Private/TickTaskManager.cpp:540]
[2020.03.12-23.42.11:734][436]LogOutputDevice: Error: [Callstack] 0x0000000006ad2abe KalimpongServer!FTickTaskManager::RunTickGroup(ETickingGroup, bool) [C:/Us
ers/hughe/code/UnrealEngine/Engine/Source/Runtime/Engine/Private/TickTaskManager.cpp:1454]
[2020.03.12-23.42.11:734][436]LogOutputDevice: Error: [Callstack] 0x000000000647660d KalimpongServer!UWorld::Tick(ELevelTick, float) [C:/Users/hughe/code/Unrea
lEngine/Engine/Source/Runtime/Engine/Private/LevelTick.cpp:779]
[2020.03.12-23.42.11:734][436]LogOutputDevice: Error: [Callstack] 0x00000000062f7253 KalimpongServer!UGameEngine::Tick(float, bool) [C:/Users/hughe/code/Unreal
Engine/Engine/Source/Runtime/Engine/Private/GameEngine.cpp:1307]
[2020.03.12-23.42.11:734][436]LogOutputDevice: Error: [Callstack] 0x0000000003da5118 KalimpongServer!FEngineLoop::Tick() [C:/Users/hughe/code/UnrealEngine/Engi
ne/Source/Runtime/Launch/Private/LaunchEngineLoop.cpp:3489]
[2020.03.12-23.42.11:734][436]LogOutputDevice: Error: [Callstack] 0x0000000003da871a KalimpongServer!GuardedMain(wchar_t const*) [C:/Users/hughe/code/UnrealEng
ine/Engine/Source/Runtime/Launch/Private/Launch.cpp:62]
[2020.03.12-23.42.11:734][436]LogOutputDevice: Error: [Callstack] 0x0000000006ee093e KalimpongServer!CommonUnixMain(int, char**, int (*)(wchar_t const*)) [C:/U
sers/hughe/code/UnrealEngine/Engine/Source/Runtime/Unix/UnixCommonStartup/Private/UnixCommonStartup.cpp:245]
[2020.03.12-23.42.11:734][436]LogOutputDevice: Error: [Callstack] 0x00007f67e4526b97 libc.so.6!__libc_start_main(+0xe6)
[2020.03.12-23.42.11:734][436]LogOutputDevice: Error: [Callstack] 0x0000000003d9d029 KalimpongServer!_start()
[2020.03.12-23.42.11:734][436]LogOutputDevice: Error: 
[2020.03.12-23.42.11:737][436]LogStats:                SubmitErrorReport -  0.000 s
[2020.03.12-23.42.11:939][436]LogCore: Warning: Unable to statfs('/home/piinecone/LinuxServer/Kalimpong/Saved/Crashes/ensureinfo-Kalimpong-pid-6-0E0A978808D7C6
DEFC38E58C30D12844/Kalimpong.log'): errno=2 (No such file or directory)
[2020.03.12-23.42.11:940][436]LogCore: Done sending crash report for ensure().
[2020.03.12-23.42.11:940][436]LogStats:                    SendNewReport -  0.203 s
[2020.03.12-23.42.11:940][436]LogStats:             FDebug::EnsureFailed -  0.295 s

And lastly, I get what appears to be a useful stack trace, but I believe it’s a crash which occurs because of whatever is causing the transform to be invalid (in the first chunk of the trace). Once the transform is invalid, normal behavior will crash.

[2020.03.12-23.42.11:949][436]LogSpawn: Warning: SpawnActor failed because the given transform (-nan,-nan,-nan|90.000000,0.000000,0.000000|1.000000,1.000000,1.
000000) is invalid
Signal 11 caught.
Malloc Size=131076 LargeMemoryPoolOffset=131092 
Malloc Size=65535 LargeMemoryPoolOffset=196655 
Malloc Size=117872 LargeMemoryPoolOffset=314544 
Malloc Size=68112 LargeMemoryPoolOffset=382672 
Malloc Size=145512 LargeMemoryPoolOffset=528200 
Realloc PtrSize=68112 NewSize=145512 PooledPtr=0x00007f67e35eccc0
[2020.03.12-23.42.11:955][436]LogSpawn: Warning: SpawnActor failed because the given transform (-nan,-nan,-nan|90.000000,0.000000,0.000000|1.000000,1.000000,1.
000000) is invalid
CommonUnixCrashHandler: Signal=11
[2020.03.12-23.42.12:164][436]LogCore: === Critical error: ===
Unhandled Exception: SIGSEGV: invalid attempt to read memory at address 0x0000000000000120
[2020.03.12-23.42.12:164][436]LogCore: 0x00000000042cd3b3 KalimpongServer!AKPVehicle::LaunchFlareImpl() [C:/Users/hughe/code/UnrealEngine/Engine/Source/Runtime
/Core/Public/Containers/EnumAsByte.h:96]
0x000000000428f4aa KalimpongServer!AKPVehicle::LaunchFireworksForSpecialEvent() [C:/Users/developer/Kalimpong/Source/Kalimpong/KPVehicle.cpp:3098]
0x000000000428f215 KalimpongServer!AKPFlag::HandleFlagCaptureByVehicle(AKPVehicle*) [C:/Users/developer/Kalimpong/Source/Kalimpong/KPFlag.cpp:505]
0x000000000428e808 KalimpongServer!AKPFlag::OnFlagAtBaseOverlapBegin(UPrimitiveComponent*, AActor*, UPrimitiveComponent*, int, bool, FHitResult const&) [C:/Use
rs/developer/Kalimpong/Source/Kalimpong/KPFlag.cpp:480]
0x0000000004303813 KalimpongServer!AKPFlag::execOnFlagAtBaseOverlapBegin(UObject*, FFrame&, void*)(+0x232)
0x00000000047b0cdc KalimpongServer!UFunction::Invoke(UObject*, FFrame&, void*) [C:/Users/hughe/code/UnrealEngine/Engine/Source/Runtime/CoreUObject/Private/UObj
ect/Class.cpp:4728]
0x00000000048f9c1f KalimpongServer!UObject::ProcessEvent(UFunction*, void*) [C:/Users/hughe/code/UnrealEngine/Engine/Source/Runtime/CoreUObject/Private/UObject
/ScriptCore.cpp:1342]
0x0000000005e882e5 KalimpongServer!AActor::ProcessEvent(UFunction*, void*) [C:/Users/hughe/code/UnrealEngine/Engine/Source/Runtime/Engine/Private/Actor.cpp:724
]
0x0000000003dca0e2 KalimpongServer!void TMulticastScriptDelegate<FWeakObjectPtr>::ProcessMulticastDelegate<UObject>(void*) const [C:/Users/hughe/code/UnrealEng
ine/Engine/Source/Runtime/Core/Public/UObject/ScriptDelegates.h:467]
0x0000000006141a4b KalimpongServer!UPrimitiveComponent::BeginComponentOverlap(FOverlapInfo const&, bool) [C:/Users/hughe/code/UnrealEngine/Engine/Source/Runtim
e/Engine/Classes/Components/PrimitiveComponent.h:145]
0x0000000006145909 KalimpongServer!UPrimitiveComponent::UpdateOverlapsImpl(TArray<FOverlapInfo, FDefaultAllocator> const*, bool, TArray<FOverlapInfo, FDefaultA
llocator> const*) [C:/Users/hughe/code/UnrealEngine/Engine/Source/Runtime/Engine/Private/Components/PrimitiveComponent.cpp:3022]
0x0000000006145a6a KalimpongServer!UPrimitiveComponent::UpdateOverlapsImpl(TArray<FOverlapInfo, FDefaultAllocator> const*, bool, TArray<FOverlapInfo, FDefaultA
llocator> const*) [C:/Users/hughe/code/UnrealEngine/Engine/Source/Runtime/Engine/Classes/Components/SceneComponent.h:1714]
0x000000000670dca8 KalimpongServer!USkeletalMeshComponent::FinalizeAnimationUpdate() [C:/Users/hughe/code/UnrealEngine/Engine/Source/Runtime/Engine/Classes/Com
ponents/SceneComponent.h:1714]
0x000000000687ace3 KalimpongServer!FSkeletalMeshComponentEndPhysicsTickFunction::ExecuteTick(float, ELevelTick, ENamedThreads::Type, TRefCountPtr<FGraphEvent> 
const&) [C:/Users/hughe/code/UnrealEngine/Engine/Source/Runtime/Engine/Private/SkeletalMeshComponentPhysics.cpp:2838]
0x0000000006ad98b9 KalimpongServer!FTickFunctionTask::DoTask(ENamedThreads::Type, TRefCountPtr<FGraphEvent> const&) [C:/Users/hughe/code/UnrealEngine/Engine/So
urce/Runtime/Engine/Private/TickTaskManager.cpp:270]
0x0000000006ad955c KalimpongServer!TGraphTask<FTickFunctionTask>::ExecuteTask(TArray<FBaseGraphTask*, FDefaultAllocator>&, ENamedThreads::Type) [C:/Users/hughe
/code/UnrealEngine/Engine/Source/Runtime/Core/Public/Async/TaskGraphInterfaces.h:829]
0x00000000043793a4 KalimpongServer!FNamedTaskThread::ProcessTasksNamedThread(int, bool) [C:/Users/hughe/code/UnrealEngine/Engine/Source/Runtime/Core/Public/Asy
nc/TaskGraphInterfaces.h:498]
0x00000000043787b3 KalimpongServer!FNamedTaskThread::ProcessTasksUntilQuit(int) [C:/Users/hughe/code/UnrealEngine/Engine/Source/Runtime/Core/Private/Async/Task
Graph.cpp:573]
Malloc Size=87025 LargeMemoryPoolOffset=615249 
0x0000000004376d1c KalimpongServer!FTaskGraphImplementation::WaitUntilTasksComplete(TArray<TRefCountPtr<FGraphEvent>, TInlineAllocator<4u, FDefaultAllocator> >
 const&, ENamedThreads::Type) [C:/Users/hughe/code/UnrealEngine/Engine/Source/Runtime/Core/Private/Async/TaskGraph.cpp:1356]
0x0000000006ad6c8b KalimpongServer!FTickTaskSequencer::ReleaseTickGroup(ETickingGroup, bool) [C:/Users/hughe/code/UnrealEngine/Engine/Source/Runtime/Engine/Pri
vate/TickTaskManager.cpp:540]
0x0000000006ad2abe KalimpongServer!FTickTaskManager::RunTickGroup(ETickingGroup, bool) [C:/Users/hughe/code/UnrealEngine/Engine/Source/Runtime/Engine/Private/T
ickTaskManager.cpp:1454]
0x000000000647660d KalimpongServer!UWorld::Tick(ELevelTick, float) [C:/Users/hughe/code/UnrealEngine/Engine/Source/Runtime/Engine/Private/LevelTick.cpp:779]
0x00000000062f7253 KalimpongServer!UGameEngine::Tick(float, bool) [C:/Users/hughe/code/UnrealEngine/Engine/Source/Runtime/Engine/Private/GameEngine.cpp:1307]
0x0000000003da5118 KalimpongServer!FEngineLoop::Tick() [C:/Users/hughe/code/UnrealEngine/Engine/Source/Runtime/Launch/Private/LaunchEngineLoop.cpp:3489]
0x0000000003da871a KalimpongServer!GuardedMain(wchar_t const*) [C:/Users/hughe/code/UnrealEngine/Engine/Source/Runtime/Launch/Private/Launch.cpp:62]
0x0000000006ee093e KalimpongServer!CommonUnixMain(int, char**, int (*)(wchar_t const*)) [C:/Users/hughe/code/UnrealEngine/Engine/Source/Runtime/Unix/UnixCommon
Startup/Private/UnixCommonStartup.cpp:245]
0x00007f67e4526b97 libc.so.6!__libc_start_main(+0xe6)
0x0000000003d9d029 KalimpongServer!_start()
[2020.03.12-23.42.12:174][436]LogExit: Executing StaticShutdownAfterError
[2020.03.12-23.42.12:184][436]LogCore: Warning: Unable to statfs('/home/piinecone/LinuxServer/Kalimpong/Saved/Crashes/crashinfo-Kalimpong-pid-6-0E0B926908D7C6D
EFC5E4DAE30D4E5F4/Kalimpong.log'): errno=2 (No such file or directory)
Engine crash handling finished; re-raising signal 11 for the default handler. Good bye.
Segmentation fault (core dumped)

I* resolved this!

*Actually TheJamsh and vr_marco (in Discord) did all of the heavy lifting here.

Here’s what I did to fix this:

  1. Enable Enhanced Determinism in the Project’s Physics Settings
  2. Reduce Max Depenetration Velocity (also in the Physics Settings)
  3. Enforce linear and angular velocity magnitude limits on Tick to prevent physics bodies from moving too quickly
  4. I am using substepping, but I do not know if this has an impact

I haven’t seen a single Ensure warning or crash in hundreds of hours of client/server play and soak tests since.