Inexplicable random physics crash

[anonymous_user_dd151c9c]

I get this crash, apparently at random, usually within a few minutes of starting the game:

APEX_DestructiblePROFILE_x64!nvidia::destructible::DestructibleUserNotify::onContact() [d:\build\++ue4+release-4.16+physx_compile\sync\engine\source	hirdparty\physx\apex_1.4\module\destructible\src\destructiblescene.cpp:231]
APEXFrameworkPROFILE_x64!nvidia::apex::ApexSceneUserNotify::onContact() [d:\build\++ue4+release-4.16+physx_compile\sync\engine\source	hirdparty\physx\apex_1.4\framework\src\apexsceneusernotify.cpp:214]
PhysX3PROFILE_x64!physx::Sc::Scene::fireQueuedContactCallbacks() [d:\build\++ue4+release-4.16+physx_compile\sync\engine\source	hirdparty\physx\physx_3.4\source\simulationcontroller\src\scscene.cpp:4354]
PhysX3PROFILE_x64!physx::NpScene::fetchResults() [d:\build\++ue4+release-4.16+physx_compile\sync\engine\source	hirdparty\physx\physx_3.4\source\physx\src
pscene.cpp:2309]
APEXFrameworkPROFILE_x64!nvidia::apex::ApexScene::fetchResults() [d:\build\++ue4+release-4.16+physx_compile\sync\engine\source	hirdparty\physx\apex_1.4\framework\src\apexscene.cpp:1377]
UE4Editor_Engine!FPhysScene::ProcessPhysScene() [c:\repositories\unrealengine\engine\source\runtime\engine\private\physicsengine\physscene.cpp:1149]
UE4Editor_Engine!TBaseRawMethodDelegateInstance<0,FPhysScene,void __cdecl(enum ENamedThreads::Type,TRefCountPtr<FGraphEvent> const & __ptr64),enum EPhysicsSceneType>::ExecuteIfSafe() [c:\repositories\unrealengine\engine\source\runtime\core\public\delegates\delegateinstancesimpl.h:648]
UE4Editor_Engine!TGraphTask<FDelegateGraphTask>::ExecuteTask() [c:\repositories\unrealengine\engine\source\runtime\core\public\async	askgraphinterfaces.h:883]
UE4Editor_Core!FNamedTaskThread::ProcessTasksNamedThread() [c:\repositories\unrealengine\engine\source\runtime\core\private\async	askgraph.cpp:954]
UE4Editor_Core!FNamedTaskThread::ProcessTasksUntilQuit() [c:\repositories\unrealengine\engine\source\runtime\core\private\async	askgraph.cpp:701]
UE4Editor_Core!FTaskGraphImplementation::WaitUntilTasksComplete() [c:\repositories\unrealengine\engine\source\runtime\core\private\async	askgraph.cpp:1809]
UE4Editor_Engine!FTickTaskSequencer::ReleaseTickGroup() [c:\repositories\unrealengine\engine\source\runtime\engine\private	icktaskmanager.cpp:538]
UE4Editor_Engine!FTickTaskManager::RunTickGroup() [c:\repositories\unrealengine\engine\source\runtime\engine\private	icktaskmanager.cpp:1450]
UE4Editor_Engine!UWorld::RunTickGroup() [c:\repositories\unrealengine\engine\source\runtime\engine\private\leveltick.cpp:758]
UE4Editor_Engine!UWorld::Tick() [c:\repositories\unrealengine\engine\source\runtime\engine\private\leveltick.cpp:1389]
UE4Editor_UnrealEd!UEditorEngine::Tick() [c:\repositories\unrealengine\engine\source\editor\unrealed\private\editorengine.cpp:1633]
UE4Editor_UnrealEd!UUnrealEdEngine::Tick() [c:\repositories\unrealengine\engine\source\editor\unrealed\private\unrealedengine.cpp:386]
UE4Editor!FEngineLoop::Tick() [c:\repositories\unrealengine\engine\source\runtime\launch\private\launchengineloop.cpp:3119]
UE4Editor!GuardedMain() [c:\repositories\unrealengine\engine\source\runtime\launch\private\launch.cpp:166]
UE4Editor!GuardedMainWrapper() [c:\repositories\unrealengine\engine\source\runtime\launch\private\windows\launchwindows.cpp:134]
UE4Editor!WinMain() [c:\repositories\unrealengine\engine\source\runtime\launch\private\windows\launchwindows.cpp:210]
UE4Editor!__scrt_common_main_seh() [f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl:253]
kernel32
ntdll

Weird things about this:

• None of our game code appears in the callstack
• To the best of my knowledge we have no destructible meshes in the scene, just plain ol’ static meshes with standard physics enabled

This started happening when I added a custom component for playing impact audio/vfx when physics objects collide with things (e.g. if you throw or drop something). That component does this on startup:

void UPhysicsImpactComponent::BeginPlay()
{
    Super::BeginPlay();

    GetOwner()->OnActorHit.AddDynamic(this, &UPhysicsImpactComponent::OnActorHit);

    UPrimitiveComponent* root = Cast<UPrimitiveComponent>(GetOwner()->GetRootComponent());
    root->SetNotifyRigidBodyCollision(true);
}

…and this on shutdown:

void UPhysicsImpactComponent::EndPlay(const EEndPlayReason::Type EndPlayReason)
{
    GetOwner()->OnActorHit.RemoveDynamic(this, &UPhysicsImpactComponent::OnActorHit);
    Super::EndPlay(EndPlayReason);
}

My best guess is some UPhysicsImpactComponent instance had subscribed to an OnActorHit, then became invalid and somehow (?) failed to unsubscribe, and then that event later got called. But if that’s correct, I don’t understand how we’d have missed the EndPlay. Otherwise… I don’t even know WTF.

We’re on UE 4.16.2 on Windows FWIW.

[anonymous_user_dd151c9c]

There’s no way we’re the only ones seeing this… is there? O_o

With logging I verified that we are getting all the EndPlay calls we’d expect, so we shouldn’t have a stale delegate subscription. Since making this post we’ve updated to 4.16.3 and the issue still occurs exactly as written. I also tried moving to 4.18.0 on a branch and the issue still occurs but with a very slightly different callstack.

If I comment out the root->SetNotifyRigidBodyCollision(true) the crash seems to go away. Of course we never receive the OnActorHit events in this case either.

I still haven’t been able to isolate a very specific repro. It’s just “make physics objects move around a bunch and it’ll crash shortly”. It’s so simple and quick to trigger and has such high exposure that I’m really surprised not to be able to find anything about this anywhere online. We’re not doing anything fancy in our game code related to physics or anything else; this is actually a pretty simple game, all things considered.

I tried looking into this engine-side but it’s crashing down in the PhysX library which is built separately from the core engine, and the PhysX build scripts don’t support VS 2017 (which we use) and make a lot of assumptions about the build environment which are not valid. Needless to say, I’ve had no success rebuilding PhysX which means an internal patch is out of the question, at least so far… which is very frustrating. (And this thread was NOT encouraging.)

I’ve done several passes both manually and programmatically to ascertain that we do NOT have any Apex destruction or Apex cloth instances in our scene. And yet, we’re crashing in an Apex scene update and collision callback handler. From digging into the PhysX source code (despite being unable to build it) it looks like when Apex support is available then it will use Apex for everything regardless. So then I tried rebuilding the engine with bCompileAPEX=false in the default TargetRules but the engine won’t even compile at that point; there appears to be a bunch of code that references Apex functions and is not wrapped with a #if WITH_APEX, so that fix is right out as well.

I’m pretty much at my wits’ end, here.

[anonymous_user_dd151c9c]

The crash appears to occur only if I have actors with our UPhysicsImpactComponent in any sublevel. If all UPhysicsImpactComponent owners exist in the persistent level only, the crash doesn’t seem to happen.

It doesn’t matter if the sublevel’s streaming method is set to “always loaded” or if it’s set to “blueprint” with “initially loaded” and “initially visible” checked. Either way, the crash will present the same. In this test, I’m never unloading any sublevel. For all practical purposes, these two scenarios should be identical:

1. Single persistent level, no sublevels at all
2. Persistent level with one sublevel, physics objects are on the sublevel, sublevel is always loaded and never unloaded

But in fact, scenario 1 works and scenario 2 crashes.

I don’t know what this really means yet, but it’s… something.

[Rapid_Wirehead]

i am linking you to same issue here → https://answers.unrealengine.com/questions/663845/physx3profile-x64-based-crash-read-more.html

[post32768]

Has anyone figured out a solution or workaround for this?

[wilsonc]

I ran into the same issue, turns out it was caused by SetNotifyRigidBodyCollision(true) being called while the actor was being destroyed. Strangely though IsValid() or IsPendingKill() checks didn’t prevent it. Putting the function in the construction script or postinitializecomponents doesn’t work (it needs to be called during runtime for the hit event to actually be called), but it seems like a a function like this shouldn’t be called at runtime. Still looking into a solution, will update if I find anything.

[PhoeniX-Storms]

POSSIBLE SOLUTION
I’ve found out that I had crashes being caused by changing settings in the PROJECT SETTINGS
If you have changed something, try resetting it to the default state. Here are my settings I’ve copied from a blank project and it solved my random physics crash with objects…

I really hope this helps…