check (which become fatal) when world using fastgeo shuts down

Development Pipeline & Plugins
, ,
1

We noticed this in our build system inconsistently on a large map that used FastGeo, but it’s proven a bit time consuming to reproduce locally. In the end I had to run our client + server smoke test over and over, looks like on our current CL I hit the check once every 3-400 runs. It’s a race condition after all.

The issue is that we hit a check in `FPhysScene_Chaos::PushAsyncDestroyPhysicsStateJob`, specifically `check(Processor->IsAsyncPhysicsStateCreated());`. This is promoted to fatal when running our test suite.

The issue is that

bool UFastGeoContainer::IsAsyncPhysicsStateCreated() const
{
	return PhysicsStateCreation == EPhysicsStateCreation::Created;
}

and

PhysicsStateCreation Creating (1) UFastGeoContainer::EPhysicsStateCreation

Basically, it hits a fastgeo container that is still in the creating state when the world is shutting down.

Callstack for reference:

>	GameServer.exe!FPhysScene_Chaos::PushAsyncDestroyPhysicsStateJob(IAsyncPhysicsStateProcessor * Processor) Line 1868	C++
 [Inline Frame] GameServer.exe!UFastGeoContainer::OnDestroyPhysicsStateBegin_GameThread() Line 761	C++
 [Inline Frame] GameServer.exe!UFastGeoWorldSubsystem::PushAsyncDestroyPhysicsStateJobs(UFastGeoContainer *) Line 519	C++
 GameServer.exe!UFastGeoContainer::Unregister() Line 260	C++
 [Inline Frame] GameServer.exe!Invoke(void(UFastGeoWorldSubsystem::*)(UWorld *, const ULevelStreaming *, ULevel *)) Line 66	C++
 [Inline Frame] GameServer.exe!UE::Core::Private::Tuple::TTupleBase<TIntegerSequence<unsigned int>>::ApplyAfter(void(UFastGeoWorldSubsystem::*)(UWorld *, const ULevelStreaming *, ULevel *) &) Line 320	C++
 GameServer.exe!TBaseUObjectMethodDelegateInstance<0,UFastGeoWorldSubsystem,void __cdecl(UWorld *,ULevelStreaming const *,ULevel *),FDefaultDelegateUserPolicy>::ExecuteIfSafe(UWorld * <Params_0>, const ULevelStreaming * <Params_1>, ULevel * <Params_2>) Line 689	C++
 [Inline Frame] GameServer.exe!TMulticastDelegateBase<FDefaultDelegateUserPolicy>::Broadcast(UWorld *) Line 258	C++
 GameServer.exe!TMulticastDelegate<void __cdecl(UWorld *,ULevelStreaming const *,ULevel *),FDefaultDelegateUserPolicy>::Broadcast(UWorld * <Params_0>, const ULevelStreaming * <Params_1>, ULevel * <Params_2>) Line 1080	C++
 GameServer.exe!UWorld::RemoveFromWorld::__l2::<lambda>() Line 3953	C++
 GameServer.exe!UWorld::RemoveFromWorld(ULevel * Level, bool bAllowIncrementalRemoval, const TOptional<UE::FTimeout const> & ExternalTimeout, FNetLevelVisibilityTransactionId TransactionId, ULevelStreaming * InOwningLevelStreaming) Line 4035	C++
 GameServer.exe!ULevelStreaming::UpdateStreamingState(bool & bOutUpdateAgain, bool & bOutRedetermineTarget, const TOptional<UE::FTimeout const> & InExternalTimeout) Line 1103	C++
 [Inline Frame] GameServer.exe!FStreamingLevelPrivateAccessor::UpdateStreamingState(ULevelStreaming *) Line 804	C++
 GameServer.exe!UWorld::UpdateLevelStreaming(const TOptional<UE::FTimeout const> & ExternalTimeout) Line 4903	C++
 [Inline Frame] GameServer.exe!UWorld::FlushLevelStreaming::__l2::<lambda>() Line 5165	C++
 GameServer.exe!UWorld::FlushLevelStreaming(EFlushLevelStreamingType FlushType) Line 5168	C++
 GameServer.exe!UGameEngine::PreExit() Line 1352	C++
 GameServer.exe!FEngineLoop::Exit() Line 4840	C++
 [Inline Frame] GameServer.exe!EngineExit() Line 71	C++
 [Inline Frame] GameServer.exe!GuardedMain::__l2::EngineLoopCleanupGuard::{dtor}() Line 129	C++
 GameServer.exe!GuardedMain(const wchar_t * CmdLine) Line 201	C++
 GameServer.exe!GuardedMainWrapper(const wchar_t * CmdLine) Line 128	C++
 GameServer.exe!LaunchWindowsStartup(HINSTANCE__ * hInInstance, HINSTANCE__ * hPrevInstance, char * __formal, int nCmdShow, const wchar_t * CmdLine) Line 282	C++

[Attachment Removed]

2

Thanks Arnout - can I check what CL you have at the moment please and I’ll get a bug report created.

Best

Geoff Stacey

Developer Relations

EPIC Games

[Attachment Removed]

3

If you (or anyone else) is still hitting this issue, we were able to get around this issue by forcibly flushing the FastGeo async tasks in UFastGeoWorldSubsystem::OnLevelStartedRemoveFromWorld using FastGeo->Tick(true). Our QA is still running more robust testing to make sure this doesn’t introduce any additional issues, but it passes first muster at least. Hope this helps!

[Attachment Removed]

4

Hi all, for those who are hitting, or have hit this - it is fixed in version 5.8 onwards.

The fixing CL was 51162673

Best

Geoff

[Attachment Removed]

5

This was based on 5.6.0, I don’t think it is changed for 5.6.1.

[Attachment Removed]

6

Hello! We’re also hitting this issue. Is there a publicly tracked bug or something we can follow on this?

[Attachment Removed]