Crash: Multi-threading timing issue- rendering while shutting down

Development Programming & Scripting
1

I’m infrequently seeing a timing-related crash due to trying to render an Instanced Static Mesh while the game is shutting down.

I can fix the symptom by changing:

FSceneInterface* UActorComponent::GetScene() const
{
	return (World ? World->Scene : NULL);
}

to become:

FSceneInterface* UActorComponent::GetScene() const
{
	UWorld* world = World;
	return (world ? world->Scene : NULL);
}

…but then I’m only addressing the symptom, not the cause (and there’s no guarantee that world->Scene won’t be removed either!). Because of that, I thought I’d ask whether this fix is worth submitting?

Here’s a list of the threads at the time of the crash:

Not Flagged		11480	0	Worker Thread	atidxx64.dll thread	atidxx64.dll!00007ffc1e5d8584	Normal
Not Flagged		20324	0	Main Thread	Main Thread	UE4Editor-Engine-Win64-Debug.dll!UPrimitiveComponent::GetCollisionResponseToChannels	Normal
Not Flagged		19296	0	Worker Thread	PoolThread 0	UE4Editor-Core-Win64-Debug.dll!FEventWin::Wait	Normal
Not Flagged		13708	0	Worker Thread	PoolThread 1	UE4Editor-Core-Win64-Debug.dll!FEventWin::Wait	Normal
Not Flagged		11636	0	Worker Thread	PoolThread 2	UE4Editor-Core-Win64-Debug.dll!FEventWin::Wait	Normal
Not Flagged		20036	0	Worker Thread	NetClient	UE4Editor-OculusRift-Win64-Debug.dll!OVR::Thread::MSleep	Normal
Not Flagged		20032	0	Worker Thread	mswsock.dll thread	mswsock.dll!00007ffc22347f21	Above Normal
Not Flagged		17804	0	Worker Thread	TaskGraphThread 0	UE4Editor-Core-Win64-Debug.dll!FEventWin::Wait	Normal
Not Flagged		20308	0	Worker Thread	StatsThread	UE4Editor-Core-Win64-Debug.dll!FEventWin::Wait	Below Normal
Not Flagged		19948	0	Worker Thread	atidxx64.dll thread	atidxx64.dll!00007ffc1ea802af	Normal
Not Flagged		19532	0	Worker Thread	ShaderCompilingThread	UE4Editor-Core-Win64-Debug.dll!FWindowsPlatformProcess::Sleep	Normal
Not Flagged		18640	0	Worker Thread	AsyncIOSystem	UE4Editor-Core-Win64-Debug.dll!FEventWin::Wait	Above Normal
Not Flagged		13844	0	Worker Thread	FMessageBus.Router	UE4Editor-Core-Win64-Debug.dll!FEventWin::Wait	Normal
Not Flagged		6496	0	Worker Thread	FDDCCleanup	UE4Editor-Core-Win64-Debug.dll!FWindowsPlatformProcess::Sleep	Below Normal
Not Flagged		19160	0	Worker Thread	msvcrt.dll thread	RTWorkQ.dll!00007ffc1e054ded	Time Critical
Not Flagged		11756	0	Worker Thread	FUdpMessageProcessor	UE4Editor-Core-Win64-Debug.dll!FEventWin::Wait	Above Normal
Not Flagged		14208	0	Worker Thread	FUdpMessageBeacon	UE4Editor-Core-Win64-Debug.dll!FEventWin::Wait	Above Normal
Not Flagged		15744	0	Worker Thread	FUdpMessageProcessor.Sender	UE4Editor-Core-Win64-Debug.dll!FEventWin::Wait	Above Normal
Not Flagged		18788	0	Worker Thread	UdpMessageMulticastReceiver	mswsock.dll!00007ffc22343fcc	Above Normal
Not Flagged		20396	0	Worker Thread	UdpMessageUnicastReceiver	mswsock.dll!00007ffc22343fcc	Above Normal
Not Flagged		13812	0	Worker Thread	FAssetDataGatherer	UE4Editor-Core-Win64-Debug.dll!FWindowsPlatformProcess::Sleep	Below Normal
Not Flagged		17644	0	Worker Thread	FFileTransferRunnable	UE4Editor-Core-Win64-Debug.dll!FEventWin::Wait	Below Normal
Not Flagged		13980	0	Worker Thread	XAudio2_7.dll thread	XAudio2_7.dll!00007ffbe4022891	14
Not Flagged		18776	0	Worker Thread	ntdll.dll thread	ntdll.dll!00007ffc257f316a	Normal
Not Flagged		16024	0	Worker Thread	atidxx64.dll thread	atidxx64.dll!00007ffc1e5d8584	Normal
Not Flagged		16788	0	Worker Thread	ntdll.dll thread	ntdll.dll!00007ffc257f316a	Normal
Not Flagged		18092	0	Worker Thread	ntdll.dll thread	ntdll.dll!00007ffc257f316a	Normal
Not Flagged		10364	0	Worker Thread	ScreenSaverInhibitor	UE4Editor-Core-Win64-Debug.dll!FWindowsPlatformProcess::Sleep	Normal
Not Flagged	>	16712	0	Worker Thread	RenderThread 9	UE4Editor-Engine-Win64-Debug.dll!UActorComponent::GetScene	Normal
Not Flagged		5992	0	Worker Thread	RTHeartBeat 9	UE4Editor-Core-Win64-Debug.dll!FWindowsPlatformProcess::Sleep	Above Normal
Not Flagged		12124	0	Worker Thread	ntdll.dll thread	ntdll.dll!00007ffc257f316a	Normal

Here’s the callstack for the thread that crashed:

>	UE4Editor-Engine-Win64-Debug.dll!UActorComponent::GetScene() Line 230	C++
 	UE4Editor-Engine-Win64-Debug.dll!FInstancedStaticMeshRenderData::ReleaseResources() Line 651	C++
 	UE4Editor-Engine-Win64-Debug.dll!FInstancedStaticMeshRenderData::~FInstancedStaticMeshRenderData() Line 616	C++
 	UE4Editor-Engine-Win64-Debug.dll!FInstancedStaticMeshSceneProxy::~FInstancedStaticMeshSceneProxy()	C++
 	UE4Editor-Engine-Win64-Debug.dll!FInstancedStaticMeshSceneProxy::`scalar deleting destructor'(unsigned int)	C++
 	UE4Editor-Renderer-Win64-Debug.dll!FScene::RemovePrimitiveSceneInfo_RenderThread(FPrimitiveSceneInfo * PrimitiveSceneInfo) Line 553	C++
 	UE4Editor-Renderer-Win64-Debug.dll!`FScene::RemovePrimitive'::`6'::EURCMacro_FRemovePrimitiveCommand::DoTask(ENamedThreads::Type CurrentThread, const TRefCountPtr<FGraphEvent> & MyCompletionGraphEvent) Line 579	C++
 	UE4Editor-Renderer-Win64-Debug.dll!TGraphTask<`FScene::RemovePrimitive'::`6'::EURCMacro_FRemovePrimitiveCommand>::ExecuteTask(TArray<FBaseGraphTask *,FDefaultAllocator> & NewTasks, ENamedThreads::Type CurrentThread) Line 656	C++
 	UE4Editor-Core-Win64-Debug.dll!FBaseGraphTask::Execute(TArray<FBaseGraphTask *,FDefaultAllocator> & NewTasks, ENamedThreads::Type CurrentThread) Line 295	C++
 	UE4Editor-Core-Win64-Debug.dll!FTaskThread::ProcessTasks(int QueueIndex, bool bAllowStall) Line 325	C++
 	UE4Editor-Core-Win64-Debug.dll!FTaskThread::ProcessTasksUntilQuit(int QueueIndex) Line 173	C++
 	UE4Editor-Core-Win64-Debug.dll!FTaskGraphImplementation::ProcessThreadUntilRequestReturn(ENamedThreads::Type CurrentThread) Line 821	C++
 	UE4Editor-RenderCore-Win64-Debug.dll!RenderingThreadMain(FEvent * TaskGraphBoundSyncEvent) Line 220	C++
 	UE4Editor-RenderCore-Win64-Debug.dll!FRenderingThread::Run() Line 342	C++
 	UE4Editor-Core-Win64-Debug.dll!FRunnableThreadWin::Run() Line 64	C++
 	UE4Editor-Core-Win64-Debug.dll!FRunnableThreadWin::GuardedRun() Line 46	C++
 	UE4Editor-Core-Win64-Debug.dll!FRunnableThreadWin::_ThreadProc(void * pThis) Line 99	C++
 	kernel32.dll!00007ffc254416ad()	Unknown
 	ntdll.dll!00007ffc257b4409()	Unknown

And here’s the thread (the main thread) that caused the crash:

  	UE4Editor-Engine-Win64-Debug.dll!UPrimitiveComponent::GetCollisionResponseToChannels() Line 626	C++
 	UE4Editor-Engine-Win64-Debug.dll!UPrimitiveComponent::IsNavigationRelevant(bool bSkipCollisionEnabledCheck) Line 1570	C++
 	UE4Editor-Engine-Win64-Debug.dll!AActor::UpdateNavigationRelevancy() Line 2134	C++
>	UE4Editor-Engine-Win64-Debug.dll!UPrimitiveComponent::OnUnregister() Line 316	C++
 	UE4Editor-Engine-Win64-Debug.dll!UStaticMeshComponent::OnUnregister() Line 465	C++
 	UE4Editor-Engine-Win64-Debug.dll!UActorComponent::ExecuteUnregisterEvents() Line 799	C++
 	UE4Editor-Engine-Win64-Debug.dll!UActorComponent::UnregisterComponent() Line 639	C++
 	UE4Editor-Engine-Win64-Debug.dll!AActor::UnregisterAllComponents() Line 3037	C++
 	UE4Editor-Engine-Win64-Debug.dll!ULevel::ClearLevelComponents() Line 660	C++
 	UE4Editor-Engine-Win64-Debug.dll!UWorld::ClearWorldComponents() Line 1199	C++
 	UE4Editor-Engine-Win64-Debug.dll!UWorld::CleanupWorld(bool bSessionEnded, bool bCleanupResources, UWorld * NewWorld) Line 2926	C++
 	UE4Editor-UnrealEd-Win64-Debug.dll!UEditorEngine::TeardownPlaySession(FWorldContext & PieWorldContext) Line 462	C++
 	UE4Editor-UnrealEd-Win64-Debug.dll!UEditorEngine::EndPlayMap() Line 138	C++
 	UE4Editor-UnrealEd-Win64-Debug.dll!UEditorEngine::Tick(float DeltaSeconds, bool bIdleMode) Line 837	C++
 	UE4Editor-UnrealEd-Win64-Debug.dll!UUnrealEdEngine::Tick(float DeltaSeconds, bool bIdleMode) Line 252	C++
 	UE4Editor-Win64-Debug.exe!FEngineLoop::Tick() Line 2098	C++
 	UE4Editor-Win64-Debug.exe!EngineTick() Line 54	C++
 	UE4Editor-Win64-Debug.exe!GuardedMain(const wchar_t * CmdLine, HINSTANCE__ * hInInstance, HINSTANCE__ * hPrevInstance, int nCmdShow) Line 136	C++
 	UE4Editor-Win64-Debug.exe!WinMain(HINSTANCE__ * hInInstance, HINSTANCE__ * hPrevInstance, char * __formal, int nCmdShow) Line 190	C++
 	UE4Editor-Win64-Debug.exe!__tmainCRTStartup() Line 618	C
 	kernel32.dll!00007ffc254416ad()	Unknown
 	ntdll.dll!00007ffc257b4409()	Unknown
1 Like
2

In case it’s not obvious, the crash is because World is valid at the start of the ternary operation but isn’t when trying to dereference World. In other words, World is being set to NULL after the code checked that it wasn’t 0.

3

Great research and solution Neil!

:slight_smile:

#IsValidLowLevel()

What happens if you do this?

FSceneInterface* UActorComponent::GetScene() const
 {
     return ( (World && World->IsValidLowLevel()) ? World->Scene : NULL);
 }

If that solves your issue that should clarify the matter for Epic, also removing the need to create additional variable

Let me know if that fixes the matter!

:slight_smile:

Rama

4

Hi Rama! :slight_smile:

Unfortunately that wouldn’t necessarily help either because it’s a timing issue - the World variable is being set to NULL between the checks (this just introduces another check) and dereferencing it when trying to evaluate World->Scene.

The real fix would be to not be rendering when the game is shutting down and cleaning up - but that’s almost certainly going to involve a mutex/semaphore - something of that nature. That’s why I’m not sure whether this band-aid solution treating the symptom is the best fix - because I’m pretty sure Epic would prefer the renderer be as lockless as possible!

5

Hi Neil!

Always great to see you around!

#:heart:

#Please Try IsValidLowLevel()

Well did you actually try it?

Because I have experienced that a UObject pointer can pass the simple pointer test, but still crash when actually dereferenced

IsValidLowLevel is what I use in my game to get around this type of crash that can occur when objects have been recently deleted or are in the process of being deleted.

I was hoping you could check and see if the world pointer passed, but IsValidLowLevel() failed

that would be valuable info :slight_smile:

My concern is whether the World pointer is actually passing, but the dereference is causing crash for UObject Subsystem reasons, such as I have experienced in my own projects since the beta.

I would not have a project right now if I had not learned to do more rigorous checks on UObjects that might have been deleted recently or might be getting deleted at the time the code runs!

Rama

6

Hi Rama!

It’s great to see that you’re still so involved in the forums too. :slight_smile:

My concern is whether the World pointer is actually passing, but the dereference is causing crash for UObject Subsystem reasons, such as I have experienced in my own projects since the beta.

No, World pointer is NULL, the crash was because it was trying to access 0x00000178 - it gets set to NULL when the component unregisters. I really, REALLY should have posted that in the original post!

As you can see, the main thread is unregistering the component and the other (rendering) thread is accessing that component at the same time. That’s the bug! And that’s why copying the pointer works - because the UWorld object is still valid (though for how long, who knows?) but the World variable gets changed between checking it not being NULL and then attempting to dereference it.

That’s why calling IsValidLowLevel() won’t help - because it would do yet another check against NULL (“if( !this)”) and then (probably - this is all timing related!) pass the rest of the tests checking its Class and that it’s in the GUObjectArray. The only thing it would do would be to alter timing - but the bug would still remain. :frowning:

The proper fix is to stop rendering while the game is shutting down - anything else is just a band-aid solution.

7

#Great Work Neil!

“it gets set to NULL when the component unregisters. I really, REALLY should have posted that in the original post!”

“As you can see, the main thread is unregistering the component and the other (rendering) thread is accessing that component at the same time. That’s the bug! And that’s why copying the pointer works - because the UWorld object is still valid”

“The proper fix is to stop rendering while the game is shutting down - anything else is just a band-aid solution”

ooooo!

Okay!

This final post of yours will surely help Epic to track down the issue, amazing detective work Neil!

You deserve a Rainbow!

2537-rainbow.jpg

Rama

8

#Please Help Epic

Just to be clear this matter is entirely unresolved but Neil has posted extensive details now that should help this issue get tracked down

#:heart:

Rama

9

#Still Needs To Be Addressed

Now that 4.5 is out please remember to address Neil’s wonderful research and this whole matter!

:slight_smile:

Rama

10

It looks like UE 4.5 addresses this problem by cleaning up properly. Nice work!