Running with wp.Runtime.UpdateStreaming.EnableAsyncUpdate=true can make the AsyncUpdateStreamingStateTask run during the GC phase as its .Wait() call happens on the next Tick - and crash in ULevelStreaming::GetLevelStreamingStatus()

Development Platform & Builds
, ,
1

This is very similar to [Content removed] - but we do believe it can also happen when not using incremental GC, just much less often. The culprit is that AsyncUpdateStreamingStateTask is started on a certain frame, but it waits on the next frame, where GC can also happen through-out. When ULevelStreaming::GetLevelStreamingStatus() gets called, the following call:

UPackage\* LevelPackage \= FindObjectFast\<UPackage\>(nullptr, GetWorldAssetPackageFName());

Will trigger with “Illegal call to StaticFindObjectFast() while garbage collecting!”.

We did notice this crash because we are using incremental GC on our project, and we intend to ship with it - but as I said, I don’t think incremental GC is the actual problem, but the fact that the task can bleed-out from the same frame.

[Attachment Removed]

2

Steps to Reproduce
* Run with wp.Runtime.UpdateStreaming.EnableAsyncUpdate=true

Run around that does streaming updates with GC happening

* Using gc.AllowIncrementalReachability=1 and gc.AllowIncrementalGather=1 makes it happen much, much more as GC runs on many frames

[Attachment Removed]

3

Hello!

We strongly recommend that you stop using the incremental reachability analysis and the incremental Gather. Those are highly experimental features that are not proven to be thread safe. The only use case that we have validated is with a single thread server.

The comment on AsyncUpdateStreamingStateTask is still valid and I’ll have the engine team review that part of the code.

Regards,

Martin

[Attachment Removed]

4

Hello!

We added a guard to prevent the Async Streaming state update from running concurrently with the GC. CL#49409329

Regards,

Martin

[Attachment Removed]

5

I’m not sure what is happening with the mirroring. Here is the change. In WorldPartitionStreamingPolicy.cpp:

		{
			// Create async tasks
			TRACE_CPUPROFILER_EVENT_SCOPE(UWorldPartitionStreamingPolicy::CreateAndDispatchAsyncTasks);
			FUpdateStreamingStateParams InputParams = FUpdateStreamingStateParams(this, AsyncTaskCurrentState).SetRequiredWorldDataLayersEffectiveStatesCopy(true);
			AsyncUpdateStreamingStateTask = UE::Tasks::Launch(UE_SOURCE_LOCATION, [InputParams = MoveTemp(InputParams), this]()
			{
				QUICK_SCOPE_CYCLE_COUNTER(STAT_WorldPartitionStreamingPolicy_AsyncUpdateStreamingState);
				
----->				FGCScopeGuard GCGuard;
				UWorldPartitionStreamingPolicy::UpdateStreamingStateInternal(InputParams, AsyncTaskTargetState);
 
			}, UE::Tasks::ETaskPriority::Normal);
			AsyncUpdateTaskState = EAsyncUpdateTaskState::Started;
		}

[Attachment Removed]

6

Alrighty ! Although the github syncs seem to not be working today, I don’t see it and all commits are at least 24 hours old.

[Attachment Removed]