UWorldPartitionSubsystem::RegisteredWorldPartitions sometimes randomly contains invalid objects on multiplayer clients

Development World Creation
, ,
1

First, sorry that I do not have working reproduction steps. I am unable to reproduce the crash myself…

Also, the reported crashes are for Test and Shipping builds, so they do not contain assertions, which would maybe have caught the issue sooner, with a different call-stack…

The issue we are seeing is that rarely the game crashes for our multiplayer clients, because their UWorldPartitionSubsystem::RegisteredWorldPartitions array contains invalid entries. Some entries are valid, others look like they had been valid once but partially overwritten, and others are complete garbage.

The crash happens in this loop:

for (UWorldPartition* RegisteredWorldPartition : GetRegisteredWorldPartitionsCopy())
{
  if (RegisteredWorldPartition->StreamingPolicy)
  {
    RegisteredWorldPartition->StreamingPolicy->UpdateStreamingState();
  }
}

Is something like this known?

Could it be related to [Content removed] (which has assertions enabled -> maybe the cause for the different call stack)?

If not, do you have any suggestions on how it might be possible to debug it from a (full) memory dump?

Thanks in advance for any help on this,

Andreas

2

Steps to Reproduce
Sadly I do not have reproduction steps. The issue happens too infrequently, so we could not narrow it down yet. We never managed to intentionally reproduce it…

The issue has been reported to us in the following 2 scenarios - but, as said, the crash happens so infrequently that these scenarios might be random:

  • Scenario 1:
    • In a multiplayer game with more than 2 players
    • A client joins the game while another client is teleporting to a different location
  • Scenario 2:
    • In a multiplayer game with more than 2 players
    • All players get teleported, but the host teleports away immediately again
      • The location to which the host teleports triggers the loading of a DataLayer

The crash has only been observed in a packaged build.

3

Seems the call stack was lost when composing the question. Here it is again:

UWorldPartitionSubsystem::UpdateStreamingStateInternal(const UWorld* InWorld, UWorldPartition* InWorldPartition) Line 1243 + 3 bytes
non-virtual thunk to UWorldPartitionSubsystem::OnUpdateStreamingState() + 23 bytes
UE::Core::Private::Function::TFunctionRefCaller<::__unnamed_type_01628a4a,void,UWorldSubsystem*>::Call(void* Obj, UWorldSubsystem*& Params) Line 321
[Inline Function] UE::Core::Private::Function::TFunctionRefBase<UE::Core::Private::Function::FFunctionRefStoragePolicy,void(UAudioEngineSubsystem*)>::operator()(UAudioEngineSubsystem* Params) Line 470 + 2 bytes
[Inline Function] FSubsystemCollection<UAudioEngineSubsystem>::(stripped function)::__unnamed_type_df4625d4::operator()(USubsystem* Subsystem) Line 165
[Inline Function] Invoke<::__unnamed_type_df4625d4&,USubsystem*>(__unnamed_type_df4625d4& Func, USubsystem*&& Args) Line 47
UE::Core::Private::Function::TFunctionRefCaller<::__unnamed_type_df4625d4,void,USubsystem*>::Call(void* Obj, USubsystem*& Params) Line 315
FSubsystemCollectionBase::ForEachSubsystemOfClass(UClass* SubsystemClass, TFunctionRef<void(USubsystem*)> Operation) Line 137	
[Inline Function] FObjectSubsystemCollection<UWorldSubsystem>::ForEachSubsystemWithInterface<UStreamingWorldSubsystemInterface>() Line 239 + 5 bytes
UWorld::InternalUpdateStreamingState() Line 4278
UWorld::Tick(enum ELevelTick TickType, float DeltaSeconds) Line 1602
UGameEngine::Tick(float DeltaSeconds, bool bIdleMode) Line 1799
FEngineLoop::Tick() Line 5869
MainFunc(void* ArgsPtr) Line 313
[Inline Function] tchar_main(int32 ArgC, TCHAR** ArgV) Line 366
main(int32 ArgC, ANSICHAR** Utf8ArgV) Line 329
_start + 63 bytes	Unknown	Symbols loaded.
4

Hi Andreas,

Unfortunately, this issue was never reported before (that we are aware of).

At first glance and without any more info/steps to reproduce, it will be difficult to help.

Perhaps you could add logs when a worldpartition is added/removed to/from the RegisteredWorldPartitions list (log the worldpartition address).

Log the list and the address of each worldpartition pointer.

Richard

5

Good news! I’ll go ahead an mark this one as closed. Feel free to re-open it if you still have the issue.

Cheers,

Richard

6

Sorry for the late answer, but I have been waiting for a QA retest with CL 46949279 cherry-picked.

Again, sorry for asking this question without more information available, and thanks for looking into it despite that.

The good news is that with CL 46949279, our QA is no longer able to reproduce the issue. Since it was a very rare crash, this is not 100% guaranteed, but assuming that QA weren’t just very unlucky, this seems indeed to be the same issue as discussed here (but with a different call stack since it was a shipping build):

[Content removed]