Intermittent crash on shutdown while cooking

Development Pipeline & Plugins
, ,
1

Hi!

We’re running into an intermittent cooking crash in the editor, about 3% of the time. The cook finishes successfully, but the editor crashes on shutdown due to a failed check(). We began to notice this crash about 8 months ago, and think it may have begun after we upgraded to 5.7, but are not sure.

The call stack for this failed check:

```

UnrealEditor-Core.dll!FDebug::CheckVerifyFailedImpl2(const char * Expr=0x00007ff96d6c42e8, const char * File=0x00007ff96d6268a0, int Line=209, const wchar_t * Format=0x00007ff96d624a74, …)

[Inline Frame] UnrealEditor-Core.dll!TLockFreeFixedSizeAllocator<65536,64,FThreadSafeCounter>::{dtor}()

[Inline Frame] UnrealEditor-Core.dll!FPageAllocator::{dtor}()

UnrealEditor-Core.dll!`FPageAllocator::Construct’::`2’::`dynamic atexit destructor for ‘PageAllocator’'()

UnrealEditor-Core.dll!dllmain_crt_process_detach(const bool is_terminating=true)

UnrealEditor-Core.dll!dllmain_dispatch(HINSTANCE__ * const instance=0x00007ff96c7b0000, const unsigned long reason=0, void * const reserved=0x0000000000000001)

UnrealEditor-Cmd.exe!00007ff7b8ba8378()

```

This is reporting that an FPageAllocator still has allocated memory, and there may be leaked/dangling pointers.

We were able to reproduce this in a build with the “memory” trace channel enabled, allowing us to investigate with Memory Insights. With that, we were able to see that there were 4 active FPageAllocator allocations using the MemStack LLM tag prior to the crash. 3 of these are in rendering code, while 1 is with a NetSerializer registered with Iris. We can’t be sure the NetSerializer is the source of the assert, but it seems likely because from the reading the code, it seems like it should have been freed earlier.

The callstack for this allocation is in an attached screenshot (I couldn’t figure out how to copy full callstack from Memory Insights)

The net serializer portion on the callstack is: C:/Specter//Engine/Plugins/Runtime/GameplayAbilities/Source/GameplayAbilities/Private/Serialization/MinimalGameplayCueReplicationProxyNetSerializer.cpp (235)

The allocation ultimately comes from FPersistentNetDebugNames::CreatePersistentNetDebugName().

We have not been able to reproduce multiple instances of this with Memory tracing enabled, so we haven’t been able to confirm if other NetSerializers show up as well.

Are you aware of any crashes like this? Do you have any suggestions or ideas for how we should continue to debug this? If there’s any more information, files, or data that would be helpful, please let me know!

Thank you!

FullTraceCallstack.png(116 KB)

2

Hello!

We addressed a crash on that assert in CL46623882 (GH Commit). This was included in 5.7.1. Could it be your problem?

It might appear unrelated but the callstack from Insights can be misleading. The allocation is a page in of the PageAllocator system. The callstack is from when it was reserved from the system and the net debug system is the first user of the page. The page was license recycled\reused by another system which would not leave info at the Insights trace level.

Regards,

Martin

3

Hey Martin,

We are on UE 5.7.4 but I did double-check to make sure that the change you referenced was actually merged successfully and I confirmed it’s all there in our project.

edit: I forgot to note, DJ and I work at the same studio. Sorry if that was confusing lol!

Thanks!

4

I found a shelved CL that contains code to instrument the PageAllocator allocations. It was used in the investigation that lead to the CL I shared.

It should be lighter than collecting Insights traces so I’m guessing you might be open to try it. The extra code collects the stack when the pages are allocated and dumps the stacks of the pages that are still in use when the PageAllocator is destroyed. That would give the real culprit of the leak vs Insights tracing.

Martin

MemStack.cpp.txt(12.4 KB)

5

Hey Martin,

I had to tweak the instrumentation because the first repro attempt didn’t actually contain the instrumentation since the crash was happening after the log file had already been closed prior to UE_LOG attempting to write from what I could tell. Please let us know if there’s any additional information you may need.

Thanks!

cook_diagnostic.log(7.56 KB)

6

I can’t find how those can end up leaking. Maybe this logging is also reporting the first allocation in a page and it’s something else that is keeping it alive.

I have a feeling this is related to the order of destruction\shutdown of the different system. There are some parts of the code that are not deterministic during the shutdown phase of the engine. In this case, I think that removing the assert that fails would be fine. This is happening after the main has completed and the static systems are being destroyed. This makes no difference at the cook level. It’s a shame to fail a cook for this.

I’ll keep on investigating in case I missed something.

7

Thank you Martin! Seems like this might be useful code to keep around with a compile-time switch in the future :slight_smile:

I’ll give this a shot and report back with more information.

8

Thank you! Non-deterministic shutdown order makes sense as a possible source of this issue.

I changed the assert to `check(GetNumUsed() == 0 || IsEngineExitRequested());`. We will report back next week, or earlier if we see crashes.

9

No issues seen in the past week since this change.