We have game servers started by GameLift on instances running Amazon Linux and sometimes they crash with the mentioned callstack.
There’s no reliable repro but it happens quite often and I see the crashed process log followed by a normal launch as GameLift starts the game server process once again after it crashes.
The game instance class calls UAssetManager::Get().LoadPrimaryAssets(), verifies the resulting stream handle is valid and calls StreamHandle->WaitUntilComplete() (RLHGameInstance.cpp:886 in the callstack).
Is this something known with a fix we could cherry-pick from a more recent engine branch?
The crash is happening while FAsyncLoadingThread2::FlushLoading is dumping the state of the in flight packages. This is following the code timing out (see GStallDetectorTimeout). There is likely a deadlock going on or another threading issue. Can you share the state of all the threads?
I’m not familiar with the Gamelift platform. You will have to reach out to Amazon to get some help here. I’m assuming they have some some crash reporting that would contain that information.
It is possible to get the CrashReporter to generate a mini dump.
To do so, you can add:
[CrashReportClient]
bAgreeToCrashUpload=true
to $UProjectPath/Config/Linux/LinuxEngine.ini.
Upon crash, this should yield a CrashContext.runtime-xml file under $HOME/.config/Epic/$ProjectName/Saved/Crashes/crashin*/ folder. If you can share this file once reproduced, this would be helpful in further diagnosing the issue.
I’d be happy but can you please advise me on how to do that on a GameLift instance running Amazon Linux? I only see the crashes post-factum when checking rotated logs downloaded from a terminated game session.
What is the process for a plain Linux environment? How do I capture the state of all threads when the engine crashes in FAsyncLoadingThread2::FlushLoading() while booting?