I have finally frozen using -sm4, but my dump file is corrupted… Both dumps made with -sm4 have been corrupted and cannot be debugged.
I too seem to only be suffering from this when in the main menu only. So it might be related to UMG?
I have experienced the freeze once when testing the project provided by MichaelArchetype. This occurred while I was on the Main menu, while switching between a few widgets, so it is likely related to Widgets. At this point I’m not exactly sure how to proceed as this seems like a niche problem that could be related to converting a project from a previous version as well as a particular kinds of set up. Due to this, I’ll be escalating this issue to someone more specialized in this area.
I’m really happy that you are looking into this, but I also have to make clear that users experience this differently.
We do NOT get the freeze in the MainMenu in our project.
We get it in the GameLevel, after connecting to our DedicatedServer. And we can simply sit in the lobby, without changing any Widget and it’ll freeze eventually.
Also another user in this question explained that they went through removing all widgets and it still froze. So I don’t direclty believe it’s only widget related.
And even if it is, then it’s deep in the engine.
We were actually able to reproduce the issue in Allar’s Generic Shooter, who also answered with “Somewhere deep down in the Engine.” We would really need someone from the Engine devs on this who can properly look into the low-level thread handling. Our project constantly shows the libOPUS library and DX11 when freezing and making a dump file.
As would I. This freeze has stopped development in its tracks. We’ve been at a brick wall for a month now. We can’t move forward with business until this is resolved and I’m willing to do anything to resolve it.
I’d like to stress that this is blocking us from releasing builds for our game that is already on Steam Early Access.
I just want to add my voice to the conversation and say that this has also been blocking us for a month now.
( and co).
And that we have serious doubts as to it being widgets, as we have gotten the freeze with 0 widgets in the level.
As we got our freeze on the DedicatedServer + full map and gameplay classes, we tried going back to a simple empty level. That one doesn’t freeze, at least not for the past 2 hours.
We now try to add back the GameMode to that empty level, so nothing but the default skybox and one widget. If this freezes then I can at least cut out the map and its actors and reduce the search to the GameMode classes and the widgets.
Hi Matthew and all,
After hundreds of different soak test, each tweaked mildly, I believe I have found a conclusion as to what was causing this bug in our project.
We have a class in our game that inherits from USkeletalMeshComponent, and in it’s begin play it sets the tick group. With this single line removed we have yet to have a successful reproduction after about 4 hours of soak.
void UAvatarPart::BeginPlay()
{
Super::BeginPlay();
SetTickGroup(TG_PostUpdateWork);
}
Tests aren’t still entirely conclusive, as we would like to soak for another day or two, and reproduce it in a fresh project with this code, but have not had ample time to do so as of yet.
I just thought I would throw this out there for any interested parties, maybe we can get some confirms or denies from the other developers on here.
We only assign tick groups in the constructor and they’re all TG_Pre and Post Physics.
Alright, since the issue seems to be caused by different factors for each of us and we kinda loose track of the right answer in this horrible, small AnswerHUB layout, I will now try to update this Answer with the different solutions we find.
The Issue
To repeat the issue and not loose any information in all those answers.
The “issue” is a complete stall of the GameThread. Audio seems to keep running, but the game appears to be frozen.
When creating a dump file and debugging it in Visual Studio, all of us seem to see the same stall with “FEventWin::Wait”.
This is most likely happening when the “TG_EndPhysics” TickGroup is processed and then the previous “TG_DuringPhysics” TickGroup is cleaned up. The specific function being called for this is:
FTaskGraphInterface::Get().WaitUntilTasksComplete(TickCompletionEvents[Block], ENamedThreads::GameThread);
I haven’t heard from all of you yet if the exact same TickGroup is being process at this point.
Furthermore, this brings us to the idea that some event in the “TG_DuringPhysics” TickGroup is not completing and the GameThread stalls itself, as it waits on it and has nothing else to do.
When does this happen?
So this seems to be already different for some of us. Here are the different cases:
- Not inside the MainMenu. Only Packaged Game. Only when joining a DedicatedServer. Doesn’t need further interaction, just waiting until it freeze.
- Already inside of the MainMenu.
- ?
Possible Reasons
- Widgets
Widgets seem to be something Epic is targeting for this, however multiple factors speak against it.
One user removed all widgets and still had this happening.
I actually created an empty map, with a rotating block (to test if it freezes), which had all our UI (GameMode etc) in it, and it DIDN’T freeze. So our issue is inside of the map itself.
- Steam
Steam is probably completely out? Don’t know why it would cause this, so if you are running Steam: We couldn’t find a reason why Steam would cause this.
- libOPUS Library / DX11
While debugging the dump files with Windbg, we found that the last call stack is actualy listing some libOPUS functions, as well as DX11. Refreshing the library or playing in DX10 did NOT resolve the issue.
- TickGroup
A wrongly set TickGroup on a broken or non-broken object might cause this. It’s not clear if this is specific to Meshes (check Solutions) or can be caused by any kind of ticking object that has the TickGroup set by hand. It’s also not clear WHY this would cause the freeze.
Solutions
Now this list will be a bit shorter, as we are still investigating.
-
TickGroup was set on a child of SkeletalMeshComponent.
void UAvatarPart::BeginPlay()
{
Super::BeginPlay();
SetTickGroup(TG_PostUpdateWork);
}
Removing the SetTickGroup line seemed to have resolved the issue for the user.
- Particles
As we are currently testing, it seems like removing several particles from our map (which seems to be the cause of the freeze) seems to resolve it. We are yet to find out which particle causes this and then hopefully see WHY it causes it.
- Epic posted a POSSIBLE fix: [GitHub][1]
Make sure to be logged in into GitHub and have it connected to your Epic account, otherwise it will give you a 404 error. The fix hasn’t been tested by everyone yet, so it can be that it’s only a solution for one of us.
I’ll update the answer once this is more clear.
In addition, here is a link to the new [Bug Report][2]
This comment is now just me explaining what we are currently trying to solve this.
Our current approach to solve this is by cutting away possible reasons. The first big divide we wanted to make is GameMode, UI, GamePlay classes (Character, Controller, etc.) and the Map.
We created empty levels, without the GameMode, just with a rotating platform.
→ NO FREEZE
Then we added the GameMode to that empty level, which gaves us back all classes of that GameMode and our UI.
→ NO FREEZE
Other devs of our project went ahead and removed all particles from the actual map.
→ NO FREEZE
Our next step is to find out if it’s really caused by some of our particles (or its actor?).
So I’ve got some updates on the solution too I think… One of our levels is being used as a background(level stream) for the main menu with a camera flying through, I removed that and the main menu doesn’t seem to freeze anymore. Playing on that map used as background map is not freezing the game either. Might have something to do with matinee cameras? I don’t know.
Interesting, have you tried removing them from the running code sample?
Our current scene is an empty map with a light source and some basic shapes for spatial context, we have removed all of our game code aside from the character, which is how we boiled it down to this point.
How long do you guys run your soak tests for? We usually do ours overnight. There have been many times a build did NOT freeze in the first couple hours, but did freeze later.
The time seems to be just random, so I would not rule out a build as not freezing, unless you’ve ran it for a good looong while.
My empty map test and another test with the main map minus particles were both over night.
Both no freeze. Our game usually freezes within the first hours. We are currently trying to figure out which actor causes it.
Hi again all, all of the fixed soaks (6 of them) are still running without a freeze after more than 12 hours now and we have managed to reproduce this in a new project created from a binary distribution of 4.17.2 : 5/6 tests failed within the first 2 hours!
I would highly recommend everyone experiencing this issue try removing actor/component code or assets changing tickgroup from their tests to see if they can get clear of the freeze.
It is still possible there are other causes, but I am really hoping that this information helps some if not all of you.
I have provided . with a link to download the test project in 4.17 so that Epic’s engineers can play with it.
Seems like one of our testers froze during gameplay this time, 10 minutes in.
Hey Matthew,
You’ve never responded to me at all. I’m pretty sure I provided my project before , as well as the package + debug files. I’m actually willing to pay for some focused support. Is there any kind of program that Epic can provide for some level of proper support?
Hello everyone,
The issue has been escalated and is being looked at by multiple engineers. I’m personally taking a look at the project that provided that was mentioned to have a 100% repro. Sending private messages simply for updates will not speed this along, please keep those messages to only sending confidential information when requested.