I feel like there should be a way to find out what task we’re waiting on but I have no clue how given the information I (and you) obtained from the debugger.
Yes main thread is waiting. Other threads are running. I wouldn’t say none of our gameplay code causes this, but its probably not our fault. While we have full access to source, UE4 still sometimes feels like a black box and behaves mysterious in response to things we do that maybe Epic doesn’t do in QA or in their projects or doesn’t expect.
Also, did you happen to upgrate from pre 4.17 to 4.17 ?
Because we had tons of blueprints issues, like variables resetting or not showing the correct values. Widgets in the package build sharing all the same reference and reacting to the same hover event and all that ■■■■.
Which makes me think there is something else that also got broken.
If you also updated your game, we might have that in common?
We upgraded from 4.14 to 4.17. The only major issue I had was in Slate’s OnPaint. Everything else, for the most part, just worked. We didn’t introduce any new code when I did the ugprade. I just updated the engine on the last stable build of our game, and fixed mostly basic issues that cropped up. Then I cooked and packaged like normal and our QA was able to test the build just fine (and discover the freeze).
Although this is kind of a shot in the dark, but it’s all I can think of at this point with the information given, have you tried temporarily removing your widgets to see if it is content related? As eXi mentioned, this could be content related which would explain the issues with trying to reproduce it outside of these few cases. I understand if it will be difficult to temporarily remove them but it would likely help narrow down if this problem is content related or not.
Is there anyway you or someone can walk us through debugging these threads/tasks? Is there some sort of job graph GUI or something I can do with STATS since that tracks EventIds? Like I can and will disable all widgets to test. I’m pretty sure I can still play my game without any widgets using the command line. But it does seem unlikely to reveal anything (or enough).
Also we could give you or someone a P4 account potentially to setup a workspace on our project and test for yourself.
The idea is that if the widgets are what is causing your issue, due to having issues from the conversion, the freeze will not occur when they’re removed.
Having the project to test with would be helpful, if you would like to provide it, you can share what is necessary through a Private Message on our forums. This link should take you directly to a form to send me a private message.
I really don’t think we can remove ALL widgets and still be able to start the game without some major modifications to the game mode. Maybe I’ll look into attempting it, and I can at least remove all superfluous widgets. However, we’ve frozen just sitting in the “Ready Up” lobby while not even playing (though the level was still loaded and running).
It would be great if I could find a way to just see what is passing the unsigned int or what it’s actually waiting on indefinitely. I’ve learned a lot about debugging this month, but not enough I guess.
The conversion from 4.14 to 4.17, from looking at your logs, the freeze seems to occur while the widgets are being loaded, which is why I’m assuming it may be related.
What conversion? I had the issue both before and after a re-save of all content. All widgets behave normally afaik. I don’t have exi’s issues and we have a ton of widgets.
I have slowed my debugging to focus on other issues while Matthew gets setup on our project.
Looked around the code a bit. Doesn’t seem like this is used anywhere. Maybe a dependency for DX11. This is basically entirely a thirdparty stack, so I feel like its unlikely that its the cause? I’ll have to look out for this when I go back to debugging the issue.
I have taken the time to remove all widgets from our project (with the exception of the default HUD debug canvas), and we are still able to reproduce the freeze.
Hope this helps. Tests are still wildly inconclusive on our end.
After seeing aboves (’ post) debug, I went ahead and renewed the libOpus library.
That had no effect either. However as DX11 is listed in the call stack, we tried to package our project for DX10 and DX11 and use the -SM4 command.
We had 2 hours of non-freeze and then closed the game on our own.
No time yet to continue testing, but maybe you peeps could test that on your end too? (:
I should’ve suggested this before but can all of you who have experienced these freezes please try checking Windows Event Viewer to get the “Exception Code” for these freezes? Even if UE4 is not throwing an error, Windows should be picking up on it. You can find documentation of how to get this information here.
MichaelArchetype has provided me with their project and I’m currently trying to reproduce the freeze but have not at this time. I do however receive a Windows error when closing the game, consistently, and decided to check Windows Event Viewer to see what happened.
I was given the exception code 0xc0000005 which led me to this page. It mentions this particular issue is related to NetworkItemFactory.dll and is due to a race condition, which could explain why the freezes you’re experiencing are inconsistent, if it is indeed the same exception. It would be helpful to know if you are all getting the same exceptions.
We’ve actually looked for, and watched for in real time, windows events in the past. I’ve never found an error or warning that coincided with a freeze’s time stamp. We never actually receive an exception though.
I’ve played 4 hours total, two hours at a time, with no freeze using -SM4 launch option. Will continue testing.
I’m not sure what exception you’re getting. The game may have its own bugs. The freeze should be easy to reproduce though. I can join the server at the same time as you and we can run some tests to try and reproduce it while you have a debugger attached.
Hello guys. I have introduced UMG Widgets to my project very recently. Simple stuff, few widgets bound to methods. I have currently removed them from my viewport and I’m not experiencing freezes. I have still to test. I will let you know if the freezes come back after I reintroduce them. Unfortunately, they seem kinda random and I can’t really be sure. I don’t want to point the “investigation” in the wrong direction but decided to share this, just in case. An update will follow later today or tomorrow.
We encountered the same or a similar issue. At first we thought it was connected to the physics substepping but reading here it would seem that enabling substepping only heightens the chance of a freeze.
The packaged version freezes regularly in 4.18 / 4.17. With enabled substepping this happens anywhere from a few seconds to a few hours (most often during the first 30 minutes).
We could also observe that using messaging with connected session frontend would prevent a freeze and one can even wake up a frozen session (e.g. by connecting the frontend tool).
All in all it seems only to freeze if there isn’t much else to do than rendering / updating the render scene.
In most instances it looked like the gamethread is waiting on a task trying to execute TStaticMeshVertexData::GetResourceArray(), but there were other instances with different tasks (not in the example project).
As a workaround we’ve set TG_DuringPhysics to block till complete (in LevelTick.cpp: RunTickGroup(TG_DuringPhysics, true) which reduced the chance of a freeze drastically.
which reduced the chance of a freeze drastically.