Re-save all assets was a mess for me, skipped tons of files for the wrong engine version then threw me some compile errors on some BPs. please let me know how yours goes.
It didn’t work for me. Maybe I didn’t do it right, but I resaved all packages (and P4 confirmed that the files had changed) and then ran a cook (iterative, hopefully not a mistake) and then played the game again and still got the freeze. My only concern is that if somehow resaving didn’t recook stuff in an iterative cook, then I need to test it again. I might come back to this later. Still working on trying to understand the threading, as posted on the forums.
I haven’t really updated the remote server. I disabled steam at one point in the defaultengine.ini by setting bEnabled=false which I think should be the same as -nosteam? I used nosteam on the client and even went so far as to completely remove online modules from the game. I will try again though, but I’ve had sessions last like 30 minutes without a crash. I always have to have someone at least join to get the crash. This thread is getting hard to read with all these layers of boxes. Probably better to update on the forums.
EDIT: I thought I tested the server before without steam but actually I think the value on the config got reset and I didn’t realize it.
Are you uploading your dedicated server to a server away from your local machine? I’ve been testing the server with -nosteam on my local machine and have not crashed yet.
Also with the exception of Creas3D (Thanks for the attention Creas!) no one is helping us. I’ll see you on the forums! I just wanna make one more nested box of doom.
Hello everyone,
I’ll be looking into this issue. I’ve read over the responses you have all left so far and have some questions.
- Can you please provide log files for the sessions where these crashes occur?
- Have you been able to reproduce this in a fresh project or is it only happening in your primary project?
- Is this only happening in a network environment or is it possible to have it occur in an offline game?
- Since these options can sometimes cause unexpected problems, have you been using Event Driven Loading and/or Blueprint Nativization? If so, try disabling them (even if it’s not an acceptable workaround) to see if they could be related to the issue.
- There was mention of disabling ASync loading previously without any helpful results. When disabling this, did you also remove any ASync loading code you were using?
I’ll be posting on the forum to let those who are posting there know, but it would be best if we kept all of the information in one area so let’s keep the discussion here.
-
I have some log files here : Box | Login It seems that the last things loaded are always widgets, but not always the same widgets. At one point I did disable a lot of the widgets I saw in the log (but not all of them), but still froze.
-
I have not tried to reproduce this in a fresh project, it happened in the same project in 4.17, 4.17.2, and 4.18. Currently unsure how I’d even attempt to reproduce it.
-
It has occurred when running the server on a local machine and connecting to 127.0.0.1, as well as on AWS dedicated servers (EC2). It does not happen in PIE when playing directly in the level.
-
We have disabled these in the project settings/packaging settings, to no avail.
-
I did not remove any ASync code when disabling ASync options in packaging and or defaultengine.ini
It never happened in our project in 4.10 or 4.15 when NOT using steam (used Gamesparks for accounts). Our last round of development integrated steam and upgraded to 4.17, and this is when we started seeing it frequently.
I have also tried to move all particle components to post-physics tick group, we’ve tried a one liner change in LevelTick.cpp, can’t remember the line number but it changed the TG_DuringPhysics tick call to block LevelTick.cpp: RunTickGroup(TG_DuringPhysics, true); - this seemed stable but today I froze.
I think the hard part about this is that it’s a freeze and not a crash.
Thank you for your attention!
Hello ,
Can you try uploading those logs directly? I’m told that that the requested page does not exist when trying to access that link. It could be a permission issue.
-
Log file, no crash because it just hangs until I force quit.
-
I have not made a fresh project yet. I’m trying to remove things from my project first to see if it goes away. Building a fresh one seems unlikely to produce the issue if your team has not experienced it. Then I’d start having to add features at which point I’m making another game.
-
I don’t believe I’ve experienced this offline. Another poster on the forums says they have. Our game is a multiplayer only game so I don’t do much offline testing.
-
Not using blueprint nativization. I didn’t know Event Driven Loader was a thing. I just checked BaseEngine.ini and it appears its enabled by default, so I guess I should disable that and test.
-
I am going to test removing our one use of async loading today. Though I just noticed that AsyncLoadingThreadEnabled is false? Is that required for StreamableManager to actually asynchronously load?
As an aside, I still don’t know how NOSTEAM disables steam. I searched the entire solution for NOSTEAM and only find it referenced in documentation/logging.
- Our engine has been modified to log for us when it stalls for more than 30 seconds and continue regardless of if the event is called or not, you will see the point when this happens in the log, rest assured that this is the moment when the game would have otherwise frozen forever. link text
- We are trying to reproduce this in a fresh project currently, while trying to hunt down what could have caused it in the first place. This is very difficult and time consuming, but we will let you know if and when we find a good repro case.
- We can reproduce this offline.
- We have tried disabling and enabling all of these features, the results are the same.
- We have been able to reproduce this error on our current project with all of our async code removed, we are working now to determine if the assets that were async loaded may somehow be tied to the error.
I removed our use of the StreamableManager (we’re not using the one that’s new as of 4.16 or 17 if that matters). Game still froze.
Thank you all for the logs and the information. First thing I want to mention is that, while looking at the “SetSkeletalMeshNoAsyncWithAnimation” log that you provided , it seems that ASync loading was still enabled, from looking at the messages near the end of the log.
From comparing the logs from all three of you, the only thing I can see that may be related is that you’re all freezing at a point that is related to rendering. Also, if these freezes are related and if can reproduce this offline, then we know that Steam being on or off does not matter.
This message from one of logs could be indicative of the cause for the issue:
[2017.11.03-23.53.43:789][795]LogTexture: Hitch ( 22.8ms) in waiting for async IO in FTexture2DResource::BlockTillAllRequestsFinished; this can happen, especially on desktop platforms. If it happens a lot, or on consoles, that would be indicative of a bug in the texture streamer.
I’m going to continue looking into this to find out more, but wanted to let you know my initial thoughts.
Yeah on our end I’m pretty certain steam is not related. I’ve frozen with it disabled on the client and server. As for it being related to the render thread, I should note that when its frozen the render thread does appear to still be looping during the freeze. The freeze is in TG_EndPhysics.
As far as I could see, the freeze happens when waiting on the “DuringPhysics” TickGroup when processing the “EndPhysics” TickGroup.
Inside of “ReleaseTickGroup” for the TG_EndPhysics group, it calls
FTaskGraphInterface::Get().WaitUntilTasksComplete(TickCompletionEvents[Block], ENamedThreads::GameThread);
Where “Block” is “TG_DuringPhysics”. Looking into the “TickCompletionEvents” for Index 2 = TG_DuringPhysics, I can see a length of 26.
https://puu.sh/yim3q/f58196e872.png
It then proceeds to “Wail Until Tasks Complete”, by passing these Tasks for “DuringPhysics” TickGroup. So I guess it’s insight there. I wonder if any of you have different numbers for the amount of tasks. Would help me find out if it’s our actors or something engine internal.
Well and then it checks the tasks if they are complete or not. If it finds one non-complete task, it fires a Thread for “Process Tasks Until Quit”
This then calls "ProcessTasksNamedThread for each member of “Queue”, passing the Queue Index (which is the GameThread Enum passed ealier?!). So I guess that’s the GameThread queue. It then “pops” a member from the “StallQueue” , which seems to be invalid, and then running into “bAllowStall”. “bAllowStall” is driven by: FPlatformProcess::SupportsMultithreading()
I don’t know if disabling Multithreadding would do anything, but I hardly believe that’s the proper way to solve this issue.
I sadly don’t know which of the tasks fails, as that is obviously “optimized away”.
https://puu.sh/yimll/56823838fa.png
So yeah, finding out what task from the “DuringPhysics” group is doing bad things would be the best, or?
PS: Why is this a thing?
/** Event that this thread blocks on when it runs out of work. **/
FEvent* StallRestartEvent;
So the Thread that was started to clean up the DuringPhysics stuff runs out of work and then blocks? Isn’t that supposed to be a different thread than the gamethread?
And if not ,why the hell would the gamethread block itself when it runs out of work?
Can’t it just idle? 
So “bAllowStall” is driven by “SupportsMultithreadding” which then says “let’s block this thread as we are multithreadding” and I assume we either are not OR this is the gamethread which should not allow that?
It’s waiting for other tasks to complete. So the thread doesn’t continue when its out of work until other tasks are complete, otherwise async things that are only supposed to happen within a specific tick would now happen outside that tick if the game thread finished early.
Fair enough, but I assume we have no way of finding out what exact task is not completing or?
Also, this is the GameThread that is calling the block, right?
So other threads are still running and not completing?
Wouldn’t that technically mean non of the GamePlay related things in our projects are causing this?
Thanks for the update,
the logs I provided had the async loading thread and EDL enabled, the ‘noasync’ part of my naming convention was indicating that the skeletal meshes we were testing had not been loaded via UAssetManager::GetStreamableManager().RequestAsyncLoad(), which was code we initially suspected in the block of our code we believe we have narrowed the freeze down into in our character.
sorry for any confusion, we are still doing what we can to track this down on our end.