Garbage Collection leading to massive frame drop

Development Platform & Builds
, ,
1

This question was created in reference to: [Freeze on one frame for Garbage [Content removed]

Analysing the trace using contextswitch also exposes the fact that much time is “lost” in thread contention:

[Image Removed]

We are in the process of upgrading to Unreal Engine 5.8.2.

Is there any performance gain to expect from the upgrade ?

The referenced post suggests that using IOStore capability. What is the expected gain from this move ?

Last but maybe not least, I could not find any tutorial / documentation regarding iostore / asset container and so on, …

Is there any publicly available ?

Thanks,

Basile

[Attachment Removed]

2

Steps to Reproduce
Hi,

Within our flight simulation application, we are suffering from massive frame drop every time the garbage collector triggers.

Below is an example from the traces :

[Image Removed]

As demonstrated the task is taking more than 100ms overall with 85ms dedicated to a task which is parallelized over 32 cores on very powerful CPU.

Can you elaborate on the possible cause for this ?

[Attachment Removed]

3

Hello!

I was able to find that the source of this problem is related to update of the linkers when using the older code paths (Pak file and loose files). The update needs to lock a global resource which explains the worker waiting. You will need use IOStore to avoid this update. IOStore is the newer data format that offers better loading performance and uses different code paths than Pak\loose files. The feature is toggled on\off through the project settings.

You can use this article as a reference to side-loading content: https://dev.epicgames.com/community/learning/knowledge\-base/44nr/creating\-platform\-agnostic\-dlcs\-in\-unreal\-engine

Regards,

Martin

[Attachment Removed]

4

Thanks Martin for the quick feedback.

Based on my discussion with the engineer working on it, we should have results next week on this. Crossing our fingers this will lead to visible improvements us.

Best,

Basile

[Attachment Removed]

5

Hi Martin,

As I said, we hope to have results this week on the IOStore usage.

For the background, we are rendering a ndisplay synchronized (6 nodes) cluster and for now, GC is leading to frame loss. This is even worse with ndisplay as each node is triggering GC on its own… As a consequence, we are also trying to take control over the GC trigger. We implement some code, quite straight forward, to trigger GC based on deterministic frame counts rather than time / resources unloading.

Unfortunately, it appears that the engine has some controls over GC delays :

The below shot shows that even if GC is manually trigger at the same frame, we fell in a situation where each node in the cluster processed it differently !

[Image Removed]

Diving into the code, it seems that even if GC is desired, the player controller might defer its actual execution:

void APlayerController::ClientUpdateLevelStreamingStatus_Implementation(FName PackageName, bool bNewShouldBeLoaded, bool bNewShouldBeVisible, bool bNewShouldBlockOnLoad, int32 LODIndex, FNetLevelVisibilityTransactionId TransactionId, bool bNewShouldBlockOnUnload)

{

…

// If we’re unloading any levels, we need to request a one frame delay of garbage collection to make sure it happens after the level is actually unloaded

if (LevelStreamingObject->ShouldBeLoaded() && !bNewShouldBeLoaded)

{

GEngine->DelayGarbageCollection();

}

Now the questions :

  • Is there a prebuilt way to synchronize GC triggers across the cluster ? I suspect not but I would not want to reinvent the wheel…
  • Is this delay required (for stability reason for instance) or simply desired (it makes more sense to collect garbage after resources are indeed unloaded) ? Since we are using a modified build for the engine, we might simply drop the delay line if there is no stability risk involved.
  • Based on the code review, it seems that only “level” / map unloading would be considered for delays. Can you confirm this reading and that WP and other streamable content are not applicable ? In that case, we can also neutralized our level subloading strategy to ensure GC won’t be delayed.

In a perfect world, using iostore, the GC spike will be small enough not to bother for all this but I am not confident…

Thanks,

Basile

[Attachment Removed]

6

Hi,

The code from the PlayerController has been like this as far as I can see. In your case, you will never hit it because you are running a WorldPartition level (UWorldPartitionLevelStreamingDynamic::CanReplicateStreamingStatus returns false).

As you could see, there is no prebuilt way to synchronize the GC across an nDisplay cluster. You won’t have to deal with a delay so it is fair to force the GC to run through a cluster event.

It will be interesting to review the reachability analysis once you are on the newer release and running IOStore. The phase will still be “bottlenecked” by the number of UObjects in the system. You can raise the verbosity level of the Garbage channel to get stats in the log. In the console:

>Log LogGarbage verbose
 
//Example from local editor
Collecting garbage and resetting GC timer.
LogGarbage: Collecting garbage
LogGarbage: Verbose: 1.058098 ms for MarkObjectsAsUnreachable Phase (57246 Objects To Serialize)
LogGarbage: Verbose: 37.807900 ms for Reachability Analysis
LogGarbage: GC Reachability Analysis total time: 38.98 ms (38.98 ms on reference traversal)
LogGarbage: 38.98 ms for GC - 16078 refs/ms while processing 626667 references from 69004 objects  with 0 clusters
LogGarbage: Freed 65536b from 16 GC contexts
LogGarbage: 0.177000 ms for Gather Unreachable Objects (2868 objects collected / 71870 scanned with 126 thread(s))
LogGarbage: 1.689300 ms for unhashing unreachable objects (2868 objects unhashed)
LogGarbage: GC purged 2868 objects (71870 -> 69002) in 47.135ms (1 iteration(s))

Regards,

Martin

[Attachment Removed]

7

Hi,

While I am indeed confident that world partition tiles streaming won’t be the cause for the delay, I have more concerns with regards to level streaming :

[Image Removed]

We are using an internal strategy for level (maps) loading (these levels do contain WP inside) hence the question.

Not my code but, as far as I can see, this should be unloading the level ;

[Image Removed]

If you say, this code should not be reached, then we need a formal debug as of why the GC are actually being executed on different frames on each cluster node. We already verified using a bookmark that the GC requests are correctly synchronized, …

Thanks for the log verbosity and pointers.

As I said, I hope to consolidate the results this week.

Basile

[Attachment Removed]

8

Hi,

To keep you posted, the iostore format did truly something since the time was somehow divided by 4 which is quite significant:

[Image Removed]

Still, the task is taking much time and we will have to review the content in more details looking for rework / optimizations…

That being said, it also implies that synchronization will be needed in the meantime and I could not identify what could be causing the delay in GC trigger. We are calling synchronously :

GEngine->ForceGarbageCollection();

It should so be starting at the end of the current frame. In my local tests, I could neither break into the delay I mentioned earlier. Also, reviewing the code, I saw that failing to acquire the GCLock may delay the GC but I did not fail any time to acquire it. It may be platform specific. More investigations still needed.

Any clue ?

Thanks,

Basile

[Attachment Removed]

9

Hi,

I have some good news for you. The bulk of the time for the reachability is spent in validation code. That code is turned off in the Shipping target. You can get rid of the validation in the Development target by adding the following launch arguments: -NoVerifyGC -DPCVars=s.VerifyUnreachableObjects=0.

You can force the cvar off by adding the following under ConsoleVariables section in DefaultEngine.ini.

[ConsoleVariables]
s.VerifyUnreachableObjects=0

The 2 PreGarbage delegates would need a sampling profile to dig further.

You will need to using the engine sources to be able to fully debug this. The precompiled editor has lots of inlining and optimizations can make it challenging to debug. When using the engine sources, you can selectively turn off the optimizer for functions by adding UE_DISABLE_OPTIMIZATION & UE_ENABLE_OPTIMIZATION around the scope:

void WillBeOptimized()
{
}
 
UE_DISABLE_OPTIMIZATION
void MethodThatWillNotBeOptimized()
{
}
UE_ENABLE_OPTIMIZATION
 
void WillBeOptimizedToo()
{
}

Regards,

Martin

[Attachment Removed]

10

Hi Martin,

I ran more tests and here are my notes :

  • The logs are consistent with the traces. What would be the points of attention in them ?

Exemple:

[2026.09.29-08.15.58:107][894]LogGarbage: Collecting garbage

[2026.09.29-08.15.58:108][894]LogGarbage: 1.54 ms for Verify GC Assumptions

[2026.09.29-08.15.58:108][894]LogGarbage: Verbose: 0.005500 ms for MarkObjectsAsUnreachable Phase (38 Objects To Serialize)

[2026.09.29-08.15.58:110][894]LogGarbage: Verbose: 1.029200 ms for Reachability Analysis

[2026.09.29-08.15.58:110][894]LogGarbage: GC Reachability Analysis total time: 1.05 ms (1.05 ms on reference traversal)

[2026.09.29-08.15.58:110][894]LogGarbage: 1.05 ms for GC - 329921 refs/ms while processing 346418 references from 47183 objects with 0 clusters

[2026.09.29-08.15.58:110][894]LogGarbage: Freed 74752b from 16 GC contexts

[2026.09.29-08.15.58:110][894]LogGarbage: 0.350203 ms for Gather Unreachable Objects (31394 objects collected / 78576 scanned with 30 thread(s))

[2026.09.29-08.15.58:135][894]LogGarbage: 20.604402 ms for unhashing unreachable objects (31394 objects unhashed)

[2026.09.29-08.15.58:300][894]LogGarbage: GC purged 31394 objects (78576 -> 47182) in 165.022ms (1 iteration(s))

You can disregard the lenghty purge time as I disabled throttling for purge on that specific run.

However, the “unhashing time” is quite high. Not sure what this is for ? Our assets have very long names. Can it be related ?

Is there any rule of thumbs applicable for object count / GC frequencies ?

  • Regarding the delegates method, :
    • One is related to the texture streaming. Until we figure out and resolve proper usage for it, it should be disabled in our deployed application. This is a miss in the reference platform.
    • The other one is an iteration on all levels pending GC. Since GC was leading to frame loss, we reduce the frequency for GC. (Less spikes but bigger ones) With the iostore usage, there is a chance we can restore frequent but smaller GC. It should hopefully help.

I am attaching a memory report in case you can spot any red flag…

In particular, I am curious to understand what could be the cause for the below line:

 Function  16086  4225\.80  5969\.02    0\.00      0\.00      0\.00      0\.00     Function  16086  4225\.80  5969\.02    0\.00      0\.00      0\.00      0\.00

As far as I could tell, those are existing even in our areas with very low quality content, …

Thanks,

Basile

[Attachment Removed]

11

Note:

This line triggers a warning in Visual Studio :

[Image Removed]

Not sure this is intentional…

Best,

[Attachment Removed]

12

Hi,

The unhashing phase should normally be timesliced under the incremental part of the GC. There are conditions that can force a full purges like moving faster than the system can complete the work. I would need a trace to be able to analyze what is happening here. One thing you should consider is tuning gc.IncrementalGCTimePerFrame. The default is 2ms per frame. You might be able to allow for more if you game thread is waiting.

As far as the memreport, my first stop is always the texture report. I see a lot of huge uncompressed textures. A lots are from the UltraDynamicSky. I doubt that there is any way for a user flying at 10,000ft to see PuddleRipples_atlas. You should tweak that content.

All the textures are fully loaded so you have lots of useless texture space loaded. I understand that you are trying to get rid of the wait for the streaming manager by this. There is a side effect that do affect the game thread during the load of the asset. It might be better overall but that is to be validated. Again, it might be that the fast movement is forcing the engine to flush.

As far as Function, it is the structures required for the reflection of the UFunction. You can reduce the ones that come from your project but the bulk likely comes from the engine. It shouldn’t affect the performance of your application.

Regards,

Martin

[Attachment Removed]

13

Hi,

You are correct, the unhashing phase should be timesliced and it actually is. I did not notice the same variable was driving this one and the actual deletion that I highlighted as being explained.

While the current usage is fast jet which probably cannot see puddles, we shall support helicopter flights and ground level applications (UAVs, …). Having one application per use case is something we are trying to avoid. That being said, I repeatedly requires modelling group to “trim the fat”, … Your comment will back up my upcoming request to them.

I suspect this is the kind of impact that we see by disabling texture streaming. So far, this is clearly visible on my laptop but I do not recall seeing them on the site platform. I remember briefly looking into the io dispatcher but dropped investigations as not critical in real life usage. (unless we overviewed the tests)

[Image Removed]Looking at them again, I see uncompression effort, I guess this is related to this :

[Image Removed]Any recommandation for fastest and smoothest loading ?

That being said, our reason for disabling texture streaming was related to this case : [Content removed] and other situations where texture streaming was completely off and messing up the user experience. Would you recommand getting this to work rather that disabling them ?

Noted for the function count. We will not make a priority of it.

Thanks again,

Basile

[Attachment Removed]

14

Hi,

The TryLoapMips event is the texture being fully loaded. It increases the level streaming times. It is the source of a hitch in this case which you will want to avoid anyway. There isn’t much to do to get this 120ms down to an acceptable level. I’m currently discussing a similar case with a licensee that prioritize the loading speed for his levels. We found out that there can be some benefits to using a different allocator than the default (BinnedMalloc3 in 5.8) but it is for a static experience, ie no openwolrd streaming, so it’s unclear if this is also beneficial in your case (loading times and framerate) . You could try launching the package with -BinnedMalloc2 or -mimalloc. Those allocators use different strategies to manage the allocations and have different result on the overall performance.

The texture streaming defers loading the top of the mip chain for when they are needed so the levels will finish loading faster. I recommend that you reactivate the streaming and use UEngine::AddTextureStreamingLoc to seed the location of the scene capture to the streaming system. This should force the mips to load for the textures around the scene capture actor.

Regards,

Martin

[Attachment Removed]

15

Hi Martin,

I will need some feedback from the reference platform regarding the -notexturestreaming hitches. Clearly, on my laptop, this is a blocking phenomenom. However, as I said, I do not recall seeing them in the traces collected on site. I guess disk access, transfer rates and many operating systems settings (AV) or hardware are parts of this equation. I hope we will not have to go that much down as the allocator investigation but thanks for the pointer. Should we come to it to, we will compare allocators at the same time as compression impacts.

That being said, coming back to the GC questions, I am uploading you some traces for the application run. It appears that the unexpected, to me at least, delay in GC execution might be tied to texture streaming being enabled or not.

As you can see :

  • in reasonable.utrace, the manual trigger every 3000 frames is immediately followed by the GC:

- [Image Removed]

  • In the reasonable_nots (same application adding -notexturestreaming in command line):

[Image Removed]Few seconds are separating the bookmarks. It feels as if loading the level is somehow preventing the GC to trigger. However, I have not yet been able to pinpoint the exact cause.

Are you able to read such an information from the traces ?

Also, regarding texture streaming, our issues were not limited to render targets but also to very narrow fields of view and possibly other situations to be confirmed like ndisplay usage.

I had a brief look into the code and I wonder if :

  • Clamping LOD scale could be an explanation for very small fov (0.1 degree for instance) looking at things 50km away
  • ndisplay viewports are doing all the needed things for setting this up

Long story short, this will be another separate investigation but any early hints are welcome !

Thanks,

[Attachment Removed]

16

Hi,

I’ll start with some good news. You can shave off around 0.5ms from the GatherUnreachableObjects phase. The 2 VerifyXX calls are validation code that you can disasble by setting s.VerifyObjectLoadFlags to 0. Those are disabled on the Shipping target so you only have to care for Development.

The reason why the “forced” GC is delayed is related to not streaming textures. The default for the GC is to not run while there is some ongoing async loading. The “forced” GC is delayed until the streaming is completed. This can be changed with gc.PerformGCWhileAsyncLoading. Mode 1 is always run even if streaming while mode 2 is only run in low memory conditions. When using mode 2, gc.LowMemory.MemoryThresholdMB must be set to the amount of RAM that you want as the minimum free to force the GC.

I spent some time looking at the Reasonable trace and I am seeing lots of hitches that are caused the frame limiter technique the app is using (Fixed FPS, t.MaxFPS or smoothing). The problem comes from FEngineLoop_UpdateTimeAndHandleMaxTickRate waiting too long so there is no buffer to deal with the 3.5ms cost for the GC. One solution to this problem is to stop limiting the frame rate and rely on V-Sync so the Game thread waits after the frame is generated instead of before. I haven’t dealt with nDisplay in a while so I’m not sure if it’s an option here.

Another option is to use a CustomTimeStep. The engine ships with UCatchupFixedRateCustomTimeStep which is a way to deal with the problem you are facing. This code was written for Virtual Production and can only deal with hitches that can be absorbed by the next frame. It might work in your case. The part that I’m less sure is that it also enforces a fixed time step (1/TargetFPS) so it will behave weirdly if multiple consecutive frames keep hitching until it is not possible to absorb the delay. It could work for you based on what I see in the trace. You might also have to make a copy and customize the code to fit your specific needs.

Regards,

Martin

[Attachment Removed]

17

I forgot to mention two things.

We are aware of licensees that have moved the GC to the beginning of the frame and have adapted FEngineLoop_UpdateTimeAndHandleMaxTickRate so that the GC time is considered wait time instead of Tick time. In your case, that would result of hiding the cost of the GC essentially making it free if it fits in the wait time. This would require to customize the engine sources but I’m still mentioning it.

You can have a look at a recent presentation from one of my colleagues that goes over frame pacing issues: A Frame's Life: Frame Timing, Synchronization, and Latency in UE | Tutorial

This presentation doesn’t deal with nDisplay clusters but it still contains lots of valuable information that should help you.

Regards,

Martin

[Attachment Removed]

18

Hi Martin,

I believe the last opened question on this matter is the reason for GC delay.

If you have any information on this, it is welcome.

I will open separate tickets to understand better the situations with regards to :

  • Texture streaming - As I said, very narrow field of view are clearly exposing the issue.
  • Remaining hitches:
    • Allocator memory management ? [Image Removed]
    • Unthrottled SetTextureReference calls [Image Removed]

Thanks,

Basile

[Attachment Removed]

19

Hi,

Did you miss my first reply today? I did cover the why the GC is delayed when texture streaming is off. The TLDR is that the GC will not run while there is ongoing async streaming. You can change the behavior with gc.PerformGCWhileAsyncLoading. See my previous reply for the details.

I agree that your question on texture streaming regarding narrow FoV require a separate ticket. My colleagues handling rendering will be able to provide better help them me on this subject.

I did a quick research on the Allocator Memory Manage, ie the Trim. This is the current behavior with both binned allocators. You should give a shot to mimalloc (-mimalloc argument). This allocator is faster but uses more RAM than both BMs and uses a different strategy for trimming.

Digging deeper regarding SetTextureReference would need a new ticket. I might still be on the receiving end but we usually try to limit the scope of tickets in case we have to transfer to a colleague. Long threads (such as this one) eventually become challenging to pick so it can be good to restart on a fresh content. I still did a quick research about what can be the cause of lots of SetTextureReference calls and it could be related to streaming. Please make sure to include an Insights trace in the new ticket.

Regards,

Martin

[Attachment Removed]

20

Hi Martin,

Seems your messages came in while I was typing mine.

The throttling is not representative from target platform, it is my way not to burn my laptop, … We are using nDisplay with nvidia hardware frame lock mechanism.

Also, never investigated why but forcing vsync did not apply so I am relaying on software sync, …

Thanks for the answers and the link to the presentation. It is already on my to watch presentations. I wished I had travelled to Chicago but I could not make it this year.

I suspect we are good to good with this one.

Still, feel free to react to the last messages if you have educated suggestions…

Thanks,

Basile

[Attachment Removed]