Summary
Two engine debug facilities keep one entry per spawned object for the LIFETIME OF THE PROCESS on a dedicated server, so a long-running server grows with everything it has ever spawned, not with what is alive. Measured +6.1 MB/h of RSS floor growth on a 24 h soak with 200 AI pawns constantly dying and respawning; +0.00 MB/h over three complete hours after disabling both. Live actor/object counts stay flat and a per-class census shows 0 of 443 classes growing.
CAUSE 1 - Visual Logger redirection maps (~145 B per spawned actor). UAbilitySystemComponent::ReadyForReplication calls UGameplayTasksComponent::ReadyForReplication, which calls REDIRECT_TO_VLOG(GetOwner()) = FVisualLogger::Redirect(this, Owner). In VisualLogger.cpp (~965-1010) this does ChildToOwnerMap.FindOrAdd(FromObject) on a global TMap<FObjectKey, TWeakObjectPtr<…>> and, under UE_DEBUG_RECORDING_USING_VLOG, GetRedirectionMap(World)[Owner]. The key is an FObjectKey, which survives the death of the object, and entries are only removed when the world goes away (VisualLogger.cpp ~1191). One permanent entry per spawned actor.
Important: the gate is not the one most people assume. REDIRECT_TO_VLOG is compiled under UE_DEBUG_RECORDING_ENABLED (VisualLogger.h ~37-39), defined in VisualLoggerDefines.h ~9-11 as (ENABLE_VISUAL_LOG) || (UE_TRACE_MINIMAL_ENABLED && !NO_LOGGING). Setting ENABLE_VISUAL_LOG=0 alone does NOT disable it on a Development server - but it does compile out the Visual Logger counters, so the instrumentation goes blind exactly where the growth remains.
CAUSE 2 - FPersistentNetDebugNames via the Iris object reference cache (~93 B per unique net reference). FObjectReferenceCache::CreateObjectReferenceHandle (ObjectReferenceCache.cpp ~571-580) builds, under #if UE_NET_TRACE_ENABLED, the string “Ref_” + object name and passes CreatePersistentNetDebugName(…) as an ARGUMENT to UE_NET_TRACE_NETHANDLE_CREATED. The argument is always evaluated: the macro expands to a direct call to FNetTrace::TraceObjectCreated, with no check for whether a trace session is recording. FPersistentNetDebugNames::Allocator is an FMemStackBase that lives for the whole process and is only released at exit; names are de-duplicated by hash, but every respawn produces a new unique name (e.g. Ref_BP_TopDownCharacter_C_2147413104), so nothing de-duplicates. UE_NET_TRACE_ENABLED is 1 in every non-Shipping build with trace enabled (NetTraceConfig.h ~10-15): Development AND Test.
WORKAROUND (and the measurement after it). In the dedicated server target only, with BuildEnvironment = TargetBuildEnvironment.Unique (which forces a full engine rebuild for the server):
GlobalDefinitions.Add(“UE_DEBUG_RECORDING_ENABLED=0”);
GlobalDefinitions.Add(“UE_NET_TRACE_ENABLED=0”);
GlobalDefinitions.Add(“ENABLE_VISUAL_LOG=0”);
Result on a 4.16 h round, 200 fighting bots, same hardware and harness: RSS floor 332 → 331 → 332 MB = +0.00 MB/h over three complete hours (it was +6.1 MB/h), 0 ensures, 0 fatals.
SUGGESTED FIX.
- Remove entries from the Visual Logger redirection maps when the keyed object is destroyed (both GetChildToOwnerRedirectionMap() and GetRedirectionMap(World) are reachable), or bound them; and/or gate REDIRECT_TO_VLOG behind a runtime switch that is off by default on dedicated servers.
- In UE_NET_TRACE_NETHANDLE_CREATED, do not evaluate the debug-name argument when no trace session is recording.
WHY IT MATTERS. Both are developer-only facilities, but they make any long-running Development/Test dedicated server look like it has a product memory leak - which is exactly the configuration in which soak tests are run. It cost us five days before we found it in the engine source.
Engine: UE 5.8.2 source build (tag 5.8.2-release), Windows 11, dedicated server target, Development configuration, Iris enabled (net.Iris.UseIrisReplication=1, net.SubObjects.DefaultUseSubObjectReplicationList=1).
What Type of Bug are you experiencing?
Networking
Steps to Reproduce
- Create a C++ project with GAS, where each pawn owns a UAbilitySystemComponent.
- Build a dedicated server target in Development configuration.
- Enable Iris: net.Iris.UseIrisReplication=1 and net.SubObjects.DefaultUseSubObjectReplicationList=1 in DefaultEngine.ini.
- Run the server with ~200 AI pawns that keep dying and respawning (~6.4 deaths/s) for several hours. Spawning and destroying ~10k actors that own a UGameplayTasksComponent reproduces it as well.
- Log resident memory and read the PER-HOUR MINIMUM (the post-GC floor), not the average; or run with -LLM -LLMCSV and watch the tags.
- Observe the floor growing linearly (~6 MB/h here) while live actor and object counts stay flat.
Expected Result
The resident memory floor (per-hour minimum, after GC) stays flat over hours, because the number of live actors and objects is flat: pawns are destroyed as fast as they are spawned.
Observed Result
The floor grows linearly and never returns: +6.1 MB/h over a 24 h soak with 200 respawning pawns, while live object/actor counts, the FName table and the per-class census stay flat (0 of 443 classes growing). Breakdown by LLM tag: ~2.6-3.8 MB/h under …/ReadyForReplication/AfterPostInit (Visual Logger redirection maps, ~145 B per spawned actor) and ~2.1 MB/h in MemStack/page allocator (FPersistentNetDebugNames, ~93 B per unique net reference name built as “Ref_” + object name). After building the dedicated server with UE_DEBUG_RECORDING_ENABLED=0, UE_NET_TRACE_ENABLED=0 and ENABLE_VISUAL_LOG=0, the same 4.16 h round measures a floor of 332 → 331 → 332 MB = +0.00 MB/h over three complete hours.
Affects Versions
5.8
Platform(s)
Windows