[UE 5.3][Unreal Insights] Guidance on reducing .utrace file size, load time, and analysis memory usage

Development Platform & Builds
, ,
1

Hello Epic Support Team,

We maintain our own source branch based on UE 5.3, and our team regularly uses Unreal Trace and Unreal Insights for day-to-day performance analysis.

Based on the long-term experience of several users on the team, we see a few recurring usability concerns:

- `.utrace` files can become quite large, making them inconvenient to store and transfer.

- Larger traces can take a long time to load and analyze in Unreal Insights.

- Unreal Insights can use a large amount of memory while loading and analyzing those traces, which also affects usability.

These are currently qualitative observations from normal use rather than results from a controlled benchmark. At this stage, we are not reporting a narrowly scoped bug. We are planning to improve the Trace/Insights workflow in our UE 5.3-based branch and would first like to understand which areas Epic recommends investigating, so that we can choose a sound starting point.

We would particularly appreciate guidance on the following questions:

1. To reduce `.utrace` file size while retaining useful analysis data, which areas would Epic recommend investigating first?

2. Are there proven optimization approaches or recommended architectural directions for improving Unreal Insights load time and memory usage during analysis?

3. Were relevant improvements introduced in UE 5.4 or later? If so, could you point us to the corresponding features, documentation, source areas, or CLs so that we can evaluate upgrading or backporting them?

4. If we improve trace generation or analysis in our own branch, are there important design constraints, compatibility requirements, or common pitfalls that we should understand in advance?

5. If Epic recommends starting with measurement or trace-content analysis, which metrics and methods would be the most useful? We can then collect targeted data based on that guidance.

We are not expecting Epic to provide a complete design for our branch. A recommended investigation order, relevant references, or information about existing practices would be enough to help us narrow the scope and plan the next steps. We can provide more specific usage scenarios and targeted measurements later if needed.

Thank you for your guidance.

2

Hi,

All three of your observations are known characteristics of the current Trace/Insights architecture rather than defects specific to your branch. Some work against them landed in 5.4 – 5.8 (I’ll detail more below).

(Q5) re measurement - I’m answering this first, because it changes the rest…

The fact that you’re asking for an investigation order rather than a fix is the right instinct here. So, yes -- we’d strongly recommend starting with trace-content analysis, and it’s cheap.

a) In UE 5.4 we have added TraceAnalyzer, a standalone tool that helps inspecting content of a trace file. This tool can be used in two ways:

  • to convert a binary trace file to a text representation, so to easily read what events are there;

TraceAnalyzer trace.utrace > trace.utrace.txt* to analyze the trace and output only stats about type of events it contains

TraceAnalyzer trace.utrace -no_new_event_log -no_event_log > trace.utrace.stats.txtThis will output a table with all type of events (Logger.Event) detected in the trace, each with count and byte sizes (uncompressed data) stats. In practice, one logger/event usually dominates by an order of magnitude, and knowing which changes the whole plan.

So, first thing would be to run the above on some real case traces from your stream.

b) There are some known trace channels that contributes significantly more to the size of traces than others.

  • The “memory” preset (especially its “memalloc” and “callstack” channels) is one of top offenders here. So, a trace that records “memory” trace events is expected to very large.
  • Enabling default “cpu” and “gpu” trace channels should be light in general. But, for very long traces (hours) trace can grow significantly. Usually few CPU timers are responsible for the majority of cpu timing events in a trace. Identifying these top timers and toggling them off in code would significantly reduce the size of trace files. These can be simply identified by running Insights on a trace, sorting the Timers panel by Count descending and looking on top timers.
  • The -statnamedevents command line option is the single biggest amplifier of cpu-channel volume.
  • The “task” channel also adds a significant amount of trace data to a normal “cpu” trace.

c) Runtime: “Trace.Status” console command --> gives high level stats about how much memory the trace system is currently using and which channels are enabled. Pay attention to the size of the important events cache. If this cache continuously grows or if its size is too large, then it may point to a problem with emitted “important” events. TraceAnalyzer helps identifying these events.

In newer UE (ex. 5.8) there are additional debug metrics that can be enabled for trace system itself. TraceLog would emit Bandwidth/MemoryStats events (behind UE_TRACE_EMIT_STATISTICS; these would also appear in the Counters panel in Insights) and TraceStall events; TraceStall is a direct signal that trace volume is costing frame time rather than just disk.

(Q1) Reducing .utrace size

Record only what you really need and only when you really need.

The UE 5.3 default preset is “cpu,gpu,frame,log,bookmark,screenshot,region” (region was added in 5.3). The “cpu” channel dominates volume in almost every default trace. If traces recorded only with default channels are heavy, one way to mitigate this is to not record cpu data all the time, but only for specific time regions.

a) Use Trace.Enable/Trace.Disable (or Trace.Pause/Trace.Resume) to keep the “cpu” channel enabled only for specific time intervals.

b) Don’t record a trace from app start (i.e. don’t use -tracehost or -tracefile), instead start tracing late, when needed (see Trace.File/Trace.Send/Trace.Stop console commands).

c) Use trace snapshots. Trace is always-on and (for the enabled channels) the trace data is recorded automatically into a circular in-memory buffer (-tracetailmb=N, default 4 MiB / 32 MiB in Editor). Use Trace.SnapshotFile / Trace.SnapshotSend commands to dump the current buffer (as a trace) without changing connection state.

d) Audit -statnamedevents. This is the single biggest amplifier of cpu-channel volume: it turns the entire SCOPE_CYCLE_COUNTER family into trace timers. Also audit -verbosenamedevents (adds stall/wait scopes).

e) Check the per-object timer-name default. UE 5.3 (CL 26815068) re-enabled object-name scopes by default whenever the process is not a commandlet. It still requires cpu + -statnamedevents, so it only bites if (d) applies - but if it does, you may be paying for the cost of per-object names tracing, and that shows up as both file size and analysis cost.

f) Trim after the fact. TraceTrimmer is a new tool added in UE 5.8. It is the direct answer to “the file is huge and I only care about 30 seconds of it.”. With this tool you can extract a smaller trace from a larger trace file (either by filtering the type of loggers and events or by extracting a trace for a specified time interval).

h) If memory traces are your problem specifically: one option is to record without callstacks; i.e. record only the “memalloc” and “memtag” channels (instead of “memory” preset that includes “callstack” and “modules”). The “memory_light” preset (memtag,memalloc) was added in UE 5.5. Compressed callstacks came in UE 5.8.

(continue…)

3

(Q2) Re load time and analysis memory

Analysis-side memory runs roughly 10x the .utrace file size (a ~1 GB CPU-only trace costs on the order of 10 GB of RAM to analyze). Treat that as a rule of thumb, not a guarantee. It’s useful in two directions -- if your measurements land near it, you’re hitting the known structural limit and file size is the lever; if they’re far worse, that’s a bug we’d like to see. In older Unreal Insights versions there were some memory leaks. The most important fixes were done in UE 5.7. So, one recommendation is to use latest version of Unreal Insights (ex. from UE 5.8) even if you record and analyze traces from UE 5.3 (Unreal Insights is backward compatible).

The honest answer is that this is a known structural limit. We have long term plans for the following directions:

- Today, opening a trace re-parses the byte utrace stream from scratch every time and materializes the analyzed data into in-memory providers; nothing is persisted between sessions (except the ucache for resolved symbols). Future possible improvement: to cache/persist some analyzed data.

- Filtering at analysis time, before events reach the providers. A configurable pre-filter at the analyzer boundary — drop events you don’t care about during analysis rather than storing them — can be the cheapest architectural win available and is well-suited to a licensee branch, because it’s additive and doesn’t change the file format. It also converges naturally with content-aware trim filtering.

- Paging analyzed data out of RAM rather than holding all providers resident. This involves more complex integration work with existing analyzers/providers.

- Indexing, to enable partial and lazy loading.

Note that we do not have any ETA for above (if/when these would land in a UE release). If you are going to invest engineering effort on the analysis side, you can develop on these directions.

Practical optimization approaches are mostly the ones described at point 1: reducing trace data --> smaller utrace files --> faster analysis.

Using TraceTrimmer (UE 5.8) --> extract a smaller trace from a larger trace --> open only the small trace in Unreal Insights for fast analysis.

(Q3) What landed in 5.4+ — and what’s worth backporting (relevant changes by release)

  • UE 5.4: TraceAnalyzer standalone tool (Engine/Source/Programs/TraceAnalyzer)
  • UE 5.5: “memory_light” preset; counter API extensions (unchecked/atomic -- cheaper counter emission); export Counter commands
  • UE 5.6: Block-pool hard cap (~80 MB) + UE_TRACE_BLOCK_SIZE / UE_TRACE_WRITER_SLEEP_MS -- also fixed out-of-order serial events under extreme load; CpuProfilerTrace FName variants + EventBatchV3; relay endpoints (supply custom open/write/close functions for trace output — the clean hook if you want your own transport or compression);
  • UE 5.7: Sampling Profiler (experimental; -samplinginterval=, default 125 µs); Direct Trace mode (-ListenForDirectTrace, bypasses the trace store); frontend memory-stats debug display and mem leak fixes
  • UE 5.8: TraceTrimmer + in-UI “Trim and Save As”; FTraceWriter (a new API that allows you to write new traces; it lives in TraceAnalysis module); compressed callstacks; TraceQuery (reads.utrace, emits JSONL to stdout --scripted analysis with no editor and no Insights app); MemoryInsights.ExportAllocs; minimal-trace support (i.e. trace support in shipping builds)

+ lots of new features, usability improvements and bug fixes in Unreal Insights --> you can benefit of them immediately, just by using latest version

(Q4) Design constraints and common pitfalls

When investigating cpu/gpu performance, keep tracing as light as possible. Any new trace channel enabled would add some runtime overhead (both performance and memory overhead). Some channels are more heavy than others from this p.o.v. For example, enabling the “memory” preset -- and especially the “callstack” channel - adds a significant perf overhead. If memory channel is enabled, the analyzed “cpu” perf data would need to be read with a grain of salt.

If your investigation goal is to measure memory, then it is ok to also have cpu enabled in the same time (as in general, in this case it does not matter if the game runs slower -- you aim would be to capture memory data and to correlate what/when something allocates with what is running; but overall cpu performance would be irrelevant in that case).

Event filtering has dependency edges, and naive name-based filtering (i.e. using TraceTrimmer) produces traces that analyze cleanly but read wrong. Example of known couplings: “callstack” needs “module” for symbol resolution.

Never make a high-volume event “important.” Important events are cached in memory for the lifetime of the process so they can be replayed on connect. Adding per-instance data to that path grows runtime memory without bound.

Channels may get enabled behind your back. GameplayInsights/Rewind Debugger toggles Object on PIE start and ObjectProperties/Animation/Frame when recording; -NetTrace=1 enables Net+Frame; VisualLogger, TraceSourceFiltering, NetworkPrediction and PoseSearch each enable their own when active. “We only enabled cpu” is often not what’s happening — confirm with Trace.Status.

Format compatibility cuts both ways. The trace protocol is versioned (v7 in 5.2) and we also version some of the trace loggers / events (ex. EventBatchV3 added in 5.6). The latest UnrealInsights from a UE stream generally reads older traces. If you modify the trace format (trace protocol), you lose both directions: your traces stop opening in official Unreal Insights, and official traces stop opening in yours. Strong recommendation: Prefer additive and config-driven changes (rather than touching the trace transport implementation), and prefer standalone tools and Insights plugins over edits inside TraceServices providers; the latter is where your merge debt will accumulate fastest.

Unreal Insights is deliberately targeted at deep technical experts — that’s a settled design position, not an oversight, and it explains the tool’s bias toward power over approachability. If part of what you’re chasing is “make this usable by more of the team,” expect the tool to meet you only partway, and consider whether a narrower purpose-built view over the same trace data serves you better than trying to soften the general tool.

Best regards,

Ionut

4

Additional resource: Unpacking High-Performance Telemetry with TraceLog | Unreal Fest Stockholm 2025