Insights trace has GPU frames of zero length after a spike

Development Platform & Builds
, ,
1

This is to report an issue in Insights. When there is a long GPU frame spike, the next GPU tags will be shown as having zero length. This is due to the way the GPU timeline is built in FGpuProfilerAnalyzer::OnEvent and the CalibrationBias “lag”.

You can see in the screenshot below that frames 1366 and 1367 are “lost” as 0-length at the end of frame 1365.

[Image Removed]

This messes up statistics by introducing false 0-length tags. If the spike is really long, there can be multiple 0-length frames in a row after the spike.

This code in insights causes the issue:

// The monolithic timeline assumes that timestamps are ever increasing, but // with gpu/cpu calibration and drift there can be a tiny bit of overlap between // frames. So we just clamp. if (ThisMinTime > LastTime) { LastTime = ThisMinTime; } ThisMinTime = LastTime;

We corrected locally it but not just clamping but also introducing a correction offset:

double CalibrationBiasCorrection = 0; while (BufferPtr < BufferEnd) { // Snip ... LastTime = double(ActualTimestamp + CalibrationBias) * 0.000001 + CalibrationBiasCorrection; // Snip … if (ThisMinTime > LastTime) { CalibrationBiasCorrection = ThisMinTime - LastTime; LastTime = ThisMinTime; } ThisMinTime = LastTime; // Snip … }With this correction, the previously 0-length frames appear just after the long frame. CalibrationBias eventually fixes itself when the corresponding long CPU frame updates it.

I’m not sure this is the best fix, but at least the aggregate statistic are better.

Thanks

2

Steps to Reproduce
Capture trace capture that contains a very long GPU frame (40+ms).

Open the trace in Unreal Insights.

Observe that the frame after the spike is zero length.

3

Hi Alexandre,

For UE 5.5 (or earlier versions) the fix you propose may be fine (it trades a lower accuracy of time location of events with increased accuracy for duration of events, which makes sense).

For UE 5.6 we are introducing a new GPU Insights (with new runtime and a new way of tracing GPU timing events). The analysis has a new code for the GPU trace events, so this fix will not propagate to future UE versions (but, in the same time, the new GPU Insights should have better GPU timing accuracy).

Cheers,

Ionut

4

Yes, also in UE 5.6 the timing events export will include GPU events. These will create a thread for each GPU Queue (i.e. multiple queues / GPU).

5

Hello Ionut, thanks for your answer. We will use that fix locally until 5.6. Looking forward to better GPU timing accuracy.

For the new GPU Insights in 5.6, will it still support exporting events and stats through TimingInsights.Export… commands? We use this extensively in our automated benchmarks.