FXSystem memory leak

Development Platform & Builds
, ,
1

Hello,

in UE5.7 we noticed a memory leak in GlobalDistanceField, which causes out of memory crashes in some of our games, on consoles.

It can be easily reproduced in the Lyra project, in editor; in Samples/Games/Lyra/Config/DefaultEngine.ini set r.RayTracing=False, so that GlobalDistanceField.PageObjectGridBuffer gets allocated. Then start and stop the Lyra PIE game more times, moving a little bit around. Each time, a new GlobalDistanceField.PageObjectGridBuffer is allocated, with the same size and parameters (we saw this with r.DumpBufferPoolMemory).

The reason seems to be that the refcount hold by FXSystems is not decremented after FFXSystemSet::OnMarkPendingKill() is called, since FXSystems are not destroyed.

Which is the correct place where calling FXSystem / FXSystemSet destructors?

Thanks, best regards,

Marco

[Attachment Removed]

2

Hi,

Could you integrate CL 50207022 to see if that fixes it?

Thanks,

Stu

[Attachment Removed]

3

Hi, yes, this CL, removing the refcount on the buffers, solves the main leak, and for us this is enough (we had out of memory crashes). It seems that the FXSystem / FXSystemSet classes are not destroyed yet, this leaves a minor leak.

Thanks, Marco

[Attachment Removed]

4

You shouldn’t see any leaks (that’s the case for us at least), so it implies something is still holding the reference.

It might be worthwhile adding some logging in to track that down, perhaps there’s a component which still has an instance which is registered?

Thanks,

Stu

[Attachment Removed]

5

Thanks Stuart, we will check.

Best regards, Marco

[Attachment Removed]

6

Which regards to what you said Daniel, is the buffer called GPUBufferFloat? If so that comes from a data set, and they come from a system instance, regarless of the second part it may mean you have a leaking system instance / component somewhere. Now as for the second part I just ran some tests and I think I see it leaking also, so I’ve made some local changes to fix that but I need to run some pretty extensive tests to confirm it won’t introduce another issue.

I’ll hopefully have some conclusion early next week.

[Attachment Removed]

7

Sounds like you have done something similar to what I have in CL 51168522.

I didn’t push this into the FSceneInterface, that’s something I may follow up on in the future.

Thanks,

Stu

[Attachment Removed]

8

Hi, we are also running into significant leaks due to FXSystems not being destroyed - not related to the global distance field, but GPUBufferFloats which have leftover reference counts to SRVs from FNiagaraGpuComputeDispatch.

I’m finding that world’s FXSystemSet returned by FFXSystemInterface::Create is never destroyed and is holding on to shared refs to the FXSystem and FNiagaraGpuComputeDispatch.

I’m struggling to find how the FXSystemSet is intended to be destroyed, since it doesn’t seem to ever be made into a shared pointer nor is it being explicitly deleted.

Am I missing something, or maybe FFXSystemInterface::Create should return a TSharedRef, and be stored as TSharedPtr on the world or FScene?

[Attachment Removed]

9

Yes, the buffer comes from a system instance whose lifetime is the same as the world, and the buffer is being leaked when the world is unloaded. I believe I tracked down that the only reference keeping it alive was an SRV referenced by FNiagaraGpuComputeDispatch::SimulationsToSort (FNiagaraGPUSortInfo::AllocationInfo.BufferSRV), and fixing the FXSystemSet leak by making the UWorld reference a TSharedPtr has fixed it.

[Attachment Removed]