Unreal insights dedicated server file activity

Development Platform & Builds
, ,
1

Hello,

we started to see health-check terminations on our development environments on unreal dedicated servers, running on linux on aarch64(arm) platform in cloud. During the investigation, we encountered an i/o operations overflow. To check that we did not introduce any heavy file reading/writing operations, I tried to use unreal insights with -trace=file,loadtime channels, but it seems that no additional information is provided. Attaching .utrace file.

Asset loading seems to be normal - big spike around 400 assets on the arena init, and around 20 during the match itself.

To add to context, all the dedicated servers are running minimal trace (cpu,bookmark), so that might be the source of file operations overflow, but this was always the case and issue started to reproduce recently.

Is there any way to profile low level read/write/open file operations on LinuxArm64? Or maybe you have any advice how to handle this situation differently to get some insights? Some other tools instead of unreal insights, completely other approach?

[Attachment Removed]

2

Hi,

The file operations are handled in FUnixPlatformFile and its methods are not Instrumented. You won’t be able to use Insights here unless you add some instrumentation. A colleague is recommending that you try the iostat command as a starting point for the investigation. You will need to use another profiler to dig deeper.

Regards,

Martin

[Attachment Removed]

3

Hello!

Thank you for the answer!

We used various linux profiler tools and figured out that deterioration was caused by automatic policies update of our cloud services provider for the os, where dedicated server is run. Zram (in-memory compressed swap file) was enabled, and with dedicated server iterating over majority of it’s memory during tick, it’s not really compatible with this tech.

Regards,

Alexey.

[Attachment Removed]