Hi! We’re in the process of upgrading to 5.6.1. We run our servers on Linux using Kubernetes. On 5.5.4 we had satisfactory performance, we were able to target 30 fps. On 5.6.1 we’re seeing crippling performance drop, down to 7-8 fps.
We inspected 5.6 and 5.5 traces of servers running on Linux and noticed that the culprit might be OS randomly taking quants from process or threads. There are multiple 30-40 ms freezes in random execution blocks, such as
I cannot show you full trace unfortunately, as it contains sensitive information. Screenshots above show physics exclusively, but there are random freezes inside ISM transform updates and our code. Physics just takes most of server CPU time, which is probably why it’s more frequent.
5.5 version and 5.6 version are based on pretty much the same codebase, save for some changes to adopt 5.6 API changes. The worker node hardware configuration and startup options are the same. We tried elevating process priority, but that did not help.
We saw that 5.6 introduced Tasks instead of ParallelFor for MidPhase, which looks like the only major change inside Chaos. This change actually made multithreading work on Linux servers. Thanks to that, when we re-enabled midphase redistribution, we saw significant gains in CPU performance (up to 3x), but those 30-40 ms spikes are still present.
Another side effect is that physics uses frame duration for substepping. Those 30-40 spikes create a positve feedback loop. They extend frame duration and cause substepping make more steps, which worsens server CPU time even more.
In short:
- Did team change anything related to thread/process scheduling in 5.6? What can we check in our OS/VM configuration?
- Seeing how midphase redistribution is really beneficial (and is actually working on Linux multithreaded configurations now), why did team decide to disable it by default? It was enabled in 5.5. This makes upgrades much harder as we have to track whether any good stuff was disabled.