[URGENT] Default Movement Component causing major dedicated server performance issues (Linux Only)

Joshua_NZ · September 18, 2017, 10:23am

Hi Epic,

After a long process of profiling, I have found a major issue with the Default Movement Component & Linux Dedicated Servers. This issue does not appear on Windows Servers & this issue did not exist on 4.16.2.

This seems related to the “Char FindFloor” function, it seems to be looping in a very strange way, causing the server’s CPU to spike & in some cases stalls the server from processing anything (Picture 1). There is also some hitches to do with the MoveComponent function (Picture 2).

This occurs when I jump on top of objects inside my map.

You can find my ue stats files here:
https://www.dropbox.com/s/xfvaa4g5d0y8729/City-LinuxServer-18-10.09.06.ue4stats?dl=1
https://www.dropbox.com/s/kzq5oodehsus06v/City-LinuxServer-18-10.07.02.ue4stats?dl=1

RCL · September 18, 2017, 3:38pm

You will unfortunately need to change source files and rebuild PhysX to fix this. This is not too hard but it can be time consuming.

First of all, you will need to download all the sources from github (you may already have that). Then, find all Linux CMakeLists.txt under Engine/ThirdParty/PhysX (there are three of importance:

Engine/Source/ThirdParty/PhysX/PxShared/src/compiler/cmake/Linux/CMakeLists.txt
Engine/Source/ThirdParty/PhysX/PhysX_3.4/Source/compiler/cmake/Linux/CMakeLists.txt
Engine/Source/ThirdParty/PhysX/APEX_1.4/compiler/cmake/Linux/CMakeLists.txt

)
and remove -ffast-math from them.

Then, open VisualStudio solution for the engine, rebuild Programs\Automation directory (right click->Rebuild), rebuild AutomationTool and then run it like this

Engine/Build/BatchFiles/RunUAT.bat BuildPhysX -SkipCreateChangelist -SkipSubmit -TargetLibs=PhysX+APEX+NvCloth -TargetPlatforms=Linux -TargetConfigs=checked+profile+release

Then you can submit the resulting .a files (in Lib/Linux/x86_64-unknown-linux-gnu) to your source control and rebuild the Linux server.

We’re looking into possibility of including this fix into 4.17 branch, but the above will get you unblocked today.

Joshua_NZ · September 18, 2017, 4:54pm

Thank you for your prompt reply & clear instructions, I appreciate it.

I had to remove [RequireP4] from BuildPhysX.Automation.cs in order for it to start compiling. I also added BuildPhysX to the command, and then NvCloth was missing so I added that also.

Engine\Build\BatchFiles\RunUAT.bat BuildPhysX -SkipCreateChangelist -SkipSubmit -TargetLibs=PhysX+APEX+NvCloth -TargetPlatforms=Linux -TargetConfigs=checked+profile+release

I will report back if this resolves the server issues.

Cheers

RCL · September 18, 2017, 4:58pm

Thank you - I’ll edit my post to correct it. Sorry, I was copying arguments verbatim from my script and missed the command - I’m glad you were able to figure it out!

Joshua_NZ · September 18, 2017, 5:14pm

No problem. Doing as you suggested has fixed the issues.

Thanks so much.

looter · June 16, 2018, 11:28am

Is this still required as of 4.19.2? I have noticed some severe performance issues related to the CPU on my linux dedicated servers as opposed to my windows servers. Linux will consume over 250% CPU while the windows server will never go past 25%.

poke1103 · June 22, 2018, 3:09pm

I have also noticed this issue in 4.19 and 4.18. Is this going to be fixed?

RCL · June 22, 2018, 3:13pm

No, this is not required for 4.19.2. The reason for CPU consumption is likely different. Please try capturing the stats and/or using perf tool to determine where the time is being spent.

RCL · June 22, 2018, 3:16pm

The issue has been fixed before 4.18. Are you sure you’re running into the same problem in PhysX?

poke1103 · June 22, 2018, 3:36pm

I checked my source code, for the -ffast-math in the CMakeLists.txt files. It’s still in those files. Is this intended? This issue is definitely the same issue from what I can tell.

RCL · June 22, 2018, 3:43pm

You are probably using a prerelease or a custom version of those? There’s no -ffast-math in 4.18 (e.g. https://github.com/EpicGames/UnrealEngine/blob/4.18/Engine/Source/ThirdParty/PhysX/PhysX_3.4/Source/compiler/cmake/Linux/CMakeLists.txt) or 4.19 (e.g. https://github.com/EpicGames/UnrealEngine/blob/4.19/Engine/Source/ThirdParty/PhysX3/PhysX_3.4/Source/compiler/cmake/linux/CMakeLists.txt) though. 4.18 has been fixed in this commit on 2017-09-22 (which is before its October release date), 4.19 has never had -ffast-math.

Either way, please remove the flag and rebuild PhysX.

poke1103 · June 22, 2018, 3:46pm

Interesting. I recently downloaded 4.19 fresh from github. I made the changes none the less.