How to maximize shader compiling threads?

General Feedback & Requests
, , ,
1

I am converting a small project to UE 5 and it has taken forever to compile all the shaders first opening the editor and then opening my level. I have a Threadripper with 32 cores and most of them are idle all the time, I can see the system spawning at most 3 treads for shader compilation. How should I configure my settings to optimize for my machine?

I have seen some suggestions online about changing the config values on the DevOptions.Shaders section, but I have tried different values and I haven’t found a good way to do it, and also I haven’t been able to find documentation about it…

[DevOptions.Shaders]
; See FShaderCompilingManager for documentation on what these do
bAllowCompilingThroughWorkers=True
bAllowAsynchronousShaderCompiling=True
; Make sure we don't starve loading threads
NumUnusedShaderCompilingThreads=3
; Make sure the game has enough cores available to maintain reasonable performance
NumUnusedShaderCompilingThreadsDuringGame=4
; Core count threshold.  Below this amount will use NumUnusedShaderCompilingThreads.  Above this threshold will use PercentageUnusedShaderCompilingThreads when determining the number of cores to reserve.
ShaderCompilerCoreCountThreshold=12
; Percentage of your available logical cores that will be reserved and NOT used for shader compilation
; 0 means use all your cores to compile Shaders
; 100 means use none of your cores to compile shaders (it will still use 1 core).
PercentageUnusedShaderCompilingThreads=50
; Batching multiple jobs to reduce file overhead, but not so many that latency of blocking compiles is hurt
MaxShaderJobBatchSize=10
bPromptToRetryFailedShaderCompiles=False
bLogJobCompletionTimes=False
; Only using 10ms of game thread time per frame to process async shader maps
ProcessGameThreadTargetTime=.01
; For regular machines, wait this many seconds before exiting an unused worker (float value)
WorkerTimeToLive=20
; For build machines, wait this many seconds before exiting an unused worker (float value)
BuildWorkerTimeToLive=1200
; Set process priority for ShaderCompileWorker (0 is normal)
WorkerProcessPriority=-1

; These values are for build machines only currently to reduce the number of SCWs spawned to reduce memory pressure
bUseVirtualCores = False
; CookerMemoryUsedInGB = 49
; MemoryToLeaveForTheOSInGB = 3
; MemoryUsedPerSCWProcessInGB = 0.4
; MinSCWsToSpawnBeforeWarning = 8

; Use SCW memory pressure calculations regardless of whether cooking is done on a build machine
; Note: CookerMemoryUsedInGB, MemoryToLeaveForTheOSInGB, MemoryUsedPerSCWProcessInGB must all be set to enable
bForceUseSCWMemoryPressureLimits = False

What can I do to accelerate this process (And make a more optimal use of the resources on my workstation?)

Thanks

2

Well try…

NumUnusedShaderCompilingThreads=1
PercentageUnusedShaderCompilingThreads=90
WorkerProcessPriority=1

3

Thanks for your answer Alberto
When you do that, do you see lots of processes compiling shaders in the task manager? I have tried and I still see only 3.
Also looking at the comments on this parameter:

; Percentage of your available logical cores that will be reserved and NOT used for shader compilation
; 0 means use all your cores to compile Shaders
; 100 means use none of your cores to compile shaders (it will still use 1 core).
PercentageUnusedShaderCompilingThreads=50

it seems to indicate that the higher the number the less threads will be used, but I have also tried lowering it to 20 and I see the same results

Am I missing other settings to activate all of this?

Thanks

4

True, try:
PercentageUnusedShaderCompilingThreads=10

5

I had a same issue.
only 3 shadercompiling workers and
only 1 worker works.

I found my solution and for now all cpu cores are working.
I changed following values.

NumUnusedShaderCompilingThreads=1
NumUnusedShaderCompilingThreads=2
PercentageUnusedShaderCompilingThreads=10
WorkerProcessPriority=0
bUseVirtualCores = Ture
and I killed all incredibuild processes.

I think incredibuild is the reason…

my env:
cpu: ryzen 3600
mem: 32gb
gpu: rtx2070

2 Likes
6

I have the same situation + Threadripper Pro 16 core and onlz 4 cores are used for compilation.

Where is the config file located to change those values?

7

I think I found the solution - it was the INCREDIBUILD addon to Visual Studio.
I removed it from Visual Studio extensions (complete uninstall) and now all 16 cores and 32 threads are used for compilation.
Hope that helps somebody someday…

4 Likes
8

I found a different alternative. On opening old (just few months older actually) projects, I had the load up screen virtually stuck for hours at compiling shaders. Then I created a new project and compared all file contents between the two. Turns out the older project has these two lines in its Unreal Projects\MyProject\Config\DefaultEngine.ini:

r.SkinCache.CompileShaders=True
r.RayTracing=True

These lines are not automatically defined anymore for newer projects. Remove them (or just put them both to ‘False’). It should not get blocked again at compiling shaders now.

9

Thank you so much! I have been trying to wrap my head around what happened. Must have checked that by mistake during a recent re-install of Visual Studio. I have had issues when Incredibuild has been installed in the past with other development tools.

1 Like