Shader compilation is slow and do not use available ressources

General Feedback & Requests
,
1

Hello, first time i open a project on UE5, it takes ages to compile shaders.while in the editor it take ages to prepare the shaders. It would not bore me if only these task would use some of the available computing power…

My specs are Intel Core I9 9900K 8Cores-16threads, RTX3090TI, SSD EVO, 32GB Memory.I use the V512.15 Studio drivers.

I guess that there are some engine parameters to tweak , the only i have found which seems to help is WorkerProcessPriority=-1=>0 or 1 in the ‘BaseEngine.ini’ .

I will not able to work of an indie project on UE5 if I have to lose 2h per day. Since i am still in learning process I often have to use example projects. How to proceed? Can anyone make a tutorial to tweak the process?

2

The first time you open the engine, or a project, it will compile all shaders. If your project has many shaders, then this will take a while. This is why using “material instances” from a “master material” is so important! Don’t just create new “materials” willy nilly; instead, create one or a few “master materials,” and make all your configured materials be Material Instances.

When the engine compiles shaders, it forks as many compile workers as you have hardware threads. All of the threads should be available and working 100% for you – if that’s not happening, then something’s wrong, but it might be system-level rather than engine-level, because this generally works for most people.

Changing the priority of the compiler workers to something higher is not what you want – this will not make the work complete any faster, it will just make the rest of the computer a lot more laggy while it’s working. In fact, I frequently go into Process Monitor and lower priority of background crunching processes to “below normal” so that using the computer isn’t so terrible while the other things are going on.

In computer science terms: Thread priority affects latency not throughput. The thing you want to have the lowest latency should have the highest priority. As long as there are free CPU cycles, those will be used for whatever work needs to be done, so the shader compiler will still do its work, even if it’s at a lower priority. It just won’t lag your mouse, or the editor, as much.

If you find that the engine keeps re-compiling all shaders all the time, then there’s something else wrong. Your hard disk might be corrupted, your disk might be full, you may be accidentally changing all your base lighting/shading parameters all the time (e g, turn on/off ray tracing, switching between forward/deferred renderer, or somesuch,) or maybe you’re creating a lot of new Projects rather than just creating new Maps/Levels in an existing Project.

If you find that you need a lot of Projects, then you’ll want to point the shader/intermediate data cache for each of those projects all to the same directory, so they can re-use the compilation of shaders that are the same. I forget the exact details, but maybe this link can help: Reducing Shader Compile Time for Unreal Engine Landscape | by Arunoda Susiripala | GameDev4k | Medium
Once your project is big, you’ll also want to take a look at: Derived Data Cache | Unreal Engine Documentation

1 Like
3

I mean, this very documentation page says that there isn’t supposed to be an initial compile (it’s under DDC Pak, for the curious), and, indeed, in some cases there isn’t. It’s not consistent behavior, though.

4

Chances are, the template projects don’t actually come with the appropriate DDC Paks. Specifically, it’s impossible for EPIC to ship pre-compiled shaders for all possible graphics architectures, so I very much think they don’t even try.

Also, if you use some non-EPIC content, that will clearly need to be compiled.

5

No, they do–or, at least, the templates use the DDC Pak in the engine directory on my current install. Problem is this behavior doesn’t seem to be consistent across different installs of the engine, even though Compressed.ddp is indeed present in //Engine/DerivedData at install time.

6

Well, these are not quite the same things though. If I understand correctly, shader instructions compilation (which is actually not supposed to happen, as the engine ships with them pre-compiled but seemingly incorrectly linked as mentioned above) builds all the shader instructions in an optimized, compiled manner (and probably does some optimization taking into account the materials calling these instruction in the engine content, for sure).

But while the creation of a new material does involve some compilation, it is absolutely not supposed to cause a recompile of all shader instructions of the engine !

In all cases, a lengthy shader compilation (of 6000+ shaders) is actually never supposed to happen when the editor is configured as intended - not on the first editor boot, and especially not when starting a new project as that would involve a huge amount of wasted time (up to 2hours per day, as mentioned by the OP). Unfortunately I feel like the widespread assumption that this is “normal” (and hype videos just skipping that step when it happens) is just causing the issue of it not being setup properly by default to not be resolved, and it’s been going on for a while now …

IMHO It is reaching a point where there is a definite need for a little helper tool, to be launched before the editor, making sure that all paths to the dcc cache are correct for the engine and for any given project. Otherwise people are just going to waste time waiting for it assuming that it is normal behavior, or assuming that their hardware isn’t appropriate.

1 Like
7

Many thanks, i have solved the problem by uninstalling UE4.27 and 5.0 and have made a fresh install. The shader cache seems to be set up properly now.

1 Like
8

I’m sorry but what is the solution for this issue then?
I also face the issue many times.
Whenever I add a new plug-in start a project, I always face something like 14000shaders compiling… even though I updated UE5.0.1

1 Like
9

Hello @nomisnibor ,
Unfortunately I have not found a definitive solution myself. Even though the compiled cache gets rid of the initial compile during splash screen, there always seem to be something triggering incredibely long shader compilation (or “preparation”, as they now call it) once a project opens up, one way or another - and sometimes the behavior seems to change from one session to another.

And since no official support has been provided here in this thread despite the view count here or in the other thread, there isn’t much one can do imho. So as far as I am concerned, the solution is to only use UE5 to test out single assets and simple BPs inside the thirdperson example, but not using it for anything more complex than that, and not importing any content pack (metahumans, Matrix contents) into it.

10

At least, I don’t feel alone in this terrible annoyement!
It is so weird though… :upside_down_face:
Loooooost of time wasted waiting for these shaders to prepare…

11

Hello,i could solve the problem by uninstalling all my ue version sand reinstalling the ue5.0. Now shaders do not compile anymore when not supposed to be needed or too slowly.

1 Like
12

I’m going to necro this post as UE5 seems to suffer from it the most. UE5.1.1 can take up to 30 minutes per project to load on a high end PC. This cache is usually generated once but if you run multiple projects this shader compiler keeps popping up. It’s on the splash screen. Isn’t there any way to skip shader compilation until we are IN the editor and make it a background process there?

2 Likes
13

Just a tip: by default shader compilation is set to launch worker threads with “LOW Priority”. You can set these to normal and high priority and it will speed up your compilation by changing a setting in an .ini file… Also you antivirus may have a slight impact…

You can read all about it here

1 Like
14

Hello. For me the solution was.

  1. setting shader compilation and unuse shader in base engine ini
  2. then go to visual studio 2019 and 2022 and update it.
  3. removing old, unnecessary versions of unreal engine 4.15, leaving one version 4.xx, the last one, and the new ue 5.3
    restarted and everything went to maximum.
    First compilin went about 5 minutes then nothing all clear.