How is this acceptable? (compiling 8000+ shaders on a brand new installation)

Community Community & Industry Discussion
, , , ,
1

I know this question has been asked by newbies a million times. Every time a veteran tells him it’s normal, nothing to worry about it. But this is exactly what stops people like me to make the first step into the amazing world of Unreal.

Since Unreal 4.x, until today’s Unreal 5 Preview 1, every time when it got a fresh install on the computer with absolutely no projects, no starter content, no any extra options installed, just the barebone UE4/5, the first time when you start the editor, it always takes 40 minutes to 5 hours to compile 2000-8000 shaders for an empty scene, with CPU being 100% occupied during the whole time. Sometimes when a UE update came out, this process happens again. Sometimes when you make absolutely no changes to your scene, it happens again. And everyone seems fine with it.

Some users replied:
"The default sky sphere is material/shader driven. The box your default pawn spawns on in the center of your screen has a material too. The camera’s editor mesh has a material applied, even though it’s hidden in-game. Then you have the default lights and such. (not to mention there’s also other things, like default engine content that is available even if starter content isn’t added to the project, but hidden from view)

Some people came up with hacks to edit INI files, or increase the priority of the worker instance, or even developed scripts to handle this issue. Is it possible for the team to handle it from the root?

Since UE team knows this handful of contents (less than a dozen? Def far less than 8000+ shaders from a blank scene) will be loaded almost every time, can they be included in the installation after they are compiled? So that they don’t have to be compiled every time? Our devs are the smartest people on earth. They can break polycount limitations, they can def do this!

People asked a million times, why does UE have to compile shaders when there is absolutely no shader in the scene? Can you compile only the used ones? Can you compile not at the time when the program starts, but when these shaders are loading into the scene? Can you show a placeholder shader to let users know that this shader is currently being compiled? Or, can you show more information about what is currently compiling instead of just the simple text of “45% - Compiling Shaders (8,348)…”

UE team is so capable that they are almost always the greatest driving force to push us so much closer to the actual matrix, but you are telling me that they can’t fix this single thing that wastes so many hours of millions of developers and hobbyists and people who want to dive in, each day for the past years, and an unknown number of years in the future? I just can’t believe it.

It’s NOT normal. The fact that everyone is used to it is NOT right! This is not who UE team truly is!

Thank you for your input. Our community is the best!

15 Likes
2

You can use the command line argument -LOG to have more detailed information of what is happening when the engine is starting. I actually made a shortcut to the engine with -LOG parameter added to it. Occasionally useful for debugging.

As far as the compile times, I haven’t really installed the engine “fresh” in a long enough time to judge how long it should take, but I can tell you that the CPU (and having 2-3GB ram per core) makes a large difference in shader compile times. Its one of those processes that scales very well with multiple cores.

If they included the shaders as compiled, then the file size could possibly be larger and put that stress on the network instead.

If the computer being unusable for most of the time is a problem, then in Windows you can set core affinity to the editor so that a couple of cores would not be used by the editor, keeping the PC as a whole more usable.

3

mine does that stuff in max 15min, since 4.2 or so.
What are specs of your hardware? You know unreal editor is not for weak setups.
And dont confuse minimal settings to PLAY unreal game with settings to confortably develop on unreal.

4

I don’t know about OP, but I’m doing this on a rig set up for VR. And switching to Forward shading somehow made it worse.
Core i7 10700K
16GB RAM
GTX 3080

I don’t know anymore how long these kinds of steps take, because I have learned that my work is simply done for a while when Unreal does one of these things (same with compiling), and leave to go for a walk outside.

Unity also used to be pretty annoying in this regard, until they stopped compiling unnecessary shaders, and brought it back to sub-second compilation times. But I don’t think they were ever in the multi-minute realm.

Shader compiling with glslang never took me anywhere near this long either. An argument could be made about Unreal doing complicated things. But I don’t see how you can end up with multiple thousands of shaders for an empty scene.

1 Like
5

As for how it is acceptable… well, the answer to that is a complicated cost benefit analysis.
It’s certainly quite silly how high this cost is. Perhaps the people with the skillset most suited to optimizing it aren’t the ones paying the cost the most, or perhaps they are deemed more valuable elsewhere.

I also suspect that there is an assumption that this kind of cost is somehow normal, or otherwise somehow the fault of the user.

2 Likes
6

sounds like the OP’s hardware is below min requirements.

When I was working on a machine that was slightly below min suggested, compile shaders takes a couple of min’s max… now with a machine that is slightly above the mind requirements, compile shaders is a non-issue.

You can adjust which shader permutations compile on a per project basis. But if you are trying to operate unreal with a potato, there is some times that you’ll just ahve to sit there and wait.

1 Like
7

Compiling shaders should take miliseconds.

It should especially not be noticeable when you make a change in a shader node graph. Not take 10 seconds for every change because you dared to enable forward shading. And that 10 seconds adds up when there’s somehow thousands of shaders to compile on a new project.

8

I just updated one of my projects from 4.26 to 4.27 and it did compile shaders at the start, like 700 of them, but it took like… maybe 10 seconds and it did not affect my editor use in any way.

But I have a processor with lots of cores and plenty of ram to spare. I did a bit of digging, and, interestingly, saw some remarks of how the process priority of the shader compiler is set to low, and changing the value made a big difference to some.

Now if I would guess, if you use a processor with fewer cores, or don’t have enough memory for the threads, or run a lot of programs while the compilation is happening, then the shader compiler would be starved for resources and use whatever leftovers it can get its hands on - which Windows won’t provide much because the process priority is set to low by default.

I’d assume it is set this way because the shader compiler is supposed to be treated like a background process so that you can work on other stuff while it’s processing.

With multicore processors this works fine because there will be enough idle threads left for the shader compiler. With setups that don’t have many threads, well… you may have issues.

9

Yeah, no, I remember trying this back when I started, and all it did was make Windows unresponsive. This was on a machine with 8 cores + hyperthreading (i.e.: 16 threads). How many cores are you talking about?

1 Like
10

I’m using 3900X (12/24) and 64GB RAM.

11

Had someone install onto a surface pro and it was unusable for 5 hours. You could build blueprints, but it would NOT accept user input during PIE until the shaders were done. In contrast, I have a 970m in one laptop and a macbook pro that finish shaders in a couple minutes. Then a 2080 in the tower that burps once and done. Hardware means a whole lot for UE4 and creative software in general.

1 Like
12

That`s not right, unless your missing your Derived Data Cache ,which is include when you install Unreal from the Launcher, Unreal should take only 10 or 15 seconds to start up, and less to load the default project.
If you compile the engine yourself then is possible you are missing you DDC if you use the flag WithDDC=false.

13

Is this DDC also supposed to cover Forward Rendering?

14

OP was talking about the first time you run the engine with no project.

1 Like
15

Happening to me, too. First launch of Unreal Editor on an M1 Max. I’ve been waiting for 20 minutes now for the first launch (at least I’m down to 3,667 remaining). I guess the verdict is - don’t fret - be patient. On the plus side it is is barely hitting my CPU while I wait so I can do other things. And look! Since I started writing it’s down to 3,492! It would be nice if the splash screen had a “time remaining” or more details in regards to the launch logs.

3 Likes
16

It took about 10min to compile shader after I hit “create project” (incl. raytracing) to start UE for the 1st time.
(I’ve just installed UE 5.0.1 on my brand new PC (i5-11400F, 16GB RAM, RTX3060 Ti, Win11) )

For me, as newbie it’s quite ackward UE’s behaviour, when you don’t know if it takes 10 min or a few hours acc. your info.

1 Like
17

It feels like the shader compile times just keep getting worse with each release.

This has been an ongoing issue for a long time now but believe it or not shaders didn’t always used to take so long. At some point things slowed down a lot compared at the early engine versions and it’s gotten worse ever since.

How people can continue to put up with it and not complain to epic i don’t know but i guess we have no choice until they do something because it is a joke now.

Other engines like unity etc don’t have an issue when it comes to materials and shaders. I’ve been trying nvidia omniverse recently and that can load up a scene pretty quick and you can watch all the materials load in within seconds to a minute or so on more complex ones. With unreal im waiting sometimes ten to twenty minutes and this is on a 12 core ryzen 5900x.

How can epic continue to think this is ok? Please fix it, optimize it or make it compile on the gpu somehow because this isn’t acceptable.

1 Like
18

With a deadline right in front of me, I am sitting here for over 2 hours, watching UE5.03’s splash screen saying “compiling 8xxx shaders”, then finally the editor opened up, showing “preparing shaders (7,190), 8%” for the past few hour. I just worked on this nearly empty project yesterday with nothing like this!
(Last night my computer restarted for some windows update, maybe that’s why?)

Why compiling 8000 shaders during UE5 launch (splash screen), then preparing these shaders again AFTER the editor is fully loaded? How to avoid this from happening again?

I haven’t got anything done for the whole day not knowing when I can get back to my work. Help please!

P.S. Someone asked me about my hardware. I know it’s not very capable by your standards, but NO software should take 5 hours to load on some days while 1 minute on other days.

My laptop CPU is Intel(R) Core™ i7-8750H CPU @ 2.20GHz, my GPU is laptop 2060 RTX. I only work on some very simple scene with no more than 3 objects moving around in sequencer, for offline rendering. No massive scenes or complicated BP, nor character animations. Not a video game development project as well. Just simple 3D animation projects. So strange…

19

should do this once when you create a new project. then you good to go. careful that when you chose some options it will trigger recompilation of all shaders. for example the second button under the Play in Editor button. That one play the preview for mobile which needs a recompilation of the shaders if not done already

1 Like
20

Ensure to set the the priority of all ShaderCompileWorkers to High in the Task-Manager

ygyu4h9iaky71
ygyu4h9iaky711080×1080 114 KB

5 Likes