UE5.1 First start after installation (Preparing shaders....)

I’ve just installed UE5.1 and when i first started a new blank project with RT on the cpu usage jumped to 90% and it preparing shaders (7000+) and its kindataking forever
my CPU threadripper 3975WX
any ideas with the compatibility with my CPU?
is this supposed to be happening?

2 Likes

I’m having the same problem. Mine wants to prepare 28,000 shaders. It has taken about an hour, and it has only finished about 15,000 of them. I am using an i9-9900k, overclocked and water cooled. CPU is cranking at 100%. This is crazy.

1 Like

Done the same recently with a watercooled 3950X. Few things make this monster hit 90+ C, Preparing Shaders is one of them.

Hello,

First of all, all users should report this just like you did as this situation is IMHO completely unacceptable. No production tool (no matter how advanced) should take more than a couple seconds to launch. The more users do not report this, the more likely it is to never going to be addressed. It was already painful in UE4 and now it reached a peak level of absurd. I can only assume that people working at Epic just don’t see the issue because their IT probably sets them up with a pre-compiled cache or something similar. At first I though that my somewhat older CPU was the issue, but thanks to your report it is clear that it isn’t the actual root cause.

Secondly, I am not sure exactly what solved it but I’ve personally attempted two things to address this (after one initial compile that I let run fully once) :

  • Setting up a shared cache location in the preferences so that the compiled shaders can be shared between projects. The documentation is very confusing on this, but if I understand correctly this is supposed to help.
  • If a new project gets stuck at 45% on editor launch, I kill the process and copy both “saved” and “deriveddatacache” folders from a previously opened project onto the new one. Once that is done, no compilation seems to happen - at least on editor startup splash.

While researching the issue I’ve read somewhere in the docs that this waiting time for shader compilation is actually not supposed to happen, as the engine is supposed to be shipping with all its default shader components already compiled. But this is obviously not the case as many of us are experiencing the initial compile. Perhaps the install location has something to do with this ?

8 Likes

Here, perhaps? It’s down in the DDC Pak section, and yes, according to this, Launcher builds are supposed to ship with a DDC Pak. The fact that 5.0 seems to not have, along with possibly multiple 4.0 versions, seems like an oversight or misconfiguration on the release pipeline side.

2 Likes

Hello @Nebulon_Ranger - indeed that’s the page I saw.

So at the very least that’s progress and confirms that there are at least two things wrong with the release : 1 - it should ship with it, and 2 - it’s not supposed to happen again on newly created projects, but it does.

As said I cannot tell for sure which action solved it on my end, but I’d be curious to see if a test protocol could repro and identify things clearly.

I am very confused now - not so much by the fact that it happened, but by how this has not been immediately addressed in an official post here or in some of their other channels …

2 Likes

I’ve lodged a bug report, let’s see how far this goes.

I explicitly told Epic to try the reproduction steps outside their own network, because chances are quite high that they won’t be able to repro in an environment with a network-shared DDC.

I was working with Unity in my last project and shader compiling is compared to unreal lightning fast there, also UE5 crashes a lot! I don’t understand why they put it as a release version, it still feels like early access. They should stop adding features and focus more on improving/fixing what they have.

2 Likes

@Nexusmaster : well to be fair it and if I understand correctly it could be two very different things : there is the compilation of all the base shader instructions used by the engine, which is suspected to not be shipping properly with UE5 at this time even though it should (that’s the 6000+ shaders thing) ; and then there’s the compilation of any new master material on top of that (which shouldn’t take more than a second or so in any engine really).

So perhaps Unity has checks in place to be 100% sure that the engine ships with everything precompliled properly and ready to use, while UE does not. In any case there is indeed something very, very wrong with the way UE is setup at this time. I have the feeling that many users just assume that it’s just the way things are and got used to it somehow.

Indeed one can suspect that Epic devs just don’t see it thanks to their specific deployed environment, and I suppose that people doing hype videos about the engine probably just edit that part out, in fear that the long compilation might make their viewer think that they have an underpowered workstation (hence looking less legitimate somehow) ?

@Nebulon_Ranger Care to share a link to the bug tracker ? (if public of course).

Also, why is this thread referring to “UE5.1” ? Is there a new build somewhere ?

I’ve submitted the report but it hasn’t been acknowledged and entered into their jira yet. Hopefully I hear something on that front before the end of the week.

The plot thickens!

I decided to reinstall UE5 just to peek into its file structure, and it does, in fact, ship with a DDC Pak. Why it doesn’t seem to be using it is beyond me, perhaps a configuration issue?

Reinstalling it from scratch huh ! I admire your dedication @Nebulon_Ranger :smiley:
Alrighty, that means two things then :

1 - There is one very specific setting (out of the few confusing ones…) that allows to plug either that one pre-compiled pack or a user-compiled one, for sure. I’ve tried a few different things myself and overall it is indeed working (as I haven’t seen any lengthy compile in days), but I am not sure which one of my settings did the trick specifically.

2 - There seems to be something else going on with the per-project Saved and DerivedDataCache folders too, as I am positive that copying these over from one project to another solved the second round of shader compilation for me (the one that is shown on the bottom right corner of the editor, post launch). But I may be remembering incorrectly.

FWIW here are my settings. I’ve plugged in four different paths (one for “Global” and “Global Shared”, and one for “Project Local” and “Project Shared”) and I honestly have no idea which one did the trick, nor do I know if it is pulling from the pack shipping with the engine or my own compile of it. But the little Derived Data icon is green, so I suppose there is at least one correct setting out of the four :smiley:

(Overall I am still completely amazed that such a high-end tool doesn’t come with a reliable configuration wizard to make sure that these things are setup properly. I understand that the user is supposed to be technically minded, but here in this case I am pretty sure that hundreds of thousands man-hours have been wasted by this nonsensical shader compilation issue. And with online forums being pretty much dead now, it just remains unsolved and mostly not reported. This is incredibly frustrating !)

3 Likes

Okay what the Hell. This shader compile behavior seems to be inconsistent, too, since I just started a new project on the Third Person template and it seems to have used the precompiled DDC! I also created a test with RT enabled to see if that was the variable at play, and it did “prepare shaders” post-launch, but only about 110 of them rather than 8000, and still looks to have used the precompiled DDC.

Something strange is afoot.

Well, if anything there is also probably some amount of crossing of the beams related to cfg files too. I’ve attempted to search for one of my own four ddc path strings in all the .ini files from my installation folder and it is not giving me any result, so the setting itself must be stored somewhere else altogether …

In any case, Compressed.ddp definitely seems like the first thing to try and hook up for sure for anyone getting a 6000+ shader compilation wait on startup. I wouldn’t be surprised if it only hooked itself properly if the engine installation path is left as default, for instance. Or something along those lines …

I’ve been waiting all day to start a new UE5 project. Every time I restart the editor I have to compile 8,000ish shaders, and then when I’m in the engine it prepares 10k more. I figured it was normal the first time, because that’s just how a new engine install is. But currently UE5 is unusable since there’s a 2+ hour wait every time I start it up. I tried specifying the DCC paths as suggested by PO_art, so we’ll see if that helps at all, but if not UE5 is just off the table for me until there’s a fix =(

1 Like

Hello @Blindspy4 - from what I can see on my machine once the path is correct it should be back to a decent launch time the very next time around. Perhaps it can be confirmed without an editor restart by looking at the status at the bottom right of the screen ? I would imagine that it will turn green as soon as a dcc pack is found or linked to, but I can’t confirm that.

And if you allow me : It really isn’t “how a new engine install” is or should be, there’s no justification for it and users should never take it as normal. Because if they do then we end up with software taking ages to load because inadequate librairies and all kind of unnecessary bloat (just like this forum reskin, actually). No productivity tool should take more than a minute to load, and ideally it should be near instant - like Blender launching in seconds as opposed to Autodesk apps taking forever, even though they do the same thing.


It would be great if a few of the 490 people who’ve seen this thread could at least mention if they do get the 6000+ shader compilation as well :smiley: And if they do, please mention if linking to a dcc path solved the issue.

1 Like

Here’s a new tip :

I have had a project crash on me (upon taking a HighResShot …). And when relaunching it it started to compile all 6000+ shaders again on splash. I force-quit the launch and copy-pasted the Config, Saved, and DerivedDataCache folder from another project onto this one. This successfully allowed to skip the useless compile.

Again I cannot tell for sure which of the three folders did the trick, but it worked in this particular instance as it seemingly re-plugged the link to the cache.

Oddly enough the “Derived Data” indicator is still gray in the project though. But it worked.

DD

This shoes that this icon does not turn green when a cache is loaded properly. Therefore, I don’t know what the green tint actually means when it happens.

1 Like

At last, I have figured out what’s happening!

The engine seems to maintain two DDCs: one in //Engine/DerivedDataCache (the compressed DDC Pak that ships with the engine) and one in %APPDATA%/Local/UnrealEngine/Common/DerivedData-this is the DDC at issue here. If this gets removed, has its name changed or the engine is otherwise unable to find it, it gets regenerated at next boot. This is what makes the engine go through the lengthy shader compile step.

You want to stick this second DDC on a network share if you can to ensure that it never gets deleted and is always in a place where the engine can find it.

2 Likes

Hi there again @Nebulon_Ranger,

Interesting - here on my end I am seing some numbered folders (0 to 9) as well as some folders named “Buckets”, “Contents”, and “TestData” inside AppData\Local\UnrealEngine\Common\DerivedDataCache. Is that what you mean ?

(it is mindblowing that UE is actually doing that, putting crucial files deep inside an APPDATA folder that is supposed to only be used for storing simple user preferences. It’s almost as bad as the recent trend of applications straight up installing themselves there. Ungh !)

My question would be then : in the case that these files get deleted for on reason or another, what should one do ?

Yep! This is a DDC; if you look into the hexadecimal folders inside Content you’ll see a bunch of UDD files. This DDC isn’t very large compared to Compressed.ddp, which I suspect contains a lot more different sets of compiled shaders given its size, nor does it generally have the hit rate, but it seems to be required.

The only solution for now is to let the engine regenerate it, then copy the whole folder somewhere you’re confident it won’t be deleted and change the global shared DDC path to point to it.

1 Like