1 line code change == 602 file rebuild

I change one line of game code in my dedicated server and it rebuilds 602 engine files almost every time.

for every hour I spend approximately 35 minutes goes to wait for this.

well if you change an engine dependency, you should probably expect that.

So, what are you changing?

You need to pay attention to the include chains. If you touch a header file that’s included in a bunch of header files, that seemingly innocent line change will trigger a big rebuild.
Also, if you touch a file included in a module’s precompiled headers, the whole module will have to be rebuilt.
Carefully organizing your includes has a huge impact on build times for small code changes.

Thanks for the replies, but I was simply complaining , lol to vent some frustration and now I regret the post, it’s pretty lame… :frowning:

No, sorry fellas, I don’t think these are the specific reasons, but you do make sense. Reason: these are completely innocuous changes in bottom chain .cpp files, e.g. deleting a comment, remove empty lines, etc, rebuild. I actually already know what causes it, not sure WHY, though: it’s to do with constantly changing from DebugGame Editor and DebugGame Server build configs.

So it’s not every build, but basically when I switch configurations.

Right now, I’m working in some areas that require builds and packaging and unfortunately I’m just going to have to get along with it.

Update:

This issue sucks.

Sometimes, I need to package just to update a version number in a .ini, and I get between 750-840 engine files to rebuild.

Since I posted the first time about this, I’ve done approximately 1200 packages and this occurs about 1 per 3 builds and lasts about 20 minutes per.

1200/3=400
400x20 = 8000
8000/60=133.3
133.3x60=7980.

THis is the cost I have paid for this time wasted.

Just sayin

incredibuild or sndbs, and dedicated build machines are all worth their cost. for sure.

1 Like

Thanks for the reply.

I’ve got more research into this issue, it turns out the problem is between doing VS builds and server packaging. When switching, is what seems to trigger it.

I discovered that UE4 was switching between VS2017 and VS2019 randomly, and thought that might be the problem, but after ensuring the VS2017 compiler was being used by the Editor, the issue still persists.

It’s really quite a strange issue, I’m still looking into it, maybe one day I’ll be able to figure it out, I’ve just for now had to figure out how to minimize the need to switch between C++ builds and Editor builds.

I’ll post here with updates.

Ugh:

image

:frowning:

From 6 lines of code changes in one of my BP library .cpp files…

you are surely doing something that doesn’t jive well with normal procedures, but hard to say what. I don’t know what you mean about ‘between doing vs builds and server packaging’.

Could be a consequence of using VS17, if you’re using 4.27 VS22 works great, i’d recommend nuking 17 and 19 and switching up.

You are configured to build Development-Editor right?

So, I’ve tried a couple things:

I fully cleaned rebuilt the entire engine. It took just over 5 hours to rebuild the engine. As an aside, the doc states 10 mins to 2 hours, depending on the machine. I have a pretty good machine. I can’t imagine the specs of a machine capable of a 10 minute build.

The cleaning and rebuilding so far seems to have helped, although I can’t confirm. I’ve switched between packaging builds and VS builds several times and thus far, it has only rebuild files I have touched, and those others affected, all at the lowest levels of the chain, thus, only my game project files.

My project uses 4.26.4 and as of yet, I am not ready to go 4.27. There are several issues I need to work out before I can switch, namely with for ‘some strange reason’ my Steam networking completely does not work in any way in 4.27 and I don’t have time to rework all that at the moment. I’ve already ripped out all the Steam functionality, since that in itself is a whole slew of other issues.

To your question about ‘VS builds and server packaging’:
When I say VS builds, specifically I mean compiling the project in Visual Studio for c++ changes, not using the Editor as it’s necessary to select the Server build configs. Then, when creating the Server build, it needs then to be packaged.

Typically, this has been a situation where if you build in Visual Studio, then build your package via the editor, there is no need for the packager to rebuild the files you just build in VS.

But it does AND they are deep in UE4 engine files.

I switch my configurations regularly between DebugGame, Development and Shipping. When the base engine files have been build for these configurations, they should not need to be rebuilt for simply game project changes.

I definitely want to thank you for your input on this. It’s a very expensive issue for me, but in the current moment, the Full Rebuild may seem to have ‘fixed’ something…

Just another quick update:

As again, I made a C++ change in VS and did a server and editor build, and proper files were recompiled.

I then went in editor and packaged the server, which as I would expect, did not recompile any files, as they were already built via VS. I packaged the game then for the client, and it did rebuild 10 of my project files, but this is OK and expected.

I guess at this point, it is possible the issue simply had to do with something in the UE4.26 Intermediate files, and the clean and rebuild may have addressed it, but Time will tell.

For now, I’ll just be thankful it seems to be somewhat cleared up or reduced for now

i7-10700K running @ 5100 with 48gig ram here, and building solo it takes about 2 hours to complete build game project + engine. WIth my studio’s parallel build system, which has about 50 machines of varying grades both below and above mine, it gets down to about 30 minutes (although that is significantly affected by network bandwidth, people local to the build system can get it done in about 20). I imagine you’d need 128GB+ and a ThreadRipper to get the entire thing done any faster, maybe several. But intermediate builds that don’t rebuild engine should take a lot less time. Even with a very, very large by most people’s standards (I think?) game project, it’s usually only a few minutes for me to build my actual game package. However, we do have a sizeable quantity of code that is in separate modules, as well. Which is something that can really help, depending on organization.

I’m not sure if 4.26 works with 2022 or not, but either way, I would completely nuke whatever version of VS you aren’t using. A couple of weeks ago, I just did a complete check out and rebuild from scratch because somehow my existing tree got screwed up and part of it was building with the 2019 toolchain, and part with t he 2022 toolchain … and i have no idea how.

Anyway, yeah, I fixed that with a total fresh checkout from source control, and rebuilt everything.

1 Like

Thanks for the specs - yes, when all works for me, my project builds and packaging usually don’t take more than at most 2-3 minutes, even if I’ve made a lot of changes. I have a AMD Ryzen 7 1700X Eight-Core Processor@3.40 and 32 gig ram. A clunker compared to yours. So maybe 5 hours isn’t too bad… lol. I’ll have to check into a higher version of VS…

As another update, I’ve done several builds and haven’t run into the issue again yet. So that is really good news. I hope it continues…

Sorry to hear about the rebuild from scratch. The fact that it takes so much time is a large deterrent to checking it as an issue, due to the cost of time.

I hope that I continue to get these builds as they have been, because it will save me hours, but it will be bitter sweet, because I delayed so long in trying an engine rebuild because of how long it takes…

press build, go to sleep :slight_smile: yeah i know devs sometimes don’t sleep much :smiley:

finding the bottlenecks when you do have to build is annoying, though. i started on this project with a i7-4700k, and went through upgrading to SSDs, upgrading memory, and finally went for a whole system upgrade, which is how i ended up with the 10700. There’s even a quite noticeable difference in Unreal between the SATA SSD and the M.2 SSD.

In general usage, I can’t really tell a difference between my old machine and new, but in Unreal, it’s a huge difference in every action.

1 Like

Bummer:

Guess that wasn’t the issue. Anyway, at least I got 2 days worth of decent builds, but I’m not rebuilding the engine again lol:

image

I’m not sure what I did wrong. I did move a lot of macro code into a separate file, but when I rebuild under DebugGame Editor in VS, it only needed to build 3 files, the 3 I made changes to, 2 .h files and 1 .cpp file.

Then I opened the editor, and did a server package…

had you built the server configuration since you blew away the intermediate?

What’s the build log look like up until Building 788 actions?

Hmm, so come to think of it, I nuked the game project’s intermediate folder, since I got some linker errors when I compiled - in VS. Then I rebuild the project an it basically built all the files, but only the game project files.

I then did the server package and it did the 788 file build. But since, I’ve built between each of the types (VS and package) and it’s back to building the correct files.

So, in this case, maybe because I did delete the intermediate files, there could be something there that triggers the large rebuild.

The build log looks very typical up to that point, it isn’t actually until you get to that line, that there is any indication of doing a large rebuild. I’ll pay closer attention next time it happens.

An update:

I’ve been getting builds that are appropriate to the state of the changes made now for just about 30 days. I’ve determined that the issue lay somewhere in the Intermediate/Build/Win64/… of the game project directory.

While I haven’t determined the specific cause, it appeared that a random file in that particular directories was changing and triggering a complete rebuild of the engine projects in that directory, basically ActorLayerUtilities thru WindowsMoviePlayer. It basically should be that once these are created, they should remain unchanged until there is a source change to those projects, and then at that, only the one in which the source changed.

So then, since I rebuilt the full engine as referenced above, only expected full game project builds have been triggered.

Unfortunately, I have not yet detected what is causing it, but I think if the issue starts again, then I’ll set up some monitors and note what could be acting on those files unexpectedly.\

Until then, I am marking this as resolved and hopefully it can be of assistance.

One thing to note is that rebuild automatically does a clean so you build from scratch. If you choose build in VS it will do an iterative build. Not sure if you are already aware of this but figured I’d mention it.