Distributing custom build to team

Development Programming & Scripting C++
, , ,
1

Hi all,

I have made some modifications to the editor and engine for internal use and I look into the best way to distribute this to the rest of my team. One thing I noticed is that when making new c++ projects based on the engine compiled from github is that it will recompile a lot of the engine source when the ‘Development’ configuration is used. I made one blank c++ project from the source compile from github and from the 4.7.6 build supplied by Epic through the launcher to compare the difference. The project from the official build does not recompile the engine source.

One thing I noticed was that the projects derived from the github source build recompile a lot of the plugins into static libs with a unique name. For example, I made a bunch of test projects in the last few weeks to prototype stuff and for every one I have a .lib file located in UnrealEngine\Engine\Plugins\2D\Paper2D\Binaries\Win64 all called [ProjectName]-Paper2D-Static.lib. The same happens with all the other plugins.

Before I distribute the modified engine to our team I would want to disable this behavior and make it behave more like the distribution from Epic. One thing I noticed is that the engine provided by Epic does not have the UE4Client/UE4Editor/UE4Game/UE4Server .Target.cs files supplied and also not the source to the programs like UnrealHeaderTool. It does have the source to the engine but not to the plugins. But I could not find any documentation of what to include and exclude.

My question is how to make a distribution of a modified version of the engine for my team that behaves similar to the one provided by Epic. What steps need to be done and are there tools/scrips available to do this?

Cheers,
Pjotr

1 Like
2

I would be interested on this too: did you look at answer hub?

I think the useful keyword is that you need to do a "rocket build.

3

Rocket is the code name that we give to the launcher-distributed version of UE4. As-of 4.8, you should be able to build it yourself using this command line:

cd Engine\Build\BatchFiles
RunUAT.bat gubp -Node=GatherRocket -CleanLocal -TargetPlatforms=Win32+Win64

The build process is pretty convoluted, but I’d be happy to answer questions about it. Most things are included by virtue of being build products, but there are rules in Engine/Build/InstalledEngineFilters.ini that specify extra files to include.

The thing about plugins using different prefixes for different projects is because UnrealBuildTool lets you set up a custom compile environment for monolithic targets by default. The general idea is that normally all games use the same editor binaries, but when you’re compiling just your game you might want to turn on some particular compiler optimization or linker switches. You can override that behavior in your target.cs file by returning true from the ShouldUseSharedBuildEnvironment() function. We disable it for all games in Rocket, precisely because we want everyone to be able to use the same static libraries.

“Rocket” encompasses everything particular to that distribution of the engine though, and may not be tailored to your needs. Some functionality exists to allow you to create “precompiled” builds of the engine for your own game without it being a “Rocket” build per-se: if you compile your game, appending -precompile to the UnrealBuildTool tool command line, it will create static libraries for any modules you list in the GetModulesToPrecompile() function of your target rules. You can run GenerateProjectFiles.bat with the -useprecompiled flag to have it treat everything you’ve precompiled as read-only.

1 Like
4

That sounds pretty good. Thank you for the write up. Looking forward to 4.8. :slight_smile:

5

Hi, Ben!

Thank you for this information, it is very valuable!

I have a few questions:

  1. Is there any way to recompile certain engine modules without recompiling the rest of them? Provided that UBT is launched with -useprecompiled option and all engine static libraries exist. I keep .dll and .lib files under source control but I don’t keep .pdb and .obj files there. I want to be able to quickly make changes to private parts of certain modules and rebuild only these modules. This would allow faster iteration on engine changes.

  2. When using -useprecompiled option, UBT still rebuilds plugins (obviously because they are not precompiled). Is there any way to tell UBT to not try to build plugins?

If this is not currently possible, what kind of modification can be made to UBT to implement these behaviors?

6

You can use the -module XXX argument with UBT to just compile a specific module (or set of modules, if it’s specified more than once). It’s not integrated into Visual Studio or anything like that though, I’m afraid.

The precompiling of plugins is hard-coded in SetupPlugins() in UEBuildTarget.cs (the code under the comment ‘Set the list of plugins that should be built’).

7

@Ben Marsh
Thank you! It worked! :slight_smile:

Here’s the fix I made to UBT, I post it here in case it is useful for the main engine:



			// Set the list of plugins that should be built
			if (bUsePrecompiled)
			{
				// Only build project plugins
				BuildPlugins = new List<PluginInfo>();
				foreach (PluginInfo Plugin in EnabledPlugins)
				{
					if (Plugin.LoadedFrom == PluginLoadedFrom.GameProject)
					{
						BuildPlugins.Add(Plugin);
					}
				}
			}
			else
			{
				if (bPrecompile && TargetType != TargetRules.TargetType.Program)
				{
					BuildPlugins = new List<PluginInfo>(ValidPlugins);
				}
				else
				{
					BuildPlugins = new List<PluginInfo>(EnabledPlugins);
				}
			}
1 Like
8

@Ben Marsh - I get an error when I run the command. Any idea what is wrong?

Excerpt of the output that includes the error:

Full output: G:\windows-slave-1\windows-build\build\UnrealEngine4\Engine\Build\BatchFiles>Run - Pastebin.com

9

FWIW it’s not working here either. Getting “ERROR: Exception in mscorlib: The given key was not present in the dictionary.” as above. I am seeing this in the very latest release branch (4.9.1), and also on 4.8.3

Full log: Unreal 4.9.1 RunUAT failure · GitHub

10

I have the same issue. “ERROR: Exception in mscorlib: The given key was not present in the dictionary.” Has anyone got a fix or workaround? I’m using the following command line: RunUAT.bat gubp -Node=GatherRocket -CleanLocal -TargetPlatforms=Win32+Win64

11

Likewise, same issue. Anyone found a solution yet?

12

Hi there,

I’ve recently built out a custom version of UE4 for my company to use as we needed to make some changes to the renderer.
After building out the engine and testing it on my own machine, I am now ready to roll it out to the rest of the team.

One thing I have noticed is that the size of the engine version is around 42GB compared to the standard version at 11GB.

A lot of this extra data is held in Engine/Intermediate/Build/Win64.

The standard engine has two folders in here “UE4” and “UE4 Editor” where as my custom version has these plus “Inc” “UE4Game” and “UnrealHeaderTool”.

Are there any console commands I can run that will clean out all the extra stuff and just leave me with a smaller build I can distribute to my team.

Thanks!

13

You can take out the following three folders when copying/distributing to the team:



.git
Engine/Intermediate
Engine/DerivedDataCache

Cheers,

14

Fantastic, I’ll give it a go!

Cheers

15

I too have that same error as Stormwind (Program.Main: ERROR: Exception in mscorlib: The given key was not present in the dictionary.)
Any ide how to fix this ?

16

After cloning fresh copy from ‘promoted’ branch istead of ‘release’ one
the command:
RunUAT.bat gubp -Node=GatherRocket -CleanLocal -TargetPlatforms=Win64
runs for a ‘while’ (seems to build at least ‘development editor’ couse the working editor exist in binaries/win64 after execution) but fails suddenly wit following log



SignRocketNode.SignFiles: Signing code: d:\UnrealEngine4\UE4\Engine\Binaries\Win64\UnrealVersionSelector-Win64-Shipping.exe -> d:\UnrealEngine4\UE4\Engine\Saved\Rocket\Signed\Win64\Engine\Binaries\Win64\UnrealVersionSelector-Win64-Shipping.exe
SignRocketNode.SignFiles: Signing code: d:\UnrealEngine4\UE4\Engine\Binaries\Win64\BootstrapPackagedGame-Win64-Shipping.exe -> d:\UnrealEngine4\UE4\Engine\Saved\Rocket\Signed\Win64\Engine\Binaries\Win64\BootstrapPackagedGame-Win64-Shipping.exe
SignRocketNode.SignFiles: Signing code: d:\UnrealEngine4\UE4\Engine\Binaries\Win32\BootstrapPackagedGame-Win32-Shipping.exe -> d:\UnrealEngine4\UE4\Engine\Saved\Rocket\Signed\Win64\Engine\Binaries\Win32\BootstrapPackagedGame-Win32-Shipping.exe
CodeSign.SignMultipleIfEXEOrDLL: Signing up to 160 files...
TempStorageManifest.Save: Saved temp manifest d:\UnrealEngine4\UE4\Engine\Saved\TmpStore\SignTools.TempManifest with 161 files and total size 175083489
GUBP.ExecuteNode: AutomationTool.AutomationException: Cannot find project 'FP_FirstPerson' in branch
   at AutomationTool.BranchInfo.FindGameChecked(String GameName) in d:\UnrealEngine4\UE4\Engine\Source\Programs\AutomationTool\AutomationUtils\ProjectUtils.cs:line 788
   at Rocket.BuildDerivedDataCacheNode.DoBuild(GUBP bp) in d:\UnrealEngine4\UE4\Engine\Source\Programs\AutomationTool\Scripts\RocketBuild.Automation.cs:line 1300
   at AutomationTool.LegacyNode.DoBuild() in d:\UnrealEngine4\UE4\Engine\Source\Programs\AutomationTool\GUBP\Nodes\LegacyNode.cs:line 77
   at GUBP.ExecuteNode(BuildNode NodeToDo, Boolean bFake) in d:\UnrealEngine4\UE4\Engine\Source\Programs\AutomationTool\GUBP\GUBP.cs:line 1365
Program.Main: AutomationTool exiting with ExitCode=1 (Error_Unknown)
Domain_ProcessExit
AutomationToolLauncher exiting with ExitCode=1
copying UAT log files...
BUILD FAILED

anyone knows what this ‘Cannot find project ‘FP_FirstPerson’ in branch’ is all about ?
do I miss some files or what ?

Anyway, I’m trying to build binary distribution of UE4 to distribute this to my co-workers (for now, to eliminate the possibility that some of my changes in core and renderer broken things, I’m just pulling ‘standard’ versions from git and trying to build them, to mimic the binary distro).

The problem is that if I remove ‘SourceDistribution.txt’ file from Engine/build directory and switch my game project (c++ based) to use the newly build engine, I’m able to compile game modules with ‘development editor’ config (-rocket used for project files generation), but when it comes to generate ‘development’ or ‘shipping’ builds (all of those for win64) - I see error about being unable to unclude some .generated.h files.
By looking at the makefiles I see that the directory for them changed from Engine\Intermediate\Build\Win64\UE4Editor\Inc to Engine\Intermediate\Build\Win64\UE4\Inc
but after (somewhat failed) building with ‘RunUAT.bat gubp -Node=GatherRocket -CleanLocal -TargetPlatforms=Win64’ command, there is no Engine\Intermediate\Build\Win64\UE4\Inc directory at all.

If I open UE4.sln and explicitly build ‘development’ and ‘shipping’ the problem with missing includes disapears (the Engine\Intermediate\Build\Win64\UE4\Inc is generated)
BUT when it comes to linking, it ends with error, there are missing all the *.lib files from Engine\Binaries\Win64 directory (for example for ‘shipping’ build it complains about 'UE4-Launch-Win64-Shipping.lib ')
HOW should I generate those files ?
If anyone messed up with this and figured this out I would be very grateful for any tips in this matter.

17

bump - anyone ?

18

I’m going to bump this one too because I also ran into the dictionary key issue as well “ERROR: Exception in mscorlib: The given key was not present in the dictionary.”.

Has anyone figured this one out what key is not in the dictionary for that rocket script and how to solve it?

19

I had some time this week to revisit this subject and I have the same error as everyone else. I used the current release branch for this (which is at 4.10 preview 3).
It seems the error is happening in GUBPBranchConfig::FindNode(string Node)
It tries to get the node with the name "“UE4_Win64_Mono_Precompiled” but that one does not exist in the map/dictionary.
This is a copy/paste of the nodes that are in the map:



{[VersionFiles, GUBP+VersionFilesNode]}	System.Collections.Generic.KeyValuePair<string, GUBP.GUBPNode>
{[ToolsForCompile, GUBP+ToolsForCompileNode]}	System.Collections.Generic.KeyValuePair<string, GUBP.GUBPNode>
{[RootEditor, GUBP+RootEditorNode]}	System.Collections.Generic.KeyValuePair<string, GUBP.GUBPNode>
{[Tools, GUBP+ToolsNode]}	System.Collections.Generic.KeyValuePair<string, GUBP.GUBPNode>
{[InternalTools, GUBP+InternalToolsNode]}	System.Collections.Generic.KeyValuePair<string, GUBP.GUBPNode>
{[LinuxTools, GUBP+ToolsCrossCompileNode]}	System.Collections.Generic.KeyValuePair<string, GUBP.GUBPNode>
{[InternalTools_BuildPatchTool, GUBP+SingleInternalToolsNode]}	System.Collections.Generic.KeyValuePair<string, GUBP.GUBPNode>
{[NonUnityTestCompile, GUBP+NonUnityTestNode]}	System.Collections.Generic.KeyValuePair<string, GUBP.GUBPNode>
{[MakeFeaturePacks, GUBP+MakeFeaturePacksNode]}	System.Collections.Generic.KeyValuePair<string, GUBP.GUBPNode>
{[ToolsForCompile_OnMac, GUBP+ToolsForCompileNode]}	System.Collections.Generic.KeyValuePair<string, GUBP.GUBPNode>
{[RootEditor_OnMac, GUBP+RootEditorNode]}	System.Collections.Generic.KeyValuePair<string, GUBP.GUBPNode>
{[Tools_OnMac, GUBP+ToolsNode]}	System.Collections.Generic.KeyValuePair<string, GUBP.GUBPNode>
{[InternalTools_OnMac, GUBP+InternalToolsNode]}	System.Collections.Generic.KeyValuePair<string, GUBP.GUBPNode>
{[InternalTools_BuildPatchTool_OnMac, GUBP+SingleInternalToolsNode]}	System.Collections.Generic.KeyValuePair<string, GUBP.GUBPNode>
{[NonUnityTestCompile_OnMac, GUBP+NonUnityTestNode]}	System.Collections.Generic.KeyValuePair<string, GUBP.GUBPNode>
{[ToolsForDocumentation, ToolsForDocumentationNode]}	System.Collections.Generic.KeyValuePair<string, GUBP.GUBPNode>
{[CodeDocumentation, CodeDocumentationNode]}	System.Collections.Generic.KeyValuePair<string, GUBP.GUBPNode>
{[BlueprintDocumentation, BlueprintDocumentationNode]}	System.Collections.Generic.KeyValuePair<string, GUBP.GUBPNode>
{[StripTools, Rocket.StripRocketToolsNode]}	System.Collections.Generic.KeyValuePair<string, GUBP.GUBPNode>
{[StripRootEditor, Rocket.StripRocketEditorNode]}	System.Collections.Generic.KeyValuePair<string, GUBP.GUBPNode>
{[SignTools, Rocket.SignRocketToolsNode]}	System.Collections.Generic.KeyValuePair<string, GUBP.GUBPNode>
{[SignRootEditor, Rocket.SignRocketEditorNode]}	System.Collections.Generic.KeyValuePair<string, GUBP.GUBPNode>
20

Yep, I ran into that very same spot in 4.9. Apparently in the dictionary there is not any key that even starts with UE4*. lol. I haven’t tried 4.8, but a co-worker tried it and ran into the missing branch/BlankProgram or something like that.

I kind of gave up with that problem because it seems like there is an essential step that I’m missing. So I’ve been trying to do this process manually by writing my own script to build the Development and Development Editor then doing a clean up on some files in the Intermediate\Build which are mainly the .obj. The only files I see that need to be in the Intermediate is the Inc directory(all .cpp and .h) and the *.lib in the build directory and mabe a couple *.res (All this was in the Intermediate directory). I’ve been kind of doing a compare & contrast with Epics installed version. I also used this person’s suggestion to create rocket project files with the context menu, http://www.unrealengine.academy/2015/08/18/something-between-regular-and-rocket-project-2/ . Every thing seems good with building a game in Development, but I am having issues with DebugGame Editor configuration when building the game project. I’m getting linker errors, so I’m kind of retracing my steps on this one because I got it to work before.

If anyone figures out this RunUAT.bat thing, I definitely would like to know

Update 10-30-15:
So I found out why the DebugGame configuration wasn’t working, the linker errors were from a scaleform integration that we are using. Without it, the above works fine.