Distributing custom build to team

OK, but that still tell us nothing about how to build binary distro of engine LIKE THE ONE SHIPPED BY EPIC
For now I use ‘-game -engine -precompile’ to generate my project files, to speed up compilation time and build development/shipment targets,
anyone figured out how to build engine the way that allows us to use ‘-game -rocket’ (now they compile just fine, but will not link, due to missing *.lib files inside Engine/Win64 (UE4-Engine-Win64-Shipping.lib and others))

So the main question now is (for me): how to build UE4-Engine-Win64-Shipping.lib (and others) in Engine/Win64 :slight_smile:

You should build Shipping configuration. I believe you should run build with Shipping parameter, maybe -clientconfig=Shipping like UEEditor do.

After a few hours of working at the UAT code, I was actually able to get the original RunUAT command given by Ben Marsh to run to completion on Linux (after changing -TargetPlatforms=Win32+Win64 to -TargetPlatforms=Linux). You have to:

  1. force the if block at line 1153 of LegacyBranchSetup.cs to execute
  2. force the if block at line 607 of LegacyBranchSetup.cs to execute

Now I have no idea what the consequences of changing these 2 locations are so it would be awesome of someone more knowledgeable about the build system could weigh in. Just wanted to share my two cents.

Building shipping configuration (from UE4.sln in vc2013) does not produce those files - try and see it for yourself :slight_smile:
Thats why I’m asking how to build them in the first place …

could you post the content of those lines (1153 and 607) - I have what’s on promoted branch and now the line 1153 is a comment and 607 is a LogVerbose :wink:

I apologize, I thought I had the Development (no target) working, but I only was only able to get the Development Editor and DebugGame Editor to work. Apparently its similar issue to what MXADD mentioned with missing UE4-*.lib. The one it was trying to link to was the UE4-Launch.lib in the Binaries/Win64.

I’m also using 4.8.3 the zip download from github, which I don’t believe it has the LegacyBranchSetup.cs in that build, I tried searching for it. I think I remember seeing it being in the 4.9 version though.

If I find anything more, I’ll post an update.

Oh I forgot to mention I’m using 4.9.2-release

Line 607 is:


 
if (DoASharedPromotable)
{ 
    var AgentSharingGroup = "Shared_EditorTests" + HostPlatformNode.StaticGetHostPlatformSuffix(HostPlatform);


and Line 1153 is:



if (NumSharedAllHosts > 0)
{
    BranchConfig.AddNode(new GameAggregatePromotableNode(BranchConfig, HostPlatforms, ActivePlatforms, Branch.BaseEngineProject, false, bNoIOSOnPC));


@Harley_J – I’v experimented with 4.8.3, 4.9, 4.9.2 and the latest content from /promoted branch … they all face that same ussue, if the -rocket is used to generate visual studio solutions, then the development & shipping builds would not link
(ofcourse generating project with -engine OR -engine -precompile works fine, but thats esentially leaves me with distributing ‘source’ distribution among team members, bot the binary one)

@thinick - thaks I’ll look at it tomorow in /promoted branch … I have no incredibuild here, so checking it now would take ages (or an hour) :confused:

Wow!! I actually think that worked for the 4.8.3 version. However, those if cases that thinick reference are located in GUBP.Automation.cs line 5735 and 6281 for the 4.8.3 version. I also had to add a BlankProject at UnrealEngine/Samples/Sandbox/BlankProject.uproject. I just opened UE4Editor and created a blank Blueprint. It has to be Blueprint a noncoded program. I also had to add the references in the ./Templates and ./Samples in the UE4Games.uprojectdirs. Because it complained about finding BlankProject and the FP_Person templates. Which that explains some of the above issues described. I also added ./FeaturePacks just in case because I didn’t want to wait in case it screwed up, doubt that needed to be in there since there are no *.uproject s in there. One other thing it complained about was missing Engine/Build/build.properties file. I just created a blank one with nothing in there which got passed that point.

The only draw back was it took a very long time. Not sure if there is a way to cut down on that possibly due to the ./Templates.

Just want to say thanks to everyone. Hopefully, there aren’t anymore issues like this.

Yes, you are right. That’s how they are named on mac build, and for game dlls ue is producing -Shipping files, not for UE libraries.

Bump again - anyone - as I’m out of ideas, and providing precompiled source distribution is a real pain :confused:

@MXADD Not sure how far you got. The one thing I noticed is our company finally received the Unreal 4 version from our distributor. It came with additions that did not come with the download from GitHub. I’m trying to find out where they grabbed it from. I think it was pulled from Epic’s perforce repository, but not quite sure. It included a bunch of Samples including Sandbox\BlankProject, Sandbox\DocumentationCode and Sandbox\ExtendingTheEditor. Also it contained the build.properties file. So apparently just calling
“RunUAT.bat gubp -Node=GatherRocket -CleanLocal -TargetPlatforms=Win32+Win64” as described above just works without any modifications. So I’m guessing those if cases that tginick described make those cases true without modifying.

However, I did get it to work with just the version I grabbed from GitHub 4.8.3. I also included tginick modifications to it. Built those scripts and automation tool before calling the RunUAT.bat command above.
I also created a BlankProject that is Blueprint only and make sure it has the path Samples/Sandbox/BlankProjcet/BlankProject.uproject, so the UE4Games.uprojectdirs should include ./Samples/Sandbox/ since it needs to search one level deep to find the .uproject. You also need to include ./Templates/ into that file too because the rocket script runs through those templates. I’m not sure if ./FeaturePacks should also be included, but I just stuck that in there just in case. Also make sure you have a build.properties file in the Engine/Build directory. I just had it blank, but from the other pull from our distributor contained a TimestampForBVT, EngineVersion, and BranchName. However, with the above modifications and just having a blank Build.properties I got it to work. After all those changes are made and compiled the script and automation tool, then do the RunUAT.bat call. See if that works.

I have to run through the Rocket script anyways because it creates a local copy of the rocket build which increases the timing for the script. So if I discover anything new, I’ll share it.

I know it’s not an elegant solution, but I use robocopy (via .bat files) to copy the relevant files to a network share. I have two .bat files, one to commit the modified engine and another one the team uses to “fetch” the updates to their machines.

commitEditor.bat:


@ECHO OFF

SET _source=<path to unreal folder root>
SET _dest=<path to network shared folder>
SET _wildcards=*.exe *.dll *.bat *.sh *.command *.map *.uproject *.u* *.png *.jpg *.jpeg *.bmp *.tga *.ico *.ttf *.fon *.xml *.txt *.version *.glsl *.hlsl *.ini *.vdf *.rt *.template *.py *.mel *.mb *.wav *.ogg *.config *.res *.icu *.cnv *.cfu *.brk *.dict *.nrm *.spp *.tps *.chm *.html *.udn
SET _options=/XD .git DerivedDataCache Intermediate Saved source /PURGE /XO /S /E /TBD /DCOPY:AT /FFT /R:3

robocopy %_source% %_dest% %_wildcards% %_options%

@pause

fetchEditor.bat:



@ECHO OFF

SET _source=<path to network shared folder>
SET _dest=<path to unreal folder>
SET _wildcards=*.*
SET _options=/PURGE /XO /S /E /TBD /DCOPY:AT /FFT /R:3

robocopy %_source% %_dest% %_wildcards% %_options%

@pause


Thanks for sharing your knowledge so far.

I have done some testing myself, and managed to create a Rocket-style build. My findings so far are in this Git repository: https://github.com/Kalmalyzer/UE4RocketBuild

Currently I can create a Rocket style build without doing any source code changes. A Win64-only build becomes 12GB in size, including template projects.

I haven’t worked out a good way to distribute and register the build to team yet, but I suspect that registration would just be a matter of assigning a non-GUID ID and manually registering it in the same way that the logic in UnrealVersionSelector does it, followed by running UnrealVersionSelector.

It appears that the Github download has some dependencies missing. So in order to get that to work without source code changes (I’m using version 4.8 right now), you’ll need the following:

  1. A blueprint project named “BlankProject” and place it in …\Samples\Sandbox\BlankProject. (that fulfills a GUBP check)
  2. A code cpp project that is promotoable. You can make the project Promotable by inserting the following into target.cs file (this fulfills the check for Promotable or shared hosts)
    public override GUBPProjectOptions GUBP_IncludeProjectInPromotedBuild_EditorTypeOnly(UnrealTargetPlatform HostPlatform)
    {
    var Result = new GUBPProjectOptions();
    Result.bIsPromotable = true;
    return Result;
    }
  3. You’ll need a build.properties file in …\Engine\Build\build.properties. The data in there doesn’t get used by these nodes, so it can be blank.
  4. Make sure that the UE4Games.uprojectdirs file has the paths for templates, the blankProject and the promotable project. Remember this searches one level deep for the *.uproject file.

I believe that is all.

I ended up writing a custom node script to avoid copying every single file over that overwrote already up to date files and to remove some dependencies that I don’t need to speed up time.

The build process is complicated as what was said above, so I would recommend reviewing the Nodes GatherRocket and all the dependencies, which are GUBP.VersionFilesNode, GUBP.ToolsForCompileNode, GUBP.RootEditorNode, Rocket.StripRocketEditorNode, GUBP.GamePlatformMonolithicsNode, Rocket.StripRocketMonolithicsNode, GUBP.ToolsNode, GUBP.MakeFeaturePacksNode, Rocket.StripRocketToolsNode, Rocket.BuildDereivedDataCacheNode, and Rocket.FilterRocket. These will give you an idea of what is happening. There are some other catches in GUBP script with perforce settings and github, but I didn’t get into those.

The RunUAT gubp command with a specified node, will run that node and dependencies. You can pass in -OnlyNode parameter which will not run the dependency nodes. Also -NoDDC param will skip the Building Derived Data Cache if you don’t need that. Its kind of an optional thing to have, but mainly used when you want a shared DDC or speed up loading assets for others.

@Kalms
What we have right now for the distribution are two repositories. One for editing the source and having the build machine compile and committing those to the second repository which is the distribution for the team. Not quite sure if this will stay because it takes like 20 minutes for us to even svn::update changes from the distribution.

The end result for GatherRocket creates a local build directory of Rocket and all that is in there is the distribution. They do have a PublishRocketNode which does zip it up in case you decide to to that.
I’m not sure what Version Source Control you are using, they do have some perforce and git settings in the scripts. I haven’t looked at them because we are using subversion. :frowning:

The install is mainly getting that distribution that the rocket output and run the editor. When running the editor, it should register the build path to a registry key which gives an engine associate ID. (Windows HKLM\Software\EpicGames\UnrealEngine\Builds). There is also the UnrealVersionSelector.exe that can be run to register build path, however, you will need to include this in the filter (…\Engine\Build\InstalledEngineFilters.ini), so the rocket build will copy it over. I believe the UnrealVersionSelector will register the keys for doing the context menu (switch version, generate project files). You can associate the game project *.uproject with the installed engine either using the context menu (right click the *.uproject) switch version. That should pop up a list of builds and just select the one to associate with. Or by running Editor you can choose the project through the Project Browser window and select the *.uproject. This will probably give you a message saying it was build in different engine, from there you should be able to choose more options and convert within. Once the *.uproject file is registered to the installed engine, just double clicking should open the editor and the game project.

Anyways, I hope that helps. :slight_smile:

Harley_J, thanks for the insight.

We have a near-identical setup now: repository A contains engine source code, repository B contains pre-built binary distribution(s). Using Perforce.

In order to make the engine version available to themselves, people have to do two things: 1) fetch latest binaries from Perforce, 2) run a batch script which creates a registry key with a predefined name. This way, the engine will can have an identifier like “4.10.0-release-forGameX” instead of a GUID. Also, everyone on the team will then have the same engine identifier on their machine. This means that the .uproject file (which is checked into source control) can have an EngineAssociation that is appropriate for everyone without needing to do changes.

(I am relying on UnrealVersionSelector for all the other Windows registry work)

Some time ago I wrote a small python script (windows only) that does that that, ensure a given EngineUUID to be set into the registry (GitHub - Tragnarion/UE4_RegisterEngine: Register a given engine using a fixed UUID, useful when having). You can just reuse the binary from the release section and just create a bat file with the right params to set the UUID of your choice.

Thanks @Kalms and @Moss for the registry idea.

So there is one thing that I am running into, not sure if anyone has encountered it.

When building maps with Lightmass, running into issues with Swarm. What’s weird is the InstalledEngineFilters.ini doesn’t contain the Swarm libraries and executable. Which caused the editor to crash when clicking on the build button to build the maps. So I added those swarm files in the Binaries/DotNet folder. Now the editor will spawn Swarm, but it fails to connect. I’ve been clearing the cache like some forums were saying. However, I feel like something is missing because the actual compiled version of Unreal 4 (not the rocket build) works perfectly fine with Swarm.

So I feel like the rocket build is missing more than just the libaries and executables. Does anyone know what it could be? FYI, I am using 4.8.3 version.

This is the log I’m getting from swarm:
12:44:45 PM: Starting up SwarmAgent …
12:44:45 PM: … registering SwarmAgent with remoting service
12:44:45 PM: … registering SwarmAgent network channels
12:44:45 PM: … initializing SwarmAgent
12:44:45 PM: … certificate check has failed
12:44:45 PM: … initializing cache
12:44:45 PM: … using cache folder ‘C:/Users/…/AppData/Local/UnrealEngine/4.8/Saved/Swarm\SwarmCache’
12:44:45 PM: … recreating SwarmAgent cache staging area
12:44:45 PM: … initializing connection to SwarmCoordinator
12:44:45 PM: … using SwarmCoordinator on RENDER-01
12:44:48 PM: … SwarmCoordinator failed to be initialized
12:44:48 PM: … initializing local performance monitoring subsystem
12:44:49 PM: … initialization successful, SwarmAgent now running

Ok I answered my own problem, but I’ll document it here.

I looked at a diff with Epic’s installation from the Launcher and the rocket build. I noticed some files missing from the rocket build.
I ended up adding to the InstalledEngineFilters.ini:

/Engine/Binaries/DotNET/Swarm*
/Engine/Binaries/DotNET/AgentInterface*
/Engine/Binaries/Win64/AgentInterface* // this one fooled the **** out of me because it was in two directories
/Engine/Binaries/DotNET/RPCUtility*
/Engine/Binaries/DotNET/UnrealControls*

There might be some that are not needed, but I just included these because I saw it in the Epic’s installation.

Do you really need a Rocket build? I ended up just having a repo where non-programmers got a pre-build engine, not rocket but a normal source build with all binaries and pdbs. We do quite a lot of engine changes so in our case pushing a rocket build took far to long compared to just push/pull (I’m not using git for this one though) the files that changed.