Announcement

Collapse
No announcement yet.

Current advice for distributing custom engine builds to team without recompiling?

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • Current advice for distributing custom engine builds to team without recompiling?

    Hi all,

    I've recently switched over to the source build of the engine and have started making changes (i.e. FASTBuild Windows integration thanks to KnownShippable & CLxS on Github), and want to make sure that this custom build of the engine can be easily distributed to a team without compiling it on each dev's machine. It would also have the advantage, hopefully, of being as convenient to use as the launcher version, i.e. not compiling against full source = faster, less chance of accidental rebuilds, etc. This way, I could build the engine on a separate, dedicated machine, then copy the binaries back to my main dev PC + teammates' PCs. Convenient!

    What's the current advice for this? Building as Rocket (as per this post) is not an option - the build system will exclude server build targets if using Rocket. As a last resort, I'll look to take all of those out, but apparently Rocket builds are impressively complex. Would rather not have to mess with it. Either way, I'm looking to keep those server build targets in the distributed/binary build.

    If you've successfully done this, please let me know! There's not much around on this topic.

    Thanks.
    Last edited by HateDread; 09-19-2016, 11:17 PM.

  • #2
    I used to make my own "semi-rocket" build in 4.12, basically I use "-precompile" flag to tell UBT to make precompiled libraries, and wrtie a bat file to collect files into my own "rocket" build folder (by comparing files difference between source build and launcher build). Not sure will apply to your situation ( I did not build build server target )...

    Btw, how is FastBuild UE4 integration performance? Will it help on signle machine build?
    | twitter | github | #ue4tip

    Comment


    • #3
      This has been changed recently, rocket is being phased out I think.

      On 4.12, just doing a regular build, then adding an empty InstalledBuild.txt to the Engine/Build directory along with a config file addition (see here), was enough to prevent engine and engine plugin rebuilds in the context of a project.

      If you look at the Build Updates section at the end of this 4.13 post however, it gives some details on what I guess is a more flexible method for determining exactly what gets put into a distributable engine build. I haven't tried using it. I'm pretty sure the above approach still works in 4.13 if that's enough for you.

      Comment


      • #4
        Does someone know if I build a module from the source (no change on public interface), if I can just send the new dll to the team without sending the full rocket built directory?

        thanks,

        Comment


        • #5
          Originally posted by kamrann View Post
          This has been changed recently, rocket is being phased out I think.

          On 4.12, just doing a regular build, then adding an empty InstalledBuild.txt to the Engine/Build directory along with a config file addition (see here), was enough to prevent engine and engine plugin rebuilds in the context of a project.

          If you look at the Build Updates section at the end of this 4.13 post however, it gives some details on what I guess is a more flexible method for determining exactly what gets put into a distributable engine build. I haven't tried using it. I'm pretty sure the above approach still works in 4.13 if that's enough for you.
          An empty InstalledBuild.txt leads to the following error when trying to run GenerateProjectFiles.bat.

          Click image for larger version

Name:	GenerateProjectFiles_Error.JPG
Views:	1
Size:	37.5 KB
ID:	1115717

          I added the following to Engine/Config/BaseEngine.ini:

          [InstalledPlatforms]
          +InstalledPlatformConfigurations=(PlatformName="Win64", Configuration="DebugGame")
          +InstalledPlatformConfigurations=(PlatformName="Win64", Configuration="Development")
          +InstalledPlatformConfigurations=(PlatformName="Win64", Configuration="Shipping")
          +InstalledPlatformConfigurations=(PlatformName="Win32", Configuration="DebugGame")
          +InstalledPlatformConfigurations=(PlatformName="Win32", Configuration="Development")
          +InstalledPlatformConfigurations=(PlatformName="Win32", Configuration="Shipping")
          Any ideas? I figured the point of the above was to not have to tie in a specific game into the engine!

          Comment


          • #6
            I think you should only add the text file and the ini entries after having built the engine. Once it's there, it's essentially saying "Don't build the engine, it's been prebuilt", which presumably accounts for the GenerateProjectFiles.bat failure.

            So this solution is just to mark an engine build as installed/prebuilt. It doesn't affect the actual process of building the engine or what gets built. If you need control of that, I think you need to look at that new build graph stuff.

            Comment


            • #7
              Originally posted by kamrann View Post
              I think you should only add the text file and the ini entries after having built the engine. Once it's there, it's essentially saying "Don't build the engine, it's been prebuilt", which presumably accounts for the GenerateProjectFiles.bat failure.

              So this solution is just to mark an engine build as installed/prebuilt. It doesn't affect the actual process of building the engine or what gets built. If you need control of that, I think you need to look at that new build graph stuff.
              Thanks; that definitely solved the issue. The resulting setup still isn't what I'm after - it protects the engine from being rebuilt under some circumstances, but sometimes the engine apparently does really need to be rebuilt even though I'm only working on a project, at which point I actually need to remove that text file to build. Not quite a Rocket-level of ease of use. I wonder if there's a way to pass something or set some flag so that generating project files for the game project results in a VS project like binary builds - no 'Programs', no project build dependency on 'ShaderCompileWorker', etc.

              I've been looking at the new BuildGraph system, but I can't seem to get it off the ground. I passed this to RunUAT.bat:

              BuildGraph -target=”Make Installed Build Win64” -script=Engine/Build/InstalledEngineBuild.xml
              But it complains with

              BuildGraph.Execute: ERROR: Missing -Script= parameter for BuildGraph
              Tried swapping out for absolute paths just in case, but same issue.

              Any advice, folks?

              Comment


              • #8
                Turns out my quotation marks from the above weren't valid, so the argument parser wasn't seeing the commands properly. I've got that all sorted now.

                Currently running with:

                BuildGraph -script="L:\_Programming\Repositories\UnrealEngine\Engine\Build\InstalledEngineBuild.xml" -target="Make Installed Build Win64" -set:HostPlatformOnly=true -set:PublishDir="L:\_Programming\RocketBuilds\"
                Which starts everything up as expected, however there is one problem; the build processes that this starts all ignore the BuildConfiguration.xml properties I've set, and you can't pass anything in to these processes because the Automation Tool starts them for you. I really need

                <bAllowXGE>false</bAllowXGE>
                <ProcessorCountMultiplier>2</ProcessorCountMultiplier>
                to be accepted into the build process. The BuildConfiguration.xml file is read when doing builds manually, and for me lives in %AppData%\Roaming\Unreal Engine\UnrealBuildTool, but I've also put a copy in Engine\Saved\UnrealBuildTool. I know you can also put one in the Documents folder, which will override these, but I have none there.

                These files work 100% for normal builds triggered via Visual Studio, but are completely ignored when builds are started internally via BuildGraph/Automation Tool, it seems.

                Any ideas?

                Comment


                • #9
                  Managed to get the build to work, and with XGE/Incredibuild not involved via -NoXGE argument. Still would be nice to get BuildConfiguration.xml involved, although the reason for this not working like 'normal' builds becomes clear later on.

                  "L:\_Programming\Repositories\UnrealEngine\Engine\Build\BatchFiles\RunUAT.bat" BuildGraph -script="L:\_Programming\Repositories\UnrealEngine\Engine\Build/InstalledEngineBuild.xml" -target="Make Installed Build Win64" -set:HostPlatformOnly=true -set:WithWin32=false -NoXGE -set:WithIOS=false -set:PublishDir="L:\_Programming\RocketBuilds\"
                  HostPlatformOnly means that only the platform you're currently using will be built. For me, that meant that Win32 would also be built, but I only wanted Win64, hence the -set:WithWin32=false argument. I tried to exclude IOS in case it was erroneously included as a result of HostPlatformOnly, but it seems like there's a mistake in the build process - even if you don't have it included, your build will still fail if you didn't download the IOS dependencies via Setup.bat.

                  My -set:PublishDir="L:\_Programming\RocketBuilds\" argument was ignored, and the engine was built into Engine/LocalBuilds. Within that folder, I had two different folders of similar size - 'Engine' and 'InstalledDDC', the former was 15.5GB, and the latter 14.0GB. What's the difference? They seemed to both have roughly the same contents - the requested binaries, the entire engine, the same folders, etc.

                  Also, I was looking to incorporate the FASTBuild changes I mentioned earlier into the BuildGraph-triggered build process, but it seems like it follows a different code path to normal builds. UE4Build.Build method ignores BuildConfiguration.xml and then calls into UE4Build.XGEPrepareBuildWithUBT to create XGEItem objects, passing the List<XGEItem> to UE4Build.ProcessXGEItems, which either passes the items to XGE/Incredibuild distributed system, or ParallelExecutor.Execute for local threaded compilation, depending on commandline arguments.

                  The issue is that this is different build process to that triggered via VS builds (of both projects and the engine itself), which call into ActionGraph.ExecuteActions with a List<Action> parameter that can be easily used for custom distributed compilation tools. By comparison, the XGEItems are just for each high-level build config, and are linked to a generated XGE .xml file, and this ProcessXGEItems function then passes another XGE .xml file to ParallelExecutor, which does its own work to generate a List<BuildAction> (which is different again from the List<Action> used in ActionGraph) and is responsible for threading out the compilation.

                  Some guidance on at least a good interception point to get a List<Action> of the current build configuration would be awesome. Actually, any more info on this process and how/why it is like it is would be appreciated! I'm sure I'll uncover more over time.

                  EDIT: An example of the FASTBuild changes to ActionGraph.ExecuteActions (courtesy of ClxS on Github, and Liamkf of KnownShippable):

                  Code:
                  // --> FASTBuild
                  if (BuildConfiguration.bAllowFastbuild)
                  {
                      ExecutorName = "Fastbuild";
                  
                      FASTBuild.ExecutionResult FastBuildResult = FASTBuild.ExecuteActions(ActionsToExecute);
                      if (FastBuildResult != FASTBuild.ExecutionResult.Unavailable)
                      {
                          ExecutorName = "FASTBuild";
                          Result = (FastBuildResult == FASTBuild.ExecutionResult.TasksSucceeded);
                          bUsedXGE = true;
                      }
                  }
                  // <-- FASTBuild
                  Something simple like this would be great in the BuildGraph builds!
                  Last edited by HateDread; 10-01-2016, 11:31 AM.

                  Comment

                  Working...
                  X