Announcement

Collapse
No announcement yet.

Unreal Build Automation and Server Deployment

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

    Unreal Build Automation and Server Deployment

    ExtroForge (http://www.extroforge.com) has a fairly healthy team size (for a hybrid FPS/RTS/Building/Exploration PC game). We have Engineers, artists, modelers, writers, testers, social media folks, etc.

    We decided on a critical montra early on that ALL team members (not just engineers) would have access to the latest build at all times. It is simply critical for the various tasks that they have to perform that they can see and experience (and capture) the latest gameplay elements, visuals and mechanics. As ExtroForge is (above all else) a multi-player experience, we also wanted to ensure that a standalone server with the latest code was always available for testing on.

    As we were fairly new to the Unreal engine (and there is sparse info out there about any ‘recommended’ techniques), we fell back on processes we were familiar with from using other engines and environments. With several experienced engineers on the team, we reached into our toolbox to create a process that provided a good amount of automation as well as maintainability.

    We hope this information has value for other teams out there!

    http://www.extroforge.com/unreal-bui...at-extroforge/
    Last edited by casperjeff; 10-06-2015, 12:38 PM.

    #2
    Fantastic information, and a really great write up.

    Cheers for the valuable knowledge, even more timely for me as I sit here configuring teamcity+perforce to do the same thing.

    Thanks!

    Comment


      #3
      Glad it was useful - How do you like TeamCity?

      Comment


        #4
        Teamcity is solid. Eas easy to install (win box) and connecting to it super easy. Setting up the agents, piece of cake.

        Living in the great age of automated build services and cloud storage. Its awesome =p

        Comment


          #5
          This is excellent. Thank you.

          Jenkins does exactly what I need and your explanation and examples were spot on.

          WIP Thread

          Comment


            #6
            Thanks for the detailed writeup!

            I'm trying to set up an automated build for the BlackJack example project as a test. I used what looks like the same command line that you've got:

            "C:\Program Files\Unreal Engine\4.9\Engine\Build\BatchFiles\RunUAT" BuildCookRun -project="C:\Users\valtrain\.jenkins\workspace\Unreal BlackJack\BlackJack.uproject" -noP4 -NoCompile -platform=Win64 -allmaps -build -clientconfig=Development -cook -stage -archive -archivedirectory="C:\Users\valtrain\.jenkins\workspace\Unreal BlackJack\BuildOutput"

            but I'm getting this error:

            C:\Users\valtrain\.jenkins\workspace\Unreal BlackJack>"C:\Program Files\Unreal Engine\4.9\Engine\Build\BatchFiles\RunUAT" BuildCookRun -project="C:\Users\valtrain\.jenkins\workspace\Unreal BlackJack\BlackJack.uproject" -noP4 -NoCompile -platform=Win64 -allmaps -build -clientconfig=Development -cook -stage -archive -archivedirectory="C:\Users\valtrain\.jenkins\workspace\Unreal BlackJack\BuildOutput"
            Running AutomationTool...
            Program.Main: CWD=C:\Program Files\Unreal Engine\4.9\Engine\Binaries\DotNET
            Automation.ParseCommandLine: Parsing command line: BuildCookRun -project=C:\Users\valtrain\.jenkins\workspace\Unreal BlackJack\BlackJack.uproject -noP4 -NoCompile -platform=Win64 -allmaps -build -clientconfig=Development -cook -stage -archive -archivedirectory=C:\Users\valtrain\.jenkins\workspace\Unreal BlackJack\BuildOutput
            Automation.Process: IsBuildMachine=False
            Automation.Process: ShouldKillProcesses=True
            Automation.Process: Setting up command environment.
            CommandUtils.SetEnvVar: SetEnvVar uebp_EngineSavedFolder=C:/Program Files/Unreal Engine/4.9/Engine/Programs/AutomationTool/Saved
            CommandUtils.DeleteDirectoryContents: DeleteDirectoryContents(C:/Program Files/Unreal Engine/4.9/Engine/Programs/AutomationTool/Saved/Logs)
            WindowsHostPlatform.SetFrameworkVars: Setting .Net Framework environment variables.
            WindowsHostPlatform.SetFrameworkVars: Supports64bitExecutables=True
            WindowsHostPlatform.SetFrameworkVars: WindowsPlatform.CompilerVisualStudio2013
            WindowsHostPlatform.SetFrameworkVars: WindowsSDKDir=C:\Program Files (x86)\Windows Kits\8.1\
            WindowsHostPlatform.SetFrameworkVars: BaseVSToolPath=C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\Tools
            WindowsHostPlatform.SetFrameworkVars: Setting VS environment variables via C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\Tools\..\..\VC\bin\x86_amd64\vcvarsx86_amd64.bat.
            CommandUtils.SetEnvVar: SetEnvVar FrameworkDir=C:\Windows\Microsoft.NET\Framework\
            CommandUtils.SetEnvVar: SetEnvVar FrameworkVersion=v4.0.30319
            WindowsHostPlatform.SetFrameworkVars: FrameworkDir=C:\Windows\Microsoft.NET\Framework\
            WindowsHostPlatform.SetFrameworkVars: FrameworkVersion=v4.0.30319
            CommandEnvironment.SetupBuildEnvironment: WARNING: C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\Tools\..\IDE\Devenv.com does not exist.
            CommandEnvironment.SetupBuildEnvironment: WARNING: Assuming no solution compilation capability.
            ProjectUtils.CleanupFolders: Cleaning up project rules folder
            CommandUtils.DeleteDirectoryContents: DeleteDirectoryContents(C:\Program Files\Unreal Engine\4.9\Engine\Programs\AutomationTool\Saved\Rules)
            Automation.Process: Compiling scripts.
            ScriptCompiler.LoadPreCompiledScriptAssemblies: Loading precompiled script DLLs
            ScriptCompiler.LoadPreCompiledScriptAssemblies: Found 9 script DLL(s).
            BuildCookRun.SetupParams: Setting up ProjectParams for C:\Users\valtrain\.jenkins\workspace\Unreal BlackJack\BlackJack.uproject
            ProjectParams.ValidateAndLog: Project Params **************
            ProjectParams.ValidateAndLog: AdditionalServerMapParams=
            ProjectParams.ValidateAndLog: Archive=True
            ProjectParams.ValidateAndLog: ArchiveMetaData=False
            ProjectParams.ValidateAndLog: CreateAppBundle=True
            ProjectParams.ValidateAndLog: BaseArchiveDirectory=C:\Users\valtrain\.jenkins\workspace\Unreal BlackJack\BuildOutput
            ProjectParams.ValidateAndLog: BaseStageDirectory=C:\Users\valtrain\.jenkins\workspace\Unreal BlackJack\Saved\StagedBuilds
            ProjectParams.ValidateAndLog: Build=True
            ProjectParams.ValidateAndLog: Cook=True
            ProjectParams.ValidateAndLog: Clean=
            ProjectParams.ValidateAndLog: Client=False
            ProjectParams.ValidateAndLog: ClientConfigsToBuild=Development
            ProjectParams.ValidateAndLog: ClientCookedTargets=UE4Game
            ProjectParams.ValidateAndLog: ClientTargetPlatform=Win64
            ProjectParams.ValidateAndLog: Compressed=False
            ProjectParams.ValidateAndLog: UseDebugParamForEditorExe=False
            ProjectParams.ValidateAndLog: CookFlavor=
            ProjectParams.ValidateAndLog: CookOnTheFly=False
            ProjectParams.ValidateAndLog: CookOnTheFlyStreaming=False
            ProjectParams.ValidateAndLog: UnversionedCookedContent=False
            ProjectParams.ValidateAndLog: SkipCookingEditorContent=False
            ProjectParams.ValidateAndLog: NumCookersToSpawn=0
            ProjectParams.ValidateAndLog: GeneratePatch=False
            ProjectParams.ValidateAndLog: CreateReleaseVersion=
            ProjectParams.ValidateAndLog: BasedOnReleaseVersion=
            ProjectParams.ValidateAndLog: DLCName=
            ProjectParams.ValidateAndLog: DLCIncludeEngineContent=False
            ProjectParams.ValidateAndLog: AdditionalCookerOptions=
            ProjectParams.ValidateAndLog: DedicatedServer=False
            ProjectParams.ValidateAndLog: DirectoriesToCook=
            ProjectParams.ValidateAndLog: CulturesToCook=
            ProjectParams.ValidateAndLog: EditorTargets=UE4Editor
            ProjectParams.ValidateAndLog: Foreign=False
            ProjectParams.ValidateAndLog: IsCodeBasedProject=False
            ProjectParams.ValidateAndLog: IsProgramTarget=False
            ProjectParams.ValidateAndLog: IterativeCooking=False
            ProjectParams.ValidateAndLog: CookAll=False
            ProjectParams.ValidateAndLog: CookMapsOnly=False
            ProjectParams.ValidateAndLog: Deploy=False
            ProjectParams.ValidateAndLog: IterativeDeploy=False
            ProjectParams.ValidateAndLog: FastCook=False
            ProjectParams.ValidateAndLog: LogWindow=False
            ProjectParams.ValidateAndLog: Manifests=False
            ProjectParams.ValidateAndLog: MapToRun=
            ProjectParams.ValidateAndLog: NoClient=False
            ProjectParams.ValidateAndLog: NumClients=0
            ProjectParams.ValidateAndLog: NoDebugInfo=False
            ProjectParams.ValidateAndLog: NoCleanStage=False
            ProjectParams.ValidateAndLog: NoXGE=False
            ProjectParams.ValidateAndLog: MapsToCook=
            ProjectParams.ValidateAndLog: Pak=False
            ProjectParams.ValidateAndLog: Package=False
            ProjectParams.ValidateAndLog: NullRHI=False
            ProjectParams.ValidateAndLog: FakeClient=False
            ProjectParams.ValidateAndLog: EditorTest=False
            ProjectParams.ValidateAndLog: RunAutomationTests=False
            ProjectParams.ValidateAndLog: RunAutomationTest=
            ProjectParams.ValidateAndLog: RunTimeoutSeconds=0
            ProjectParams.ValidateAndLog: CrashIndex=0
            ProjectParams.ValidateAndLog: ProgramTargets=
            ProjectParams.ValidateAndLog: ProjectBinariesFolder=C:\Program Files\Unreal Engine\4.9\Engine\Binaries\Win64
            ProjectParams.ValidateAndLog: ProjectBinariesPath=C:\Program Files\Unreal Engine\4.9\Engine\Binaries\Win64
            ProjectParams.ValidateAndLog: ProjectGameExeFilename=C:\Program Files\Unreal Engine\4.9\Engine\Binaries\Win64\UE4Game.exe
            ProjectParams.ValidateAndLog: ProjectGameExePath=C:\Program Files\Unreal Engine\4.9\Engine\Binaries\Win64\UE4Game.exe
            ProjectParams.ValidateAndLog: Distribution=False
            ProjectParams.ValidateAndLog: Prebuilt=False
            ProjectParams.ValidateAndLog: Prereqs=False
            ProjectParams.ValidateAndLog: NoBootstrapExe=False
            ProjectParams.ValidateAndLog: RawProjectPath=C:\Users\valtrain\.jenkins\workspace\Unreal BlackJack\BlackJack.uproject
            ProjectParams.ValidateAndLog: Rocket=False
            ProjectParams.ValidateAndLog: Run=False
            ProjectParams.ValidateAndLog: ServerConfigsToBuild=Development
            ProjectParams.ValidateAndLog: ServerCookedTargets=
            ProjectParams.ValidateAndLog: ServerTargetPlatform=Win64
            ProjectParams.ValidateAndLog: ShortProjectName=BlackJack
            ProjectParams.ValidateAndLog: SignedPak=False
            ProjectParams.ValidateAndLog: SignPak=
            ProjectParams.ValidateAndLog: SkipCook=False
            ProjectParams.ValidateAndLog: SkipCookOnTheFly=False
            ProjectParams.ValidateAndLog: SkipPak=False
            ProjectParams.ValidateAndLog: SkipStage=False
            ProjectParams.ValidateAndLog: Stage=True
            ProjectParams.ValidateAndLog: bUsesSteam=False
            ProjectParams.ValidateAndLog: bUsesCEF3=False
            ProjectParams.ValidateAndLog: bUsesSlate=True
            ProjectParams.ValidateAndLog: bDebugBuildsActuallyUseDebugCRT=False
            ProjectParams.ValidateAndLog: Project Params **************
            Project.Build: ********** BUILD COMMAND STARTED **********
            UE4Build.Build: XGE was requested, but is unavailable, so we won't use it.
            UE4Build.Build: ************************* UE4Build:
            UE4Build.Build: ************************* ForceMonolithic: False
            UE4Build.Build: ************************* ForceNonUnity:False
            UE4Build.Build: ************************* ForceDebugInfo: False
            UE4Build.Build: ************************* UseXGE: False
            UE4Build.Build: ************************* UseParallelExecutor: False
            CommandUtils.Run: Run: C:\Program Files\Unreal Engine\4.9\Engine\Binaries\DotNET\UnrealBuildTool.exe UE4Editor Win64 Development -noxge -generatemanifest -NoHotReloadFromIDE
            CommandUtils.Run: Run: Took 2.4038966s to run UnrealBuildTool.exe, ExitCode=0
            UE4Build.PrepareManifest: Copied UBT manifest to C:\Program Files\Unreal Engine\4.9\Engine\Programs\AutomationTool\Saved\Logs\UBTManifest.0.xml
            CommandUtils.Run: Run: C:\Program Files\Unreal Engine\4.9\Engine\Binaries\DotNET\UnrealBuildTool.exe UE4Editor Win64 Development -noxge -NoHotReloadFromIDE -ignorejunk
            UnrealBuildTool: Creating makefile for UE4Editor (no existing makefile)
            UnrealBuildTool: Performing full C++ include scan (no include cache file)
            UnrealBuildTool: Building UnrealHeaderTool...
            UnrealBuildTool: Creating makefile for UnrealHeaderTool (no existing makefile)
            UnrealBuildTool: ERROR: Couldn't find target rules file for target 'UnrealHeaderTool' in rules assembly 'UE4ModuleRules, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null'.
            UnrealBuildTool: Location: C:\Program Files\Unreal Engine\4.9\Engine\Intermediate\Build\BuildRules\UE4ModuleRules.dll
            UnrealBuildTool: Target rules found:
            UnrealBuildTool: UE4Editor - C:\Program Files\Unreal Engine\4.9\Engine\Source\UE4Editor.Target.cs
            UnrealBuildTool: UE4Game - C:\Program Files\Unreal Engine\4.9\Engine\Source\UE4Game.Target.cs
            UnrealBuildTool: UnrealHeaderTool failed for target 'UE4Editor' (platform: Win64, module info: C:\Program Files\Unreal Engine\4.9\Engine\Intermediate\Build\Win64\UE4Editor\Development\UnrealHeaderTool.manifest).
            CommandUtils.Run: Run: Took 56.9662771s to run UnrealBuildTool.exe, ExitCode=5
            BuildCommand.Execute: ERROR: BUILD FAILED
            Program.Main: ERROR: AutomationTool terminated with exception:
            Program.Main: ERROR: Exception in AutomationUtils.Automation: Command failed (Result:5): C:\Program Files\Unreal Engine\4.9\Engine\Binaries\DotNET\UnrealBuildTool.exe UE4Editor Win64 Development -noxge -NoHotReloadFromIDE -ignorejunk. See logfile for details: 'UnrealBuildTool-2015.10.22-17.25.52.txt'
            Stacktrace: at AutomationTool.CommandUtils.RunAndLog(String App, String CommandLine, String Logfile, Int32 MaxSuccessCode, String Input, ERunOptions Options, Dictionary`2 EnvVars)
            at AutomationTool.CommandUtils.RunUBT(CommandEnvironment Env, String UBTExecutable, String CommandLine, String LogName, Dictionary`2 EnvVars)
            at AutomationTool.UE4Build.BuildWithUBT(String ProjectName, String TargetName, UnrealTargetPlatform TargetPlatform, String Config, String UprojectPath, Boolean ForceMonolithic, Boolean ForceNonUnity, Boolean ForceDebugInfo, Boolean ForceFlushMac, Boolean DisableXGE, String InAddArgs, Boolean ForceUnity, Dictionary`2 EnvVars)
            at AutomationTool.UE4Build.Build(BuildAgenda Agenda, Nullable`1 InDeleteBuildProducts, Boolean InUpdateVersionFiles, Boolean InForceNoXGE, Boolean InUseParallelExecutor, Boolean InForceNonUnity, Boolean InForceUnity, Boolean InShowProgress, Dictionary`2 PlatformEnvVars)
            at Project.Build(BuildCommand Command, ProjectParams Params, Int32 WorkingCL)
            at BuildCookRun.DoBuildCookRun(ProjectParams Params)
            at BuildCommand.Execute()
            at AutomationTool.Automation.Execute(List`1 CommandsToExecute, CaselessDictionary`1 Commands)
            at AutomationTool.Automation.Process(String[] CommandLine)
            at AutomationTool.Program.MainProc(Object Param)
            at AutomationTool.InternalUtils.RunSingleInstance(Action`1 Main, Object Param)
            at AutomationTool.Program.Main()
            ProcessManager.KillAll: Trying to kill 0 spawned processes.
            Program.Main: AutomationTool exiting with ExitCode=5
            Domain_ProcessExit
            copying UAT log files...
            RunUAT.bat ERROR: AutomationTool was unable to run successfully.
            BUILD FAILED
            Build step 'Execute Windows batch command' marked build as failure
            Finished: FAILURE


            Do you have any idea what might be wrong?

            Thanks!

            Comment


              #7
              Nevermind, I tracked down my issue. I had the command line wrong.

              Thanks again!

              Comment


                #8
                Hi valtrain!

                I'm having the same error, what was wrong with your command line and how you got it working?

                Thanks!

                Comment

                Working...
                X