Run automated testing from command line

Development Platform & Builds
, , , ,
21

How do you run the command line in your jenkins pipeline? using the bat command?
I have this:

stage( "UnitTests" ) {
    try {
        bat "\"${env.UE_ROOT_FOLDER}\\Engine\\Binaries\\Win64\\UE4Editor-Cmd.exe\" \"${env.WORKSPACE}\\${env.PROJECT_NAME}.uproject\" -ExecCmds=\"Automation RunTests ${env.PROJECT_NAME}\" -unattended -nopause -testexit=\"Automation Test Queue Empty\" -log -log=RunTests.log"
    } catch ( Exception e ) {
    }
}

but before the editor tries to run the unit tests, I have this error:

[2018.11.06-18.45.01:561][  0]LogD3D11RHI: Error: D3DRHI->GetFactory()->CreateSwapChain(DXGIDevice, &SwapChainDesc, SwapChain.GetInitReference()) failed 
at D:\Build\++UE4\Sync\Engine\Source\Runtime\Windows\D3D11RHI\Private\Windows\WindowsD3D11Viewport.cpp:126 
with error 887A0022

I guess thatā€™s because the editor is created in the bat process and does not have a chance to create the RHI (I tried with -NullRHI, with the same error)

22

OK I found the issue. The slave was connected to the master from a Windows Service, which does not have the rights to create the DX11 swap chain.
When connecting using the JLNP file, it does work :slight_smile:

Iā€™ll have a look at the warnings regexp tomorrow :slight_smile:

23

I am running it from a batch script, containing the command line I posted above:

	stage("Run Unit Tests") {
		steps {
			bat 'Tools\\BuildSystem\\run_unit_tests.bat'
		}
	}

Your problem might be that your Jenkins build machine needs to have a DirectX 11 GPU, and also perhaps a user session open. -NullRHI should solve this, I think, but it crashed on UE4.20, I had to backport a fix from 4.21 to have it work

24

I had a look and indeed the cleanest way would be to comvert the JSON file from the report into a valid jUnit XML file. I quickly looked for any plugin which would help do that but did not find any.
So for now, I donā€™t export the report, and I configured a Warnings task to parse the log.

Here is the regexp:
^\s*.]AutomationTestingLog: Error:\s(.)((\d+)): (.)(.*)$

And the mapping script:
import hudson.plugins.warnings.parser.Warning
import hudson.plugins.analysis.util.model.Priority

String category = "Unit Tests"
String fileName = matcher.group( 1 )
String lineNumber = matcher.group( 2 )
String message = matcher.group( 3 )
Priority priority = Priority.HIGH;

return new Warning( fileName, Integer.parseInt(lineNumber), "UE4", category, message, priority );
25

I managed to do just that, and wrote a blog post about it : UE4 Unit Tests in Jenkins | MICHAEL DELVA

26

adding small note:

The opened session frontend issue also affects 4.16 so most likely every version above until 4.21

27

You can save presets which are saved into a config file. Can a saved preset be used in a command line invocation? It would be great to configure the list in the Session Frontend and then use it in a command line invocation. Using the name of the preset in the -execcmds is a fail.

28

Hi,
there is any chance that someone has a clue on how to run Automation Tests with command line in an UE5 project?
It seems that the cmd lines changed from ue4, I donā€™t see anything when I try to launch [UEBinaryPath]UnrealEditor-Cmd.exe "[myProjectPath]MyProject.uproject" -unattended -nopause -NullRHI -ExecCmds="Automation List.
Did I miss something ?

EDIT: Something weird: When i run all tests like so[UEBinaryPath]UnrealEditor-Cmd.exe "[myProjectPath]MyProject.uproject" -unattended -nopause -NullRHI -ExecCmds="Automation RunAll; quit" -TestExit="Automation Test Queue Empty" -log -log=RunTests.log -ReportExportPath="!PROJ_DIR!\TestsReports\reports\ue" they are successfully launched.

But when I try to run a specific test using
[UEBinaryPath]UnrealEditor-Cmd.exe "[myProjectPath]MyProject.uproject" -unattended -nopause -NullRHI -ExecCmds="Automation RunTests MyProject.Test1; quit" -TestExit="Automation Test Queue Empty" -log -log=RunTests.log -ReportExportPath="!PROJ_DIR!\TestsReports\reports\ue"
i get
ERROR: No automation tests matched 'MyProject.Test1' (I use the pretty name)

And in the frontend window in editor i can see this test

29

OK my bad, iā€™ve just forgotten to declare the module of my project in the .uproject file :person_facepalming:ā€¦ sorry

30

Does somebody know if i can Run All the test except one via CMD. Iā€™m trying to run the test for System and i have a fatal error just for one test. So I want to exclude that test.

1 Like
31

@ptherrien24

Iā€™m thrilled to share that Iā€™m currently working on a new automated testing plugin for Unreal Engine. One of the standout features of this plugin is the support for tags and priorities for tests. This means you can now run tests with specific tags or even based on a required priority, making the testing process more streamlined and efficient.

I believe this plugin can be a game-changer for many developers, especially those working on larger projects where efficient testing becomes crucial.

Iā€™d love to hear your thoughts, feedback, and any suggestions you might have. Do you think this is something youā€™d use in your projects? Are there any additional features youā€™d like to see?

Thank you in advance for your insights and feedback. Together, we can make this tool even better for the Unreal Engine community!

Warm regards,
RamĆ³n

32

@Ramon_Janousch
Hello, any news about new automated testing plugin ? Are you still working on it?

1 Like
33

Hi :slight_smile:
Weā€™ve started working on a new version of the plugin and will publish it on Github in the next few weeks.
What are your goals with the plugin?

34

Im launching unit, functional, perf test by it. im using test on single node.

i couldā€™nt set automation gauntlet system, didnā€™t find how to setup nodes.

a few issues what i have.

  1. Donā€™t have any report if client/editor crush while doing tests
  2. Canā€™t continue other tests after crush
  3. Screenshot comparing with calling from automation spec, working weird, its making screen with different resulution every time.
  4. Have to add custom method for expected errors/warnign for flacky not fixeble problems. (add expected error with Occurrences=0, AddError, right after)
  5. Didnā€™t found way to set LattentTimout for tests, incide created test class.
35

Quick note to save someone else time: the -Log= parameter accepts a path thatā€™s relative to the platform-dependent, system log path, not the current working directory or the project. For example, if I provided -Log=SomePath/MyLog.log, on macOS this would write to /Users/guy/Library/logs/MyProjectName/SomePath/MyLog.log. The path is treated as relative even if you provide an absolute path. If you want to write the logs to a specific absolute path, use the -AbsLog= command-line parameter instead of -Log=.

Counterintuitively, the -Log and -Log= parameters are handled by two different parts of the code and mean different things. The former controls whether a log window appears during runs (kind of reminds me of old school Quake 3 dedicated server logging) and is parsed in FEngineLoop::AppInit(). The latter controls whether logs are written to a file and is parsed in FGenericPlatformOutputDevices::GetAbsoluteLogFilename() (which also handles -AbsLog=). You do not need to pass -Log if you just want to log to a file and donā€™t want a log window.

Using -NullRHI with -nosplash allows things to run totally headlessly.

1 Like