Gauntlet Namespaces CLI param does nothing?

Development Programming & Scripting
,
1

I have some spec tests that I’d like to invoke using the RunUnreal Gauntlet command. RunUnreal.cs lists a Namespaces param that should allow me to specify where to search for my tests, but not only are my tests not being found, the Namespaces param is being ignored entirely.

And indeed, if you go through the RunUnreal.cs file you can see in the Execute() method that the Namespaces context value gets hardcoded to ContextOptions.Namespaces = "Gauntlet.UnrealTest,UnrealGame,UnrealEditor";

How is this supposed to work? And if I can’t specify a namespace, where do I put my tests so Gauntlet knows about them?

Full Logs below:

Starting AutomationTool...
Parsing command line: -nop4 -NoCompile RunUnreal -project=E:\Dev\main_filtered/Game/MyGame.uproject -build=Editor -test=MyTest -Namespaces=MyNamespace -VeryVerbose
Initializing script modules...
Total script module initialization time: 0.32 s.
Executing commands...
Will use Editor for all roles
Added device Win64:default to pool
Will use Editor for role Unknown
Mapped Role Unknown to RoleContext Win64 Development Unknown
Will use Editor for role Editor
Mapped Role Editor to RoleContext Win64 Development Editor
Will use Editor for role EditorGame
Mapped Role EditorGame to RoleContext Win64 Development EditorGame
Will use Editor for role EditorServer
Mapped Role EditorServer to RoleContext Win64 Development EditorServer
Will use Editor for role Client
Mapped Role Client to RoleContext Win64 Development EditorGame
Will use Editor for role Server
Mapped Role Server to RoleContext Win64 Development EditorServer
Will use Editor for role CookedEditor
Mapped Role CookedEditor to RoleContext Win64 Development CookedEditor
Will search MyTest Gauntlet.UnrealTest.MyTest UnrealGame.MyTest UnrealEditor.MyTest for test MyTest
Possible candidates for MyTest: UnrealGame.ElementalDemoTest UnrealGame.FortGPUTestbedPerfTest UnrealGame.DefaultTest EpicGame.EpicGameTestNode`1[TConfigClass] UnrealEditor.EditorTests UE.EditorAutomation UE.TargetAutomation UE.AutomationNodeBase`1[TConfigClass] UE.BootTest UE.Edi
torBootTest UE.TargetBootTest UE.TestEnsuresAreCaught UE.TestChecksAreCaught UE.TestFatalErrorsAreCaught UE.TestCrashesAreCaught UE.ErrorTestBase UE.NetworkingNodeBase`1[TConfigClass] UE.PLMTest UE.NetworkAutomation.DedicatedServer UE.NetworkAutomation.ListenServer Gauntlet.BaseTe
st Gauntlet.ITestNode Gauntlet.PGONode`1[TConfigClass] Gauntlet.BasicReplayPGONode Gauntlet.UnrealTestNode`1[TConfigClass] Gauntlet.UnrealTest.BootTest Gauntlet.UnrealTest.EngineTest Gauntlet.UnrealTest.EngineTestBase`1[TConfigClass] Gauntlet.SelfTest.TestTargetDeviceLinux Gauntle
t.SelfTest.TestTargetDeviceMac Gauntlet.SelfTest.TestUnrealInstallThenRunMac Gauntlet.SelfTest.TestTargetDeviceWindows Gauntlet.SelfTest.TestTargetDevice Gauntlet.SelfTest.CreateGif Gauntlet.SelfTest.OrderOfOpsTest Gauntlet.SelfTest.TestAutoParam Gauntlet.SelfTest.TestParamParsing
 Gauntlet.SelfTest.BaseTestNode Gauntlet.SelfTest.Blank Gauntlet.SelfTest.TestUnrealLogParserBase Gauntlet.SelfTest.LogParserTestGauntletExitSuccess Gauntlet.SelfTest.LogParserTestEnsure Gauntlet.SelfTest.LogParserTestAssert Gauntlet.SelfTest.LogParserTestFatalError Gauntlet.SelfT
est.LogParserTestException Gauntlet.SelfTest.LogParserTestRequestExit Gauntlet.SelfTest.LogParserTestChannels Gauntlet.SelfTest.LogParserTestWarnings Gauntlet.SelfTest.LogParserTestErrors Gauntlet.SelfTest.LogParserPerfSummary Gauntlet.SelfTest.TestOrionShortSoloGame Gauntlet.Self
Test.TestUnrealBase Gauntlet.SelfTest.TestUnrealBuildParsing Gauntlet.SelfTest.TestUnrealBuildSource Gauntlet.SelfTest.TestUnrealInstallAndRunBase Gauntlet.SelfTest.TestUnrealOptions Gauntlet.SelfTest.TestUnrealSession Gauntlet.SelfTest.TestTargetDevicePS4 Gauntlet.SelfTest.TestUn
realInstallThenRunPS4 Gauntlet.SelfTest.TestTargetDevicePS5 Gauntlet.SelfTest.TestUnrealInstallThenRunPS5 Gauntlet.SelfTest.TestTargetDeviceWinGDK Gauntlet.SelfTest.TestTargetDeviceXB1 Gauntlet.SelfTest.TestUnrealInstallThenRunXB1 Gauntlet.SelfTest.TestTargetDeviceXboxCommon`1[TTa
rgetDevice] Gauntlet.SelfTest.TestUnrealInstallThenRunXboxCommon`1[TTargetDevice] Gauntlet.SelfTest.TestTargetDeviceXboxOneGDK Gauntlet.SelfTest.TestUnrealInstallThenRunXboxOneGDK Gauntlet.SelfTest.TestTargetDeviceXSX Gauntlet.SelfTest.TestUnrealInstallThenRunXSX EditorTest.BootTest EditorTest.EditorTestNode LowLevelTests.LowLevelTests
Unable to find type MyTest in assemblies. Namespaces=Gauntlet.UnrealTest, UnrealGame, UnrealEditor.
(see E:\Dev\main_filtered\Engine\Programs\AutomationTool\Saved\Logs\Log.txt for full exception trace)
AutomationTool executed for 0h 0m 0s
AutomationTool exiting with ExitCode=1 (Error_Unknown)

Process finished with exit code 1.
1 Like
2

Finally discovered a solution, as per this thread: https://forums.unrealengine.com/t/gauntlet-test-unable-to-find-test-in-assemblies-5-4/1920436

Since the project is outside of the engine, it requires adding -scriptdir= (the folder containing the .csproj)
For example: -scriptdir=D:\Unreal_Projects\Lyra\Build\Scripts

1 Like
3

I’ve encountered this exact problem, moreover, the Lyra example project doesn’t locate/execute its’ LyraTest.BootTest, it falls back to Gauntlet.UnrealTest.BootTest.

Following the docs:
" Run the Boot Test Example"
Running Gauntlet Tests In Unreal Engine

Attaching the debugger with a breakpoint under Gauntlet.Utils.cs Utils.TestConstructor.GetTypeForTest() shows that LyraTest.BootTest won’t be found, because it’s not in assemblies: var SearchAssemblies = AppDomain.CurrentDomain.GetAssemblies();

This is really confusing!

1 Like