Download

FAQ: Developer Tools for UE4

Nov 20, 2020.Knowledge

What is the intent/future of Gauntlet as a plugin? What role does it fill for automated testing?

  • For clarity - Gauntlet has two parts. A UAT (Unreal Automation Tool) based framework written in C#, and a helper plugin for Unreal that provides convenience functions for managing and responding to state changes that can occur during testing.
  • It’s important to understand that Gauntlet is not a testing framework per-se (there are already several options in Unreal for this that cover different needs and preferences). Gauntlet is “a way to run things” and provides the following -
    • Low level interfaces around platforms, devices, builds, and sessions.
      • E.g. you can (and we do) write a Gauntlet script that says "take a build at this path, copy it to the N devices in this list, then run them with these command-lines.
    • High-level interfaces that can define “sessions”
      • E.g. “Run a server and 100 clients using this build, this mix of platforms, and these arguments”
  • In both cases Gauntlet will manage discovery/reservation of devices, installation of builds, launching processes, monitoring the health of those processes, and retrieval of artifacts (logfiles, screenshots, crashdumps). We use Gauntlet both for automation on the farm and day to day developer and QA workflows. E.g. devs have a Gauntlet script that can pull any Fortnite build from the network, deploy it on their kit, and launch it with their locally built executables.

Specifically, what are the recommended ways to use FunctionalTesting, Gauntlet, Automation Spec, Automation Driver holistically?

  • See Gauntlet description above.
  • Functional testing is where you author your tests in Blueprint using the provided classes. You could have multiple tests per map, one test per map, or some mixture. The “EngineTest” project is almost entirely built using Functional Testing
  • Automation Spec works best for testing code that can be executed outside the context of a game (e.g. no world, pawns, controllers).
  • Latent testing (and automation spec with latent commands) are useful for writing texts that do need the context of a game
  • In Fortnite we have a lot of automation that is triggered purely based on command lines. E.g. launching a replay with profiling mode.
  • We also use the native Gauntlet plugin to implement certain tests that are performed by executing console commands
  • The Launcher team created the Automation Driver for their needs but it’s had very little use in the context of a game. We do not have a reliable way to playback recorded input.

Is there any way to extend private engine plugins (not modules) like PerforceSourceControl and Gauntlet?

  • You can create a plugin that depends on other plugins. Once you have that dependency you can derive from any public class in that plugin. Specific to these examples I do not know if there is a way to force the engine to instantiate MyPerforceSourceControl in lieu of PerforceSourceControl, but with Gauntlet the game itself can take responsibility for initializing Gauntlet with whatever class it likes.

How could we add a new p4 client command to be run by the PerforceSourceControl module?

  • Providing the classes are public, you would create MyPerforceSourceControl that depends on PerforceSourceControl and derives from the desired base class. You would then use FModuleManager::LoadModulePtr(“MyPerforceSourceControl”) to get a pointer to your module where you could call your desired function. As mentioned above there is a difference between you calling your derived plugin and the engine using it in lieu of the default. The latter is going to be very case by case.

Are there any recommendations on how to use Gauntlet?

Anything special needed for implementing a multi-user session? Advice / roadblocks?

  • Here is a video going over the Collaborative-Viewer Template and another for Multi-User Collaboration
  • Collaborative user is designed to be more purpose-built and easy to deploy for non-users. Multi-user is for collaboration within the Editor.
  • (You’ll need to be on source control to make the Multi-User Collaboration work).

Does the Collaborative-Viewer Template provide constraints?

It seems the .modules file for a plugin must have the Unreal build number embedded in it. Is this true? Does this mean one must offer a new download for each minor version point-release of Unreal? How many releases per year would that be?

What does UE provide out of the box with support for automated testing? Does the engine come with unit tests?

  • We have the Gauntlet Automation Framework out of the box for sessions of Unreal (think driving gameplay tests that require multiple components), as well as unit tests available through our Automation System for most subsystems.

Gauntlet Docs:

Automation System Docs and Examples:

  • If your team has access to our Perforce streams, then the EngineTest project is also a helpful resource.

Examples of some unit tests are in [UERoot]\Engine\Source\Runtime\Engine\Private\Tests and there’s also [UERoot]\Engine\Source\Runtime\Core\Private\Tests for things like Math/Strings/Serialization. But yes, the engine does have unit tests out of the box, and testing/automation frameworks that can handle a lot of different scenarios.

UE VERSION

[quote=“Rudy_Triplett, post:1, topic:265034”]
In both cases Gauntlet will manage discovery/reservation of devices
[/quote]Hi Rudy, the “official” Gauntlet documentation does not give any details about this feature or how to use it. Is there anywhere else we can get this information?

So far I can get gauntlet to discover “default” console devkits with the proper SDKs installed, and it will run up to 12 local instances (this seems to be hardcoded) on the Windows machine UAT is being run from. But is there a way to get it to do remote installation & execution on other Win64 clients the way it can for consoles?

There are references in the Gauntlet source code for http calls to a device pool service for “reserving” clients (arguments like -deviceurl and -devicepool), but there is no documentation for this, nor have I found any code for running an instance of that service myself.