I’ve started exploring the use of Functional Tests written in Blueprints, and I’ve encountered a “limitation” — in order to run a functional test, an instance of it must be placed directly in the level.
For a small number of functional tests, this is probably fine. However, for larger test sets and general reusability, I would expect the functional tests to be stored in separate plugins (e.g. organized by feature or type), and then dynamically loaded into the level at runtime only when needed.
In other words, the level itself wouldn’t contain any functional test actors by default, but for example, if I wanted to run tests matching PluginA.FeatureB.*, the system would load all relevant tests from PluginA that belong to FeatureB.
In our environment, I expect we’ll eventually have thousands of feature tests, covering various mechanics, organized by category.
Is this approach compatible with Unreal Engine and the “Epic Way” of doing things? Or are feature tests meant to be used in a different way?
Would it be appropriate to use a custom GauntletController implementation for loading the tests? For example, a custom controller could receive parameters specifying which tests to load, dynamically load the corresponding .uasset files into the level, label the instances for clear identification in the results, and then execute each test.
Or is there another recommended approach for managing functional tests? Or all are functional tests placed directly into level without exceptions?
Is there any advanced documentation about testing? I was looking into ShooterGame and Lyra sample project, but i lack more advanced test scenarios here.
The BP functional Tests are actors indeed so they must be part of map to be listed as tests.
The BP functional test manager however can be triggered outside of test therefor triggered by other mean. But we are not providing a specific workflow for those.
A Gauntlet Controller can be used to trigger it for example.
If your tests are meant for the Editor, the Utility Blueprint tests can be used. They don’t require a map and allow full access to the editor like normal Utility Blueprint assets.
We are suggesting grouping tests into plugins and they can be divided as you see fit, but we are not offering any mechanic to load plugins on the fly based on requested tests. The plugins need to be loaded first for the tests to be discovered.
Im not sure if im understant correctly about triggering functional test manager outside of the test, or why i wana to trigger it by Gauntlet controller, but instead of original thought of loading Functional test BPs via Gauntlet Controller, im investigating creation of plugin, that will extend FunctionalTestingModule (or something around that) to insert BP instances into specified level.
Utility Blueprint tests are not valid options, because we aim for testing game in editor and in packaged/staged version as well.
So as you suggested, we will divide tests into plugins, plugins will be part of a Development/Test configuration (prob excluded from shipping) and we will also create some custom plugin to load them according to params or something like that. Plugin for this job seems more suitable that gauntlet controller for me.
The BP functional test actors are triggered because the BP Functional Test manager is triggered by the Engine automation test framework. And this framework knows about it because the BP functional test actors have a post edit callback that store information about the tests into the Asset registry.
So if you are going to instantiate new BP functional test actor on a client build the test framework will not know about it.
The BP Functional Test manager will not be triggered.
The BP functional Test manager make it easier to run what is available in the loaded map. But if you have other mean to trigger your tests it is all good.
thanks for your answers, but im not sure if im getting it correctly. Im not seeking seeking full solution that will fit what i wrote here, im trying to understand pipeline how it was meant, and adjust it to our needs, or our needs to it. So please let me start again
1) We are expeting that we will have many of BP Functional Tests (BPFT). Then as you wrote, BPFT needs to be instantiated in map/level to be discovered by testing mechanism. So is common approach that all BPFTs are instantiated in map/level by hand of one who created them?
2) You suggested that we should split BPFT into plugins, and i totally agree with that. It would be nice to have BPFTs split/grouped by functionality, so they can be managed nicely. But it looks like to me only like “code suggar”, or is there some functionality that im missing in grouping them?
3) I watched some presentations about Functional Tests from UE Fest, where speakers mentioned, that they modified FunctionalTestBase or other class so they are not dependend on map or they are loaded automaticaly (internal Gearbox plugin called Polaris). According to that i assume that someone has similar thoughs as we have. That placing all those actors into level can be done better, or am i missing something? Maybe i trying to use BPFTs incorrectly, at its one of main reasons why im asking here.
4) If idea with loading BPFTs into level is somehow okay, im currios where it fits. Currenly im thinking of Asset Registry and SpawnActor method. But im not sure, when or where is correct place to load those tests. I noticed that FAutomationTestBase has method GetTests and RunTests. So maybe i can do Complex test, that will works as BPFTs loader? Idea behind this is, that i noticed that Complex test can provide multiple tests, and why not to load them from some plugin? And i assume that i can “refresh” available tests in AutomationWorkerModule.cpp->HandleRequestTestsMessage() via message.
Yes, the BP functional tests were meant to be instantiated in a map and that map is a test map. Other map could be nested in them for example if your test is to go over a map features. But the expectation is that the map is part of the test environment. Also a map may have several BP functional tests. They will be evaluated in sequence, triggering the On Test Start event on each actor one after the other.
The natural “group” that all automated tests in UE will get is the dot notation of their full name. And in the case of BP functional tests that includes the map content folder and map name. They can be leverage in this way. Additionally BP functional tests can be tagged and use for filtering; however that feature is still in development and the support for it is partial. Additionally test group can be formed through the project config: https://dev.epicgames.com/documentation/en\-us/unreal\-engine/configure\-automation\-tests\-in\-unreal\-engine\#testgroups ; using partial name match and tags can be leverage to formed groups.
UE is not enforcing a specific way to add tests, we try to give multiple options to users. And we offer what we use internally as tools. We don’t require the type of functional tests that you described internally.
Indeed, I did not go over what exactly needs to happen in order to collect procedural tests list. Complex test class is the way to go. GetTests is what populate the test collection for a custom test type. What is output by this method is what is going to be feed to RunTests once a test is about to run. If you implement those, your code is responsible to collect the test list and run the matching tests when the automation test work ask for it. https://dev.epicgames.com/documentation/en\-us/unreal\-engine/write\-cplusplus\-tests\-in\-unreal\-engine\#complextests
