Functional Tests trigger PIE but do not end it

Development Pipeline & Plugins
, ,
1

When using the functional test framework, if we use “FTEST_” as a prefix for a map the session front end will pick up those tests and add them to the “Standard Tests” group.

During Functional tests, PIE is triggered at the start of the test and ended when loading a new level. But at the end of the last functional tests, the editor will leave PIE running as all the other tests are run and only shuts down when the whole test run is complete.

This seems like an issue where the PIE session that is running alongside the test suite may cause issues. Is there a good way to separate functional tests into a separate run outside of the Standard Tests group, or a way to make sure it ends PIE at the end of each Functional test map?

2

Steps to Reproduce
Create a functional test map by prefacing the level name with “FTest_”

verify it shows up in session frontend under “Project.Functional Tests”, The map doesn’t need any functional test instances in it to repro this issue.

Run all “Standard Tests” and observe that after performing functional tests, the editor continues to run PIE while all later tests are run.

3

The Blueprint Functional tests were designed to run with the same PIE session to optimize time execution.

It is possible to force the map to be reloaded each time but you will need to implement your own BP functional tests runner and uses AutomationCommon::AutomationLoadMap(<map name>, true /* forceReload */)

Another way do this is to force the PIE to close at the end of each test using the OnTestEndEvent callback:

You need to register the callback during a module startup (can be done through a plugin module)

FAutomationTestFramework::Get().OnTestEndEvent.AddRaw(this, &FTestSamplesModule::OnTestEnd);and in the corresponding module shutdown

FAutomationTestFramework::Get().OnTestEndEvent.RemoveAll(this);then

void FTestSamplesModule::OnTestEnd(FAutomationTestBase* Test) { if (Test != nullptr) { UE_LOG(LogTestSamples, Verbose, TEXT("Ending test: %s"), *Test->GetTestFullName()) // maybe detect if it is a test type that needs this action // Stop PIE when leaving functional tests IAutomationControllerManagerRef AutomationController = FModuleManager::LoadModuleChecked<IAutomationControllerModule>("AutomationController").GetAutomationController(); if (GUnrealEd && !AutomationController->KeepPIEOpen()) { GUnrealEd->RequestEndPlayMap(); } } }

4

Thank you! This seems like a good option.

To improve our control over what test suites get run, I updated the list of AutomationTestFlags filterto add “Functional Tests” and FClientFunctionalTestingMapsBase to put the Map based Functional tests into that filter instead of Product.

This seems to resolve the problem because Functional tests will refresh the PIE session between tests via the calls to load the next map, and when the tests are done the pie session will automatically get shut down.