Blueprint Functional Tests - Force Pass/Fail via "Finish Test"

Development Pipeline & Plugins
, ,
1

I’m running Blueprint Functional Tests, and I have a pair of related questions.

Is there any way to tell, definitely, what has caused a test to be marked as a failure? I attached a screenshot of me running a test via the Automation tab of the Session Frontend, and I only get a Pass/Fail message - no explanation as to why the test was marked as a fail. I can export the results via the menu in the lower right, and it will give me the log messages but not tell me what failed the test. Checking the editor logs also doesn’t tell me what specific thing failed the test.

My best guess is that any error output to the logs is failing the test. If this is correct, what is the best way to disable this? Ideally, when we write tests, we want the test code to be 100% in control of the test passing or failing.

Thank you in advance.

2

Steps to Reproduce

  1. Write Blueprint Functional Test
    1. Include a node “Finish Test” and mark as Success
  2. Add it to a level
  3. Use any method of running the tests

Expected Behavior: If Test Result = Succeeded, test should be marked as a pass.

Actual Behavior: Test is being marked as a failure.

3

Selecting the failing test row in the automation tab should give the list of collected events during the test run.

Also BP functional tests need to call Finish test at some point or it will be considered a failure through a timeout.

All BP functional tests have a timeout, it is configurable on a test basis by selecting the related actor in the map and searching from timeout in details panel.

To come back on the Finish Test node call, one can attach a message to the call and that message is going to be attached to the test events.

4

To be clear, as mentioned in the original post, I do have a node called Finish Test and when I step through with the debugger, the node is being reached. Further, the message attached to the Finish Test node is showing up in the log. The Finish Test node is marked as a success, and the test is STILL being marked as a failure by the Session Frontend/Automation tab. (and also when I run the test via command line tools and/or Gauntlet).

How can I force it such that I can control whether the test is marked as a failure or a success? I would expect calling Finish Test with Succeeded to do this, but it does not.

5

Also, is there any way to definitely tell exactly what marked a test as a failure?

I see log lines like this:

[2025.10.15-17.29.50:993][257]LogAutomationController: Error: Test Completed. Result={Fail} Name={FTG_DrawAimFireStowTest} Path={Project.Functional Tests.Maps.TestLevels.FunctionalTests.TM_FunctionalTests.FTG_DrawAimFireStowTest}

Which isn’t clear as to what Error caused the failure.

6

As a side note, is there any way to force test running order? For example, if I have one test that I always want to run first, can I do that?

7

Is there any log error that were raised during the test execution?

If yes, the test will marked as failing by default.

The behavior can be overridden through the BP functional test actor or through the project config to apply that behavior systematically.

It is also possible to add expected error messages through a BP functional test node.

8

You should see before that line all the collected events during the test. Which should include the errors and messages that were raised during the test.

9

How do I do this? “through the project config”

I’ve tried:

[/Script/AutomationController.AutomationControllerSettings]

bTreatLogErrorsAsTestErrors=False

bTreatLogWarningsAsTestErrors=False

[/Script/FunctionalTesting.FunctionalTestingManager]

LogErrorHandling=Ignore

[Automation]

bTreatLogErrorsAsTestErrors=False

bTreatLogWarningsAsTestErrors=False

[Gauntlet]

bFailOnLogWarnings=False

bFailOnLogErrors=False

My test also has expected error messages with Count 0 and string match “” which should expect all errors and not fail the test over them.

Despite having both of these things in place my test is still failing.

LogMelee: NPC has no valid collision component! [log]

PIE: NPC has no valid collision component! [log]

LogCore: Invalid input P=0.000000 Y=0.000000 R=-nan to FRotator::Quaternion - generated NaN output: X=nan Y=nan Z=nan W=-nan [log]

10

And every single collected event is marking the test as a failure, and every single event must be addressed/fixed before the test will be marked as a pass? (assuming I also mark the test as a pass)

11

The project config setting in DefaultEngine.ini is:

[/Script/AutomationController.AutomationControllerSettings]
bSuppressLogWarnings=true
bSuppressLogErrors=true
bElevateLogWarningsToErrors=false

If you are using AddExpectedLogMessage/AddExpectedLogError node make sure it is called before the log messages are triggered.

Also from the BP functional test actor the settings can be set from here

[Image Removed]