Functional Test Errors on Warnings and Errors in UE_LOG's from other actors (UE4-27-2)

[againstao]

So I came across this weird bug with functional testing. My tests appears to error out on ANY UE_LOG warnings or errors despite what settings I have in the functional test. I noticed it whenever I left my DefaultPawnClass as my PlayerCharacter in the maps and modes project settings whenever running the tests. If I set the DefaultPawnClass to none the test executes as normal, however when switching it back to the playercharacter any UE_LOG marked with warning or error will cause it to fail, as it is failing on my default pawn throwing warnings I made within the actual playercharacter itself. It is only on the first one that is ran in that world while the default pawn is spawning, and not anywhere else. If a run one test alone it always fails. If I run multiple everything after the first one tends to pass. How can I make my functional tests only care about the test conditions its supposed to verify?

Everything functions normal:

After switching to the default pawn to the Player Character:

My actual test in the functional tests passes:

These settings are ignored although documentation makes you think this should ignore the errors or warnings.

[againstao]

Edit: The default pawn thing proved to be a red herring. Even though I got it to pass/fail consistently that night, once I picked the project back up at another day it’s doing it with the default pawn being none. It is still however ignoring the settings set in the functional test and erroring out on warnings from the other actors beginplay function within the actual test editor menu itself.

[Ramon_Janousch]

The FAutomationControllerManager stores the number of warnings and errors in a FAutomatedTestResult when the test is executed.
So far I have not found a way to prevent this. The only solution at the moment is to somehow fix the warnings and errors or not throw them during automated tests.

[againstao]

Thank you! It is still weird to me how it’s a least not marking them as warnings but instead treating them all as errors. I’d be fine with it if they marked them correctly in the functional tests, but all those red items are actually warnings. Warnings from within the functional test itself seem to be marked correctly and are highlighted yellow, but not from other actors.