So I came across this weird bug with functional testing. My tests appears to error out on ANY UE_LOG warnings or errors despite what settings I have in the functional test. I noticed it whenever I left my DefaultPawnClass as my PlayerCharacter in the maps and modes project settings whenever running the tests. If I set the DefaultPawnClass to none the test executes as normal, however when switching it back to the playercharacter any UE_LOG marked with warning or error will cause it to fail, as it is failing on my default pawn throwing warnings I made within the actual playercharacter itself. It is only on the first one that is ran in that world while the default pawn is spawning, and not anywhere else. If a run one test alone it always fails. If I run multiple everything after the first one tends to pass. How can I make my functional tests only care about the test conditions its supposed to verify?
Edit: The default pawn thing proved to be a red herring. Even though I got it to pass/fail consistently that night, once I picked the project back up at another day it’s doing it with the default pawn being none. It is still however ignoring the settings set in the functional test and erroring out on warnings from the other actors beginplay function within the actual test editor menu itself.
The FAutomationControllerManager stores the number of warnings and errors in a FAutomatedTestResult when the test is executed.
So far I have not found a way to prevent this. The only solution at the moment is to somehow fix the warnings and errors or not throw them during automated tests.
Thank you! It is still weird to me how it’s a least not marking them as warnings but instead treating them all as errors. I’d be fine with it if they marked them correctly in the functional tests, but all those red items are actually warnings. Warnings from within the functional test itself seem to be marked correctly and are highlighted yellow, but not from other actors.
Sorry for necro’ing an old thread, but this is the top post on this topic when searching so I want to post my solution here. To stop logged warnings appearing as test errors across the project add the following to DefaultEngine.ini:
If you still want logged warnings to appear as warnings leave off that last line, and if you want to suppress logged errors add bSuppressLogErrors=true to the end. This works for me in 5.4 right now but I’m pretty sure this worked since 5.3 at least. Earlier versions have different names for the config settings, I think it was bTreatLogWarningsAsTestErrors?
