[INFO DUMP: QA Automation: Functional Tests]
I did some investigating around the automated functional tests in Unreal 5.x and here are my initial findings (which might change as I learn the systems and rules around those more), which I feel are worth sharing:
- Most likely due to performance/time considerations (couldn’t find a good source for that other than the very old posts in this forum - when I find the post again, I will post a link here) the ‘Take Automation Screenshot’ blueprints can take only a single screenshot per started test run, whether the test run consists of only one individual test or multiple tests.
As a temporary workaround (thanks to this post: How to take a screenshot including UI only using Blueprint? - #26 by Danthomo87) I have created a custom Macro using the ‘Execute Console Command’ node (when making tests work in a macro first, you will thank me later, as automation is lots of repeated code).
This way I have a first step towards recording the game visuals per test step, even if those screenshots are not high quality ones (not sure if those can be used as ground truths for Screen Comparisons - something I will look into later).
Reference images below. Since it’s a macro, you need to make it variable agnostic - what I mean by that is that all the variables should be provider from the outside of the macro as once the macro is edited, it will edit itself globally (think editing ‘for loop’ that would affect whole project instead of single blueprint).
In simple terms:
- Create a macro
- Add the macro to the Blueprint
- Hook up and set variables needed by the macro
- Annoyingly, though to a degree it makes sense, any red type error message which gets triggered during the test run (regardless if set by you or in the project itself) will result in that particular Functional Test result being marked as ‘Fail’ even when the test itself succeeds (reference example below).
I am yet to find any command or option in the Unreal engine that would suppress those errors just for the Functional Tests so that the test can remain as Success despite the potentially unrelated issues.
Not really a workaround but a way to make a less invasive warnings at least for your own control that I have figured out is by simply using the ‘Log Message’ node instead, should you want to raise awareness while not impacting the test status.
- Sadly, there is no way of ordering tests you are starting via the Session Frontend. Presets might be the workaround (or some C++ way), however, I have not given it a proper look yet, but soon.
- Functional Tests have no way to emulate inputs on their own, at least I couldn’t find any answers on how to do that. The only way to go I have managed to find and simple enough to understand and replicate is by creating your own events side by side with the ones called in the UI widgets. Then you can call them in your Functional Test blueprint.
This does not subsistute UI testing, it’s just a means to move through UI without user input (i.e., for comparison screenshot gathering), until proper input can be emulated. Clicking through UI is still advised to be the manuał job for QA. 
- Functional Tests need to be attached to a level for them to be started. This poses a problem since when you switch levels the test itself is not persistent, resulting in it being marked as ‘Aborted’ and failing. I am yet to find a way to make it as a persistent presence (something in the likes of DoNotDestroy in Unity).
That is all for now. More to come as I learn via the Cropout Sample Project. 