Hey Unreal folks. Unreal Engine is a thing of beauty and I feel it deserves to have beautiful automation testing as well. When I say “automation testing” I’m referring to the spectrum of unit, functional and smoke tests.
Why isn’t it on the Roadmap?
I looked through the Trello Roadmap and the only card I see that is related to automation involves the screenshot capture system. I don’t see any other cards that are related to automation testing. I can perhaps understand some reasoning for this by looking at numbers. The intersection of the subset of UE4 devs that are C++ programmers combined with the subset of C++ programmers that write unit tests could be an arguably tiny slice of a Venn diagram.
I’m not saying that is the true reason that automation isn’t on the roadmap, it’s just me trying to understand the busy world of Epic and feature priorities.
Why it should be on the Roadmap
It’s simple. Quality shipping products use automation, kind of like you folks do for Unreal Engine. You might make a thin argument that from the subset of users who will ever release a product on UE4, the majority of them would at least like to have automated tests in their shipping code. And if we can’t agree on that then I’m sure we can all agree that all shipping products would benefit from some level of automation testing.
It seems like all the pieces are in place for automation testing in UE4, but some of the glue is missing. If an experienced developer can read all of the documentation and then spend 2 hours failing to get sample code to compile then something somewhere is fundamentally broken. This is the current state of automation testing in UE4.
What are the specific issues?
**It just doesn’t work!
**Imagine Joe User sitting down at his computer for a day of work. He’s got Visual Studio 2013, UE4 binary 4.9 and is an experienced C++ developer of 20 years. He has been tasked with writing a couple unit tests. He reads through the automation documentation and decides to start with what the docs call a Simple Test using IMPLEMENT_SIMPLE_AUTOMATION_TEST. He copies and pastes the sample code into his unit test, hits compile, the compile fails.
Poor Joe! He scours the documentation again, looking for any mention of required #include files that he missed but comes up empty. After manually tracking down required include files by grep’ing through the source, he realizes that his project is missing an entire module. He starts to wonder if this is even meant to work on the binary engine, he looks through the documentation again to see if this is mentioned but finds nothing. He considers adding the missing module but is concerned about how this might affect the shipping game.
This user story highlights a couple issues.
[SIZE=2]Unclear Documentation & Requirements
[/SIZE]As a coder, the documentation doesn’t tell me how to write a full automation test. It doesn’t tell me:
- What include files are required
- What engine distribution is required (binary vs source)
- What module dependencies need to be added to my build
- A general strategy for testing, mocking, etc.
Automation Includes have Broken Dependencies
The include file for AutomationEditorCommon.h has some seriously broken dependencies. If these broken chains are meant to be filled by a missing module then it would be great to at least have a pragma somewhere to print a warning out. The sample code in the documentation calls FEngineAutomationTestUtilities::LoadMap and so AutomationEditorCommon.h is going to naturally be the first include that people try to add to get their test to build.
The workflow for adding a new automation test is disconnected from all other workflows. As a C++ developer, you add new source through the editor. Automation tests could benefit from using this workflow to generate the skeleton class with the correct includes. The workflow right now is:
- Kill VS
- Manually create a file like some kind of animal
- Rebuild UE4 VS solution
- Load VS
Links to Existing Illustrative Questions
Here is a list of all of the automation related questions that I could find:
How do you write and run C++ tests? - UE4 AnswerHubhttps://answers.unrealengine.com/questions/148157/writing-automation-tests-do-i-need-to-build-engine.html
Unit testing methods in custom gamestate class. - UE4 AnswerHub
Mocking for functional and unit tests - UE4 AnswerHub
Automation Testing On Binary Engine is Broken - UE4 AnswerHub
[SIZE=2]If nothing else, please take 15 minutes to sit down on binary 4.9 with a new C++ project. Read the docs for unit testing and copy/paste the guts of the simple test provided in the documentation. Hit compile and see what happens. Please consider beefing up automation in future releases![/SIZE]