Automation Testing Should be on the Roadmap

General Feedback & Requests
1

Hey Unreal folks. Unreal Engine is a thing of beauty and I feel it deserves to have beautiful automation testing as well. When I say “automation testing” I’m referring to the spectrum of unit, functional and smoke tests.

Why isn’t it on the Roadmap?
I looked through the Trello Roadmap and the only card I see that is related to automation involves the screenshot capture system. I don’t see any other cards that are related to automation testing. I can perhaps understand some reasoning for this by looking at numbers. The intersection of the subset of UE4 devs that are C++ programmers combined with the subset of C++ programmers that write unit tests could be an arguably tiny slice of a Venn diagram.

I’m not saying that is the true reason that automation isn’t on the roadmap, it’s just me trying to understand the busy world of Epic and feature priorities.

Why it should be on the Roadmap
It’s simple. Quality shipping products use automation, kind of like you folks do for Unreal Engine. You might make a thin argument that from the subset of users who will ever release a product on UE4, the majority of them would at least like to have automated tests in their shipping code. And if we can’t agree on that then I’m sure we can all agree that all shipping products would benefit from some level of automation testing.

It seems like all the pieces are in place for automation testing in UE4, but some of the glue is missing. If an experienced developer can read all of the documentation and then spend 2 hours failing to get sample code to compile then something somewhere is fundamentally broken. This is the current state of automation testing in UE4.

What are the specific issues?
**It just doesn’t work!
**Imagine User sitting down at his computer for a day of work. He’s got Visual 2013, UE4 binary 4.9 and is an experienced C++ developer of 20 years. He has been tasked with writing a couple unit tests. He reads through the automation documentation and decides to start with what the docs call a Simple Test using IMPLEMENT_SIMPLE_AUTOMATION_TEST. He copies and pastes the sample code into his unit test, hits compile, the compile fails.

Poor ! He scours the documentation again, looking for any mention of required #include files that he missed but comes up empty. After manually tracking down required include files by grep’ing through the source, he realizes that his project is missing an entire module. He starts to wonder if this is even meant to work on the binary engine, he looks through the documentation again to see if this is mentioned but finds nothing. He considers adding the missing module but is concerned about how this might affect the shipping game.

This user story highlights a couple issues.

[SIZE=2]Unclear Documentation & Requirements
[/SIZE]As a coder, the documentation doesn’t tell me how to write a full automation test. It doesn’t tell me:

  • What include files are required
  • What engine distribution is required (binary vs source)
  • What module dependencies need to be added to my build
  • A general strategy for testing, mocking, etc.

Automation Includes have Broken Dependencies
The include file for AutomationEditorCommon.h has some seriously broken dependencies. If these broken chains are meant to be filled by a missing module then it would be great to at least have a pragma somewhere to print a warning out. The sample code in the documentation calls FEngineAutomationTestUtilities::LoadMap and so AutomationEditorCommon.h is going to naturally be the first include that people try to add to get their test to build.

Disconnected Workflow
The workflow for adding a new automation test is disconnected from all other workflows. As a C++ developer, you add new source through the editor. Automation tests could benefit from using this workflow to generate the skeleton class with the correct includes. The workflow right now is:

  1. Kill VS
  2. Manually create a file like some kind of animal
  3. Rebuild UE4 VS solution
  4. Load VS

Links to Existing Illustrative Questions
Here is a list of all of the automation related questions that I could find:

https://answers.unrealengine.com/questions/193799/how-do-you-write-and-run-c-tests.html[https://answers.unrealengine.com/questions/148157/writing-automation-tests-do-i-need-to-build-engine.html](https://answers.unrealengine.com/questions/148157/writing-automation-tests-do-i-need-to-build-engine.html)

Closing
[SIZE=2]If nothing else, please take 15 minutes to sit down on binary 4.9 with a new C++ project. Read the docs for unit testing and copy/paste the guts of the simple test provided in the documentation. Hit compile and see what happens. Please consider beefing up automation in future releases![/SIZE]

2

Hi Hyperdrive,

We already have an automation testing system that allows users to not only use tests that have already been pre-designed, but implement their own tests as well. Check out Unreal FrontEnd. This can be accessed in editor or as a standalone application and should cover what you are looking for here. More information on FrontEnd can be found here:

3

[= ;381102]
Hi Hyperdrive,

We already have an automation testing system that allows users to not only use tests that have already been pre-designed, but implement their own tests as well. Check out Unreal FrontEnd. This can be accessed in editor or as a standalone application and should cover what you are looking for here. More information on FrontEnd can be found here:

[/]

I wasn’t clear enough. Everything I said above was a direct criticism of the existing UnrealFrontend system. All of the documentation, existing examples and functionality that exists for automation tests in UnrealFrontend is designed for people that are writing automation tests for the actual engine. For someone working on the engine, automation tests are something of a 1st class feature.

For someone that wants to write automation tests for an actual game, the system is more of a 3rd class experience. Outdated documentation, unclear support and everything else I mentioned above. What I’m asking for is that Game-Centric Automation Testing be on the Roadmap as a 1st class feature.

1 Like
4

I agree with 's suggestion. I hope testing workflow for UE project will be better in the future.

As someone who is new to game development, I wanted to isolate and test my logic classes with 3rd party library like catch.
I came across this thread by Wolfjagger, which showed how to do it . It is rather hackish as it involves modification of UnrealBuildTool.
Unfortunately I also use the binary installation of UE4.

1 Like
5

Yepp, I agree completely. Is it still the case that you can’t hot-reload unit tests? If so, that means that every time you change the test, you have to close the editor and relaunch it, then go to the session frontend, untick the default tests, and then tick your own. That takes a long time! And no word on mocking.

1 Like
6

[=;451601]
Yepp, I agree completely. Is it still the case that you can’t hot-reload unit tests? If so, that means that every time you change the test, you have to close the editor and relaunch it, then go to the session frontend, untick the default tests, and then tick your own. That takes a long time! And no word on mocking.
[/]

It is still the case that you can’t hot-reload unit tests. No mocking, but that’s another story I think with the whole uproperty system.

7

[=;452038]
It is still the case that you can’t hot-reload unit tests. No mocking, but that’s another story I think with the whole uproperty system.
[/]

I have submitted a bug report on the lack of hot-reload here. This would at least be something.

EDIT: Whoops, still waiting for approval! Content (listed as being for 4.11 preview):

[]

I have also tested this in 4.9.

Repro:

  1. Create C++ project
  2. Create a simple automation test as per this gist.
  3. Run the test via the in-editor Frontend and observe the console output.
  4. Change the UE_LOG output to some other phrase.
  5. Hit the ‘compile’ button in-editor.
  6. Run the test again and notice that it has the same output as before the hot-reload.

You can try this with any sort of change if for some reason console outputs aren’t hot-reloaded by default. The results are the same.

This is a huge problem for the speed and practicality of conducting regular unit tests. At the moment, I have to the editor every single time I change a test. It takes too long to be practical or useful.

Thanks!
[/]

8

Minor update: this Answerhub post regarding tests not hot-reloading was accepted, and a report has been entered (UE-25350). It’s not much towards making tests usable on the ‘project’ level (vs engine level), but it’s a good start! Would love to see this fix in 4.11 or 4.12.

1 Like
9

+1 to this thread. Even hot-reload alone would save an enormous amount of time.

1 Like
10

I’d really like some additional documentation, examples, and possibly a way to mock for our unit tests! If we had some way to tie into Google Test: GitHub - google/googletest: GoogleTest - Google Testing and Mocking Framework That would really accelerate our workflows for creating tests and more stable code!

1 Like
11

This is something I’d like to see too. I prefer the TDD style of development and I’d love to be able to do more of that in UE.

1 Like
12

Bad news, friends - automation tests not hot-reloading is not a bug. Still hoping for a change so we can actually use this framework for quick and iterative project-level testing.

13

I strongly agree that we need to be able to hot-reload the tests. TDD and Automated testing are how modern software is made.
Please make automation a first-class feature for game developers.

2 Likes
14

Oof. I’m trying to assess some of the “lesser talked about features” in UE4 and this one stood out as a biggie. We tend to write turn-based strategy games, so a lot of game logic tends to be written in testable form. The front end all looks great, and the tools seem nice, but man… lacking hot-reload is a huge time suck. If anyone has any work-arounds via external tools or other strategies to writing unit/integration tests in UE4 I’d love to hear it.

2 Likes
15

Yeah, not having testing seems like a big oversight. This is 2016. A test suite should be a given.

2 Likes
16

Hi, I’ve taken your feedback here and escalated it to the appropriate teams for review. Thanks for giving us your feedback and I’m sorry for any headaches this has caused

1 Like
17

Hi guys,

As i´m coming from a enterprise java background and love to write my game in spare time, testing with UE4 is really hard to achieve to be honest.
Is there any update on that topic? Maybe it is a reachable goal to have some proper testing framework like junit + mockito from java in the UE4 world, nobody says it needs to be 100%, but usually a proper optimized workflow would help.

I´m not that experience in c++ right now, but i think if we work all together, this is a possible thing!

1 Like
18

Are there any more updates on this?

Would really be a massive help to have a proper documentation Automation System and proper unit tests.

1 Like
19

Bump ! Any news about this “issue” ?

Unit Testing is a must in Software Engineering… and Unit tests deserve some Love :smiley:

1 Like
20

+1

Really a sad state of affairs as far as unit testing is concerned. It would seem to me as if unit testing would be quite useful for many of the more math-heavy parts of code that one develops (such as AI or simulation code).

Having to jump through absurd hoops to even get a single test to run (took me more than an hour, especially because all docs you can easily find are outdated and you have to go with your gut/experiment quite a bit), let alone lacking support for hot-plugging, etc. is so painful that it makes skipping unit tests seem a very attractive option. Has anyone experimented with writing simulation code in unmanaged parts of the game (POCOs) and using the VS test toolkit?

1 Like