Announcement

Collapse
No announcement yet.

Automation Testing Should be on the Roadmap

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • Automation Testing Should be on the Roadmap

    Hey Unreal folks. Unreal Engine is a thing of beauty and I feel it deserves to have beautiful automation testing as well. When I say "automation testing" I'm referring to the spectrum of unit, functional and smoke tests.

    Why isn't it on the Roadmap?
    I looked through the Trello Roadmap and the only card I see that is related to automation involves the screenshot capture system. I don't see any other cards that are related to automation testing. I can perhaps understand some reasoning for this by looking at numbers. The intersection of the subset of UE4 devs that are C++ programmers combined with the subset of C++ programmers that write unit tests could be an arguably tiny slice of a Venn diagram.

    I'm not saying that is the true reason that automation isn't on the roadmap, it's just me trying to understand the busy world of Epic and feature priorities.

    Why it should be on the Roadmap
    It's simple. Quality shipping products use automation, kind of like you folks do for Unreal Engine. You might make a thin argument that from the subset of users who will ever release a product on UE4, the majority of them would at least like to have automated tests in their shipping code. And if we can't agree on that then I'm sure we can all agree that all shipping products would benefit from some level of automation testing.

    It seems like all the pieces are in place for automation testing in UE4, but some of the glue is missing. If an experienced developer can read all of the documentation and then spend 2 hours failing to get sample code to compile then something somewhere is fundamentally broken. This is the current state of automation testing in UE4.

    What are the specific issues?
    It just doesn't work!
    Imagine Joe User sitting down at his computer for a day of work. He's got Visual Studio 2013, UE4 binary 4.9 and is an experienced C++ developer of 20 years. He has been tasked with writing a couple unit tests. He reads through the automation documentation and decides to start with what the docs call a Simple Test using IMPLEMENT_SIMPLE_AUTOMATION_TEST. He copies and pastes the sample code into his unit test, hits compile, the compile fails.

    Poor Joe! He scours the documentation again, looking for any mention of required #include files that he missed but comes up empty. After manually tracking down required include files by grep'ing through the source, he realizes that his project is missing an entire module. He starts to wonder if this is even meant to work on the binary engine, he looks through the documentation again to see if this is mentioned but finds nothing. He considers adding the missing module but is concerned about how this might affect the shipping game.

    This user story highlights a couple issues.

    Unclear Documentation & Requirements
    As a coder, the documentation doesn't tell me how to write a full automation test. It doesn't tell me:
    • What include files are required
    • What engine distribution is required (binary vs source)
    • What module dependencies need to be added to my build
    • A general strategy for testing, mocking, etc.


    Automation Includes have Broken Dependencies
    The include file for AutomationEditorCommon.h has some seriously broken dependencies. If these broken chains are meant to be filled by a missing module then it would be great to at least have a pragma somewhere to print a warning out. The sample code in the documentation calls FEngineAutomationTestUtilities::LoadMap and so AutomationEditorCommon.h is going to naturally be the first include that people try to add to get their test to build.

    Disconnected Workflow
    The workflow for adding a new automation test is disconnected from all other workflows. As a C++ developer, you add new source through the editor. Automation tests could benefit from using this workflow to generate the skeleton class with the correct includes. The workflow right now is:
    1. Kill VS
    2. Manually create a file like some kind of animal
    3. Rebuild UE4 VS solution
    4. Load VS


    Links to Existing Illustrative Questions
    Here is a list of all of the automation related questions that I could find:

    https://forums.unrealengine.com/show...Testing-in-4-6
    https://forums.unrealengine.com/show...mation+testing
    https://answers.unrealengine.com/que...n-c-tests.html
    https://answers.unrealengine.com/questions/148157/writing-automation-tests-do-i-need-to-build-engine.html

    https://answers.unrealengine.com/que...ate-class.html
    https://answers.unrealengine.com/que...nit-tests.html
    https://answers.unrealengine.com/que...is-broken.html

    Closing
    If nothing else, please take 15 minutes to sit down on binary 4.9 with a new C++ project. Read the docs for unit testing and copy/paste the guts of the simple test provided in the documentation. Hit compile and see what happens. Please consider beefing up automation in future releases!
    Last edited by hyperdr1ve; 09-17-2015, 12:46 PM. Reason: add link for unit testing docs

  • replied
    Originally posted by alpaka View Post
    ... Has anyone experimented with writing simulation code in unmanaged parts of the game (POCOs) and using the VS test toolkit?
    To answer my own earlier question: If you are fine with (or even prefer) foregoing all the UE4 C++ extensions by using POCOs, it is quite possible to create a library project, which you can unit test with the regular VS test system, then link it into your project. This allows you to use unmanaged code both in the test explorer and in your game. If you need to unit test UObject code, need reflection, or are using managed code otherwise, I'm afraid you're stuck with the built-in testing framework.

    There are basically two options: a static library (.lib) or dynamic library (.dll). I couldn't easily figure out how to use the hot reload feature for non-module DLLs, or how to build them as stand-alone DLLs (can't link to the generated DLLs from the build tool, it seems), so I went with the static library. There is an introduction to linking them using UBT here: https://wiki.unrealengine.com/Linkin...e_Build_System, but you should update it to 4.16 using ReadOnlyTargetRules instead of TargetInfo.

    I would also like to note the following workaround for using the automation system a little more efficiently (quoting myself on the Answer hub):
    Just in case anyone is looking for a sorta workaround to this: I played around a bit with hot reloading modules that contain tests and while you cannot change an existing test, adding a new one seems to work fine (recompile module from the editor - or, if the test is in your main module, it also works when building from VS - then hit Refresh Tests in the session frontend).

    So when you want to change an existing test to make it pass or whatever, you can change its name from Test to something else (I use Test2, Test3, etc.). Unfortunately, the old test class stays around as a ghost until you restart the editor, so it messes up the "Run All tests in a category" functionality. Note that you can have multiple test classes in a single source file so genuinely adding new tests is not a big issue.

    While this is obviously not a perfect solution, it's somewhat workable and is at least better than having to restart the editor all the time. Once in a while, you can clean up all the numbers and just restart the editor.

    Leave a comment:


  • replied
    I kind of think that those people who seriously use this sort of thing would probably pay for a license.

    I think it is a bit sneaky as well and has a bit of wisdom. I've being in work places where devs (having no budget on the own) just start using these sorts of tools on the sly. Along comes some compliance person (or busy body, whatever) and kind of forces the issue (which if the devs went through the "proper process" would just being blown off - sometimes it is easier for the money person just to say yes).

    Leave a comment:


  • replied
    Originally posted by funkinessfactor View Post
    I just followed this...

    Their blog post about using PVS Studio for free.

    https://www.viva64.com/en/b/0457/
    Now that's an interesting license... they're basically asking us to write some kind of automation script to add/remove those comments

    Leave a comment:


  • replied
    I just followed this...

    Their blog post about using PVS Studio for free.

    https://www.viva64.com/en/b/0457/

    Leave a comment:


  • replied
    Originally posted by funkinessfactor View Post
    and us single independent developers working on individual independent projects get it for free.
    Where/how did you get PVS studio for free?

    Leave a comment:


  • replied
    Obviously there are many AAA studios that use Unreal. Maybe Epic could reach out and ask what they do on this front and provide suggestions to the great unwashed. Maybe even a guest blog post. Hopefully they don't come back saying they employee testers @ $2 per hour

    For myself, I have heaps of checks in my code and a few check(TestWorldConsitency()); type things. I have a few "test maps" that automate stuff on Tick(), but it is really unrefined. Please don't flame me to say but you are not doing real automated tests or unit tests.

    I've also got some millage from using the PVS-Studio which you can see that better support is on the roadmap for 4.17. I'm sure I have a relatively small project, but it only took me a day to get everything passed (or marked as a false positive) and us single independent developers working on individual independent projects get it for free.

    Leave a comment:


  • replied
    +1

    Really a sad state of affairs as far as unit testing is concerned. It would seem to me as if unit testing would be quite useful for many of the more math-heavy parts of code that one develops (such as AI or simulation code).

    Having to jump through absurd hoops to even get a single test to run (took me more than an hour, especially because all docs you can easily find are outdated and you have to go with your gut/experiment quite a bit), let alone lacking support for hot-plugging, etc. is so painful that it makes skipping unit tests seem a very attractive option. Has anyone experimented with writing simulation code in unmanaged parts of the game (POCOs) and using the VS test toolkit?

    Leave a comment:


  • replied
    Bump ! Any news about this "issue" ?

    Unit Testing is a must in Software Engineering... and Unit tests deserve some Love
    Last edited by Cenovis; 06-14-2017, 04:26 AM.

    Leave a comment:


  • replied
    Are there any more updates on this?

    Would really be a massive help to have a proper documentation Automation System and proper unit tests.

    Leave a comment:


  • replied
    Hi guys,

    As i´m coming from a enterprise java background and love to write my game in spare time, testing with UE4 is really hard to achieve to be honest.
    Is there any update on that topic? Maybe it is a reachable goal to have some proper testing framework like junit + mockito from java in the UE4 world, nobody says it needs to be 100%, but usually a proper optimized workflow would help.

    I´m not that experience in c++ right now, but i think if we work all together, this is a possible thing!

    Leave a comment:


  • replied
    Hi, I've taken your feedback here and escalated it to the appropriate teams for review. Thanks for giving us your feedback and I'm sorry for any headaches this has caused

    Leave a comment:


  • replied
    Yeah, not having testing seems like a big oversight. This is 2016. A test suite should be a given.

    Leave a comment:


  • replied
    Oof. I'm trying to assess some of the "lesser talked about features" in UE4 and this one stood out as a biggie. We tend to write turn-based strategy games, so a lot of game logic tends to be written in testable form. The front end all looks great, and the tools seem nice, but man... lacking hot-reload is a huge time suck. If anyone has any work-arounds via external tools or other strategies to writing unit/integration tests in UE4 I'd love to hear it.

    Leave a comment:


  • replied
    I strongly agree that we need to be able to hot-reload the tests. TDD and Automated testing are how modern software is made.
    Please make automation a first-class feature for game developers.

    Leave a comment:

Working...
X