Announcement

Collapse
No announcement yet.

Automation Testing Should be on the Roadmap

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

    [FEATURE REQUEST] Automation Testing Should be on the Roadmap

    Hey Unreal folks. Unreal Engine is a thing of beauty and I feel it deserves to have beautiful automation testing as well. When I say "automation testing" I'm referring to the spectrum of unit, functional and smoke tests.

    Why isn't it on the Roadmap?
    I looked through the Trello Roadmap and the only card I see that is related to automation involves the screenshot capture system. I don't see any other cards that are related to automation testing. I can perhaps understand some reasoning for this by looking at numbers. The intersection of the subset of UE4 devs that are C++ programmers combined with the subset of C++ programmers that write unit tests could be an arguably tiny slice of a Venn diagram.

    I'm not saying that is the true reason that automation isn't on the roadmap, it's just me trying to understand the busy world of Epic and feature priorities.

    Why it should be on the Roadmap
    It's simple. Quality shipping products use automation, kind of like you folks do for Unreal Engine. You might make a thin argument that from the subset of users who will ever release a product on UE4, the majority of them would at least like to have automated tests in their shipping code. And if we can't agree on that then I'm sure we can all agree that all shipping products would benefit from some level of automation testing.

    It seems like all the pieces are in place for automation testing in UE4, but some of the glue is missing. If an experienced developer can read all of the documentation and then spend 2 hours failing to get sample code to compile then something somewhere is fundamentally broken. This is the current state of automation testing in UE4.

    What are the specific issues?
    It just doesn't work!
    Imagine Joe User sitting down at his computer for a day of work. He's got Visual Studio 2013, UE4 binary 4.9 and is an experienced C++ developer of 20 years. He has been tasked with writing a couple unit tests. He reads through the automation documentation and decides to start with what the docs call a Simple Test using IMPLEMENT_SIMPLE_AUTOMATION_TEST. He copies and pastes the sample code into his unit test, hits compile, the compile fails.

    Poor Joe! He scours the documentation again, looking for any mention of required #include files that he missed but comes up empty. After manually tracking down required include files by grep'ing through the source, he realizes that his project is missing an entire module. He starts to wonder if this is even meant to work on the binary engine, he looks through the documentation again to see if this is mentioned but finds nothing. He considers adding the missing module but is concerned about how this might affect the shipping game.

    This user story highlights a couple issues.

    Unclear Documentation & Requirements
    As a coder, the documentation doesn't tell me how to write a full automation test. It doesn't tell me:
    • What include files are required
    • What engine distribution is required (binary vs source)
    • What module dependencies need to be added to my build
    • A general strategy for testing, mocking, etc.


    Automation Includes have Broken Dependencies
    The include file for AutomationEditorCommon.h has some seriously broken dependencies. If these broken chains are meant to be filled by a missing module then it would be great to at least have a pragma somewhere to print a warning out. The sample code in the documentation calls FEngineAutomationTestUtilities::LoadMap and so AutomationEditorCommon.h is going to naturally be the first include that people try to add to get their test to build.

    Disconnected Workflow
    The workflow for adding a new automation test is disconnected from all other workflows. As a C++ developer, you add new source through the editor. Automation tests could benefit from using this workflow to generate the skeleton class with the correct includes. The workflow right now is:
    1. Kill VS
    2. Manually create a file like some kind of animal
    3. Rebuild UE4 VS solution
    4. Load VS


    Links to Existing Illustrative Questions
    Here is a list of all of the automation related questions that I could find:

    https://forums.unrealengine.com/show...Testing-in-4-6
    https://forums.unrealengine.com/show...mation+testing
    https://answers.unrealengine.com/que...n-c-tests.html
    https://answers.unrealengine.com/questions/148157/writing-automation-tests-do-i-need-to-build-engine.html

    https://answers.unrealengine.com/que...ate-class.html
    https://answers.unrealengine.com/que...nit-tests.html
    https://answers.unrealengine.com/que...is-broken.html

    Closing
    If nothing else, please take 15 minutes to sit down on binary 4.9 with a new C++ project. Read the docs for unit testing and copy/paste the guts of the simple test provided in the documentation. Hit compile and see what happens. Please consider beefing up automation in future releases!
    Last edited by hyperdr1ve; 09-17-2015, 12:46 PM. Reason: add link for unit testing docs
    Dev Blog http://www.error454.com
    Arconia: Twinstick Shooter Adventure

    #2
    Hi Hyperdrive,

    We already have an automation testing system that allows users to not only use tests that have already been pre-designed, but implement their own tests as well. Check out Unreal FrontEnd. This can be accessed in editor or as a standalone application and should cover what you are looking for here. More information on FrontEnd can be found here:

    https://docs.unrealengine.com/latest...end/index.html
    Adam Davis | Marketplace Support | Epic Games
    How to report a bug? | Installation & Setup issues? | Answerhub Bug Reports | Twitter

    Comment


      #3
      Originally posted by Adam Davis View Post
      Hi Hyperdrive,

      We already have an automation testing system that allows users to not only use tests that have already been pre-designed, but implement their own tests as well. Check out Unreal FrontEnd. This can be accessed in editor or as a standalone application and should cover what you are looking for here. More information on FrontEnd can be found here:

      https://docs.unrealengine.com/latest...end/index.html
      I wasn't clear enough. Everything I said above was a direct criticism of the existing UnrealFrontend system. All of the documentation, existing examples and functionality that exists for automation tests in UnrealFrontend is designed for people that are writing automation tests for the actual engine. For someone working on the engine, automation tests are something of a 1st class feature.

      For someone that wants to write automation tests for an actual game, the system is more of a 3rd class experience. Outdated documentation, unclear support and everything else I mentioned above. What I'm asking for is that Game-Centric Automation Testing be on the Roadmap as a 1st class feature.
      Dev Blog http://www.error454.com
      Arconia: Twinstick Shooter Adventure

      Comment


        #4
        I agree with hyperdr1ve's suggestion. I hope testing workflow for UE project will be better in the future.

        As someone who is new to game development, I wanted to isolate and test my logic classes with 3rd party library like catch.
        I came across this thread by Wolfjagger, which showed how to do it . It is rather hackish as it involves modification of UnrealBuildTool.
        Unfortunately I also use the binary installation of UE4.

        Comment


          #5
          Yepp, I agree completely. Is it still the case that you can't hot-reload unit tests? If so, that means that every time you change the test, you have to close the editor and relaunch it, then go to the session frontend, untick the default tests, and then tick your own. That takes a long time! And no word on mocking.

          Comment


            #6
            Originally posted by HateDread View Post
            Yepp, I agree completely. Is it still the case that you can't hot-reload unit tests? If so, that means that every time you change the test, you have to close the editor and relaunch it, then go to the session frontend, untick the default tests, and then tick your own. That takes a long time! And no word on mocking.
            It is still the case that you can't hot-reload unit tests. No mocking, but that's another story I think with the whole uproperty system.
            Dev Blog http://www.error454.com
            Arconia: Twinstick Shooter Adventure

            Comment


              #7
              Originally posted by hyperdr1ve View Post
              It is still the case that you can't hot-reload unit tests. No mocking, but that's another story I think with the whole uproperty system.
              I have submitted a bug report on the lack of hot-reload here. This would at least be something.

              EDIT: Whoops, still waiting for approval! Content (listed as being for 4.11 preview):

              I have also tested this in 4.9.

              Repro:

              1. Create C++ project
              2. Create a simple automation test as per this gist.
              3. Run the test via the in-editor Frontend and observe the console output.
              4. Change the UE_LOG output to some other phrase.
              5. Hit the 'compile' button in-editor.
              6. Run the test again and notice that it has the same output as before the hot-reload.

              You can try this with any sort of change if for some reason console outputs aren't hot-reloaded by default. The results are the same.

              This is a huge problem for the speed and practicality of conducting regular unit tests. At the moment, I have to reboot the editor every single time I change a test. It takes too long to be practical or useful.

              Thanks!
              Last edited by HateDread; 01-12-2016, 06:14 AM.

              Comment


                #8
                Minor update: this Answerhub post regarding tests not hot-reloading was accepted, and a report has been entered (UE-25350). It's not much towards making tests usable on the 'project' level (vs engine level), but it's a good start! Would love to see this fix in 4.11 or 4.12.
                Last edited by HateDread; 01-14-2016, 11:58 PM.

                Comment


                  #9
                  +1 to this thread. Even hot-reload alone would save an enormous amount of time.

                  Comment


                    #10
                    I'd really like some additional documentation, examples, and possibly a way to mock for our unit tests! If we had some way to tie into Google Test: https://github.com/google/googletest That would really accelerate our workflows for creating tests and more stable code!

                    Comment


                      #11
                      This is something I'd like to see too. I prefer the TDD style of development and I'd love to be able to do more of that in UE.

                      Comment


                        #12
                        Bad news, friends - automation tests not hot-reloading is not a bug. Still hoping for a change so we can actually use this framework for quick and iterative project-level testing.

                        Comment


                          #13
                          I strongly agree that we need to be able to hot-reload the tests. TDD and Automated testing are how modern software is made.
                          Please make automation a first-class feature for game developers.

                          Comment


                            #14
                            Oof. I'm trying to assess some of the "lesser talked about features" in UE4 and this one stood out as a biggie. We tend to write turn-based strategy games, so a lot of game logic tends to be written in testable form. The front end all looks great, and the tools seem nice, but man... lacking hot-reload is a huge time suck. If anyone has any work-arounds via external tools or other strategies to writing unit/integration tests in UE4 I'd love to hear it.

                            Comment


                              #15
                              Yeah, not having testing seems like a big oversight. This is 2016. A test suite should be a given.

                              Comment

                              Working...
                              X