Announcement

Collapse
No announcement yet.

Need Help with Automation Testing in 4.6

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

    Need Help with Automation Testing in 4.6

    I desperately need to get unit tests going in my project. Does anyone have samples of UE4 automation tests written for a game?

    I've been going through the existing automation tests but haven't had luck getting them to work with my game.

    For example, this simple test taken from EditAutomationTests.cpp fails to compile, Rob1E.h is my game module.

    Code:
    #include "Rob1E.h"
    #include "AutomationEditorCommon.h" // AutomationEditorCommonUtils
    #include "LevelEditorViewport.h" // FLevelEditorViewportClient
    #include "LevelEditor.h" // GEditor
    #include "AssetSelection.h" // FActorFactoryAssetProxy
    
    IMPLEMENT_SIMPLE_AUTOMATION_TEST( FAutoTargetTest, "Rob1e.AutoTarget", EAutomationTestFlags::ATF_Editor)
    
    bool FAutoTargetTest::RunTest(const FString& Parameters)
    {
    	//Open a new blank map.
    	UWorld* World = AutomationEditorCommonUtils::CreateNewMap();
    
    	//Move the perspective viewport view to show the test.
    	for (int32 i = 0; i < GEditor->LevelViewportClients.Num(); i++)
    	{
    		FLevelEditorViewportClient* ViewportClient = GEditor->LevelViewportClients[i];
    		if (!ViewportClient->IsOrtho())
    		{
    			ViewportClient->SetViewLocation(FVector(890, 70, 280));
    			ViewportClient->SetViewRotation(FRotator(0, 180, 0));
    		}
    	}
    	return true;
    }
    The compile log seems to implicate the included headers but I'm a little out of my depth here.

    All I want to do is load an existing map or create a new one, spawn my classes and call latent functions on them. I can't find an actual description of the architecture of the framework, why can't I just use GetWorld()->SpawnActor like I would in my game code vs using FActorFactoryAssetProxy? Please tell me what I'm missing here, unit testing can't be this painful.
    Dev Blog http://www.error454.com
    Arconia: Twinstick Shooter Adventure

    #2
    Hello,

    Apologies for any inconvenience you've may have had with writing your automation tests.

    For your test it looks like you may be missing this: #include "UnrealEd.h"
    There's a chance that may still not work for you. If it doesn't please let me know.

    For more information please check out this page: https://docs.unrealengine.com/latest...ide/index.html

    --Justin

    Comment


      #3
      I've added UnrealEd.h but it doesn't seem to have changed the footprint of the compile errors.

      I've read through all the docs on automation testing a few times now but there are some gaps in my knowledge that the docs do not answer.

      Required Headers
      First of all it's not clear what the required minimum headers are for say IMPLEMENT_SIMPLE_AUTOMATION_TEST. The docs don't actually mention any headers except for "AutomationTest.h" in passing as a reference to where EAutomationTestFlags can be found.

      Helper Classes & Basic Explanation
      The second thing I haven't grok'd is how in the world I do the everyday things that I would do in my actor. In a normal set of game code, my level is loaded, my game mode kicks in, my default pawn is spawned and I can write custom code in my game type or an actor to spawn and manipulate things. Whereas the unit test seem to start with nothing, if all I'm doing is functional testing this makes sense. But for everything else, for the really non-trivial stuff that I'm trying to test in the first place, I need all of those equivalents in place, levels need loading, pawns need spawning etc.

      I'm familiar with doing these things from a game perspective, spawning actors, streaming levels, etc. But whatever equivalents might exist on the testing side escape my immediate notice or are lower level functions I've never needed, for instance the test I've tried copying above that is directly manipulating the viewport location/rotation - this is something where I'd expect to spawn a camera, set a view target and set the location of the camera.

      A Real Example
      The final thing missing is a real example in an actual game project. I've grep'd through all of the downloaded sample projects I have for IMPLEMENT_SIMPLE_AUTOMATION_TEST and came up empty. Since the engine unit tests and a game unit test don't seem to be interchangeable it would be great to have an example to base things on.
      Dev Blog http://www.error454.com
      Arconia: Twinstick Shooter Adventure

      Comment


        #4
        Automation Testing

        Hello again,
        These are great questions. I’m going to try and answer some here as well as try to get one of our other programmers to chime in as well with their input.

        >>I've added UnrealEd.h but it doesn't seem to have changed the footprint of the compile errors.
        Are you using a project created from the engine downloaded from the launcher or Github? Projects created through the launcher are not able to create editor automation tests due to the fact that the editor source doesn’t come with it. Through Github it is expected to work.

        >>First of all it's not clear what the required minimum headers.
        AutomationTest.h is really the minimum for an engine/runtime automation test. The UnrealEd.h include is something you’d need for writing editor automation tests.

        >>I need all of those equivalents in place, levels need loading, pawns need spawning etc
        For automation testing there are only a few helper functions and even fewer that are exposed for you to use. We do have a ticket in place to get the rest of them available. You can find some things in the AutomationCommon.h file.

        >>I've tried copying above that is directly manipulating the viewport location/rotation
        This is because, most likely, you’re attempting to use editor code and that isn’t possible using a project created from the launcher.

        >>this is something where I'd expect to spawn a camera, set a view target and set the location of the camera.
        I would imagine for in game automation camera movement that isn't tied to a specific player you’d use the spectator camera.


        Game Automation Example Code:
        The below example was created using the First Person Code Template. I named this project AUTOTest.
        The goal of this test is to verify that the playable characters can fire their weapons.


        I will say this code could be better as I need a real way to verify that the weapon actually fired.

        This code was placed in it's own .cpp file.

        Code:
        #include "AUTOTestExample.h"
        #include "AUTOTestExampleCharacter.h"
        #include "AutomationTest.h"
        #include "AutomationCommon.h" //To get the wait latent command.
        
        //MYAutomation Logs
        DECLARE_LOG_CATEGORY_EXTERN(MYLOG, All, All);
        DEFINE_LOG_CATEGORY(MYLOG);
        
        /**
        * Latent command used to fire the players weapon since we want this to happen over multiple frames.
        * @param InExampleCharacter - An existing character using our example character class.
        */
        DEFINE_LATENT_AUTOMATION_COMMAND_ONE_PARAMETER(FFireWeapon, AAUTOTestExampleCharacter*, InExampleCharacter);
        
        bool FFireWeapon::Update()
        {
        	if (InExampleCharacter)
        	{
        		//Log to display, so that it shows in the UFE results, which character is firing their weapon.
                        UE_LOG(AUTOLOG, Display, TEXT("Fired weapon for '%s'."), *InExampleCharacter->GetName());
        		
                        //Function added to the character class that allows us to use the protected: function named "OnFire()".
                        InExampleCharacter->AutomationExecuteOnFire();
        	}
        	return true;
        }
        
        /**
        * Gun Shoot - Verifies that the player can fire their weapon
        */
        IMPLEMENT_SIMPLE_AUTOMATION_TEST(FFPSGunShoot, "Automation Tests.Gun Shoot", EAutomationTestFlags::ATF_Game)
        
        bool FFPSGunShoot::RunTest(const FString& Parameters)
        {
        	//Cycles through each character in the level using the AAUTOTestExampleCharacter class
        	for (TObjectIterator<AAUTOTestExampleCharacter> It; It; ++It)
        	{
        		AAUTOTestExampleCharacter* ExampleCharacter = *It;
        
        		if (ExampleCharacter)
        		{
        				ADD_LATENT_AUTOMATION_COMMAND(FFireWeapon(ExampleCharacter));
        				ADD_LATENT_AUTOMATION_COMMAND(FWaitLatentCommand(1.0f));
        		}
        		else if (!ExampleCharacter)
        		{
        			UE_LOG(MYLOG, Error, TEXT("Unable to find and fire the weapon for '%s'."), *ExampleCharacter->GetName());
        		}
        	}
        
        	return true;
        }
        You can also add automation functional tests into your maps. I'm going to ping one of our programmers and see if he can give you more information on that.

        hopefully this stuff helps.

        Comment


          #5
          Thanks for the response! I will be revisiting this next week and going through the example you provided and will let you know how far I get.
          Dev Blog http://www.error454.com
          Arconia: Twinstick Shooter Adventure

          Comment


            #6
            So I've finally gotten back to this. To answer your question, I am using the github version of 4.6. I was able to get some functional tests going ok based on your example! I think the piece I didn't understand there was the entry point, for instance the test flag indicates that a map is already loaded and things already exist in the world.

            What I'm still unable to do is use anything in AutomationEditorCommonUtils. Is it possible I'm missing a module in my Build.cs?
            PublicDependencyModuleNames.AddRange(new string[] { "Core", "CoreUObject", "Engine", "InputCore", "UMG", "Slate", "SlateCore" });

            The linker basically can't find any of the symbols referenced in AutomationEditorCommon.h, GLevelEditorModeTools, FBuiltinEditorModes etc.

            Am I doing something abnormal? I mean, are editor-level tests only ever meant to exist alongside the other editor tests? Do these need to live in the engine codebase instead of our game project? I'm basically trying to write a test that iterates through all our maps and finds SceneCapture2D components to validate that we don't accidentally enable certain show flags.
            Dev Blog http://www.error454.com
            Arconia: Twinstick Shooter Adventure

            Comment


              #7
              At this point I'm able to write some useful automation tests. But the workflow to run some of my validation tests is still a little confusing to me. I can manually load a map in the editor and then launch various tests from the Session Frontend (yay it works!).

              But how can I achieve the ability to load all maps and run a set of tests on them? Given that I can't seem to use the utils.
              Dev Blog http://www.error454.com
              Arconia: Twinstick Shooter Adventure

              Comment


                #8
                Hello,

                Sorry for the late reply as apparently I don't get notified when a thread is updated. :-/

                I'm happy to hear you are getting traction on your own tests. :-)


                >>Am I doing something abnormal?
                I don't believe you are doing anything out of the ordinary.

                >>I mean, are editor-level tests only ever meant to exist alongside the other editor tests? Do these need to live in the engine codebase instead of our game project?
                Yes and yes. I don't believe you can compile game code with editor code along side it.
                Well you maybe able to if you do something like this:

                Code:
                #if WITH_EDITOR
                ...do stuff...(you may have to do this with certain headers as well)....
                #endif
                >>Given that I can't seem to use the utils.
                We do have a ticket in already for some of the utils not being public.

                >>But how can I achieve the ability to load all maps and run a set of tests on them?
                You'll definitely want to make a complex test for this. The complex test uses the GetTests() function to create a set of tests without having to rewrite the same test over and over and over again. The function adds the test names and test commands to the OutBeautifiedNames and OutTestCommands Fstring TArrays. You'd then access these names in the RunTests() function.

                Running a set of tests on a map you'll want to use the gettests() to grab the map name and file location so that it knows where to open. The map name is what will be listed in the unreal frontend.
                You can grab that map info in several ways. I would suggest looking into the FPackageName and FPaths classes to help. FEditorFileUtils class is useful as well (located in FileHelpers.h)


                hopefully this helps.

                Comment

                Working...
                X