Struggling with the implementation of Automation tests

[ZRogers-US]

In the mean time I added a log to my slate UI’s constuct function, playing in the editor I can see the log but when running the test I don’t see the log, I’ve tried to run my test in latent functions as well as putting the It function on a separate thread. Someone on the Unreal Discord asked me if I was running the tests in editor, PIE or ***. Tbh I’m not 100% sure on this as this is the first I’ve seen different methods mentioned but I decided to try have a look into this more and found there’s a RunPIE function under the FAutomationEditorCommonUtils class so I tried implementing this and I can at least see that my log runs but it still fails to locate the button.

Ben UI got back to me and unfortunately has only had experience with the simple and complex test and suggested asking in his discord which ill link here in case anyone else would like it. Unfortunately, I cant access it at work but will be joining it afterwards.

I decided to try load the map with the AutomationOpenMap function and found this was causing the slate log to display as well with out the need for RunPIE. When testing this I found that the log saying the button exists was also displaying however, the test was hanging and using debug I found it was hanging on the Click() function. So I set the It function to run on a separate thread again and success

Now to build on this

[ZRogers-US]

I’ve implemented functionality to my test button to load another map, and testing this manually I can see its working correctly. So I’ve added another It function which will check the map name and test if its the expected map. After running the test the first It function, like before, passes with no problems However, my second test failed, I believe this is because the world was still loading. Unfortunately I don’t seem to be able to use Latent commands so I cant wait for the map to load so I’m currently changing it to an LatentIt.

Okay after reading up on LatentIt again I’m not sure this will help as a LatentIt will not move onto the next bit of code until the Done delegate is executed. Because of this I’m thinking I might need a LatentIt before the test to wait for the map to load. This is assuming that the LatentIt functions loop like Latent Commands.

So my latent It was timing out and after adding some logs I realised my BeforeEach is running and loading the starting map before check the new map name so they would never be equal. I tried to nest another describe inside the first in hope it would run the BeforeEach before the second describe and the AfterEach after it so they would only run once for the three It functions. But that doesn’t seem to be the case, so I’ve moved my map loading into the first It function. This hasn’t worked because the It function is running on a separate thread.

Unfortunately, I’m going to be moving to training only 1 day a week soon, so progress might be a bit slower, but I plan to continue this post if only for my own tracking purposes.

[ZRogers-US]

So I worked out how to get my map to load once, I’ve added a nested describe with the first It function and a Before Each, this loads the map for the first test but not for the second which is in the first describe function.

But my second test (to check the correct map has been loaded) which is a latent It still times out despite a log showing the correct map was loaded.

Decided to test it without it being a latent It and it worked

Thinking I’m going to expand on this a bit so I wont be pushing to git just yet but I will post when I have along with the Slate UI I’m using :).

[ZRogers-US]

Oh wow 21 days that’s quite some time, been on holiday :).

So came back in today and decided to look into the fbx builder I haven’t forgotten about uploading the driver code but I recently found an example with UMG so going to try implement that first ( ArabicProjectClean/Source/ChibaChaba/Private/Tests/Test_UMG.cpp at main · PablushaRubeCube/ArabicProjectClean (github.com)).

The FBX Test Builder is no longer under Tools> Automation but is now located under Tools>Debug>FBX Test Builder

unfortunately the fbx selection is only showing fbx files in the engine contents folder, so for now I’ve just selected the Cube_Blue.fbx to experiment with. So I created a testplan for the Cube_Blue.fbx and tried setting the LOD number in the expected result to 0 which should fail as it has 1 LOD, which I did and setting it to 1 passes. The test can be found under Editor>Import>Fbx>FBX Name

I was able to perform a test to check there’s no errors when importing, without having a FBX to import I first ran the test with it expecting an error which failed and then changed to 0 for a pass.

So adding the second test plan wipes the first from being tested so would seem they have to be implemented into one test plan.

This isn’t quite what I was expecting it to be, I was more expecting it to test each time something gets imported but can see how this can be useful.

Ahh okay so further down it states to move the fbx file to “[Unreal Engine Root Directory]/Engine/Content/FBXEditorAutomation”, hmmm still not showing but could be because this model I downloaded for the test doesn’t import correctly.

So I’ve downloaded a new model and added it to the FBXEditorAutomation folder but I’m still unable to select it from the dropdown. Turns out I had just put in in the wrong folder (different version of UE). I thought this model had multiple LODs that I could test but I was mistaken, instead I tested that it created 5 materials, a static mesh when imported and only one LOD. which passes successfully.

Having got the hang of the basics of the FBX test builder I thought I’d look into running test from the CMD and generate a report. Found this page in the documentation that’s very useful (Run Automation Tests in Unreal Engine | Unreal Engine 5.4 Documentation | Epic Developer Community (epicgames.com)). Using this page I was able to run all the tests in the CropOut sample project, I was working on, quit the Unreal Engine Editor and save the report to another location. I then added this to a batch script for ease of use. Only thing that page didn’t mention was adding the uproject file location after the UnrealEditor.exe.

My Command:
“D:\Program Files\Epic Games\UE_5.3\Engine\Binaries\Win64\UnrealEditor.exe” “D:\UnrealEngine - ZR\Cropout\CropoutSampleProject\CropoutSampleProject.uproject” -ExecCmds=“Automation RunTest Project.Functional Tests.Tests;Quit” -ReportExportPath=“C:\Users\zrogers\OneDrive - Universally Speaking Ltd\Documents.AutomationTesting\Run Tests From CMD/Results”

One nice thing I noticed is that the fails are put at the top of the list in the report

[ZRogers-US]

I’ve gone back to look at the automation driver for UMG today I’ve created a basic CPP UserWidget class with one button for now, then created a BP widget based of that class. I’ve been trying to add the automation ID meta data into the UPROPERTY but so far with no success. Looking at BenUI’s page on UPROPERTY All UPROPERTY Specifiers · ben🌱ui (benui.ca) and the Unreal documentation Metadata Specifiers in Unreal Engine | Unreal Engine 5.4 Documentation | Epic Developer Community (epicgames.com) I’m not sure this is possible but someone in the discord servers stated they had done something similar in the past and the documentation for the automation driver states:

Okay my code has just compiled with the UPROPERTY meta “id” which doesn’t appear to be on those lists, going to replace my main menu with it and see if the driver can find the button, I’m not expecting it too tbh, yer that failed to locate the button.

I’ve been asked to focus on GAS atm so mainly learning how to implement that correctly now, I did manage to run some tests on GAS in the Lyra project but tbh i really wasn’t too sure how the system works. When enabling the testing plugins again for this project I noticed the WidgetAutomationTests plugin so might try look at this later.

[ZRogers-US]

So I’ve implemented a basic mana attribute that is reduced when the player attacks. The first test I wrote was a blueprint test were I was checking the mana reduced as expected when attacking, this didn’t take very long to implement in blueprint. However, I am now trying to implement the same test in cpp.

I started with a simple test however my character is a BP actor and I was unable to spawn it in the same way I was for the spec test so for now I’ve switched to the spec test. I’ve got the player spawning but when I try access the attribute set to get the value of the Mana attribute I’m getting an access issue and in debug the attribute set is Null (I’m using the get function created in the attribute set through the macros). I added a log to the character’s begin play where the attribute set is assigned but that’s not being called during the test.

Again I’ve run into a similar issue that I’ve had before and I have to assume that tests don’t tick because I cant find anything about this online. I’ve tried to use the world->tick function to get the begin play to run. Which doesn’t appear to work as I’m not getting the log and my It function doesn’t appear to run but the test still fails. Going to try again but by using world->SetShouldTick(true). This also has not worked, character was spawned but Access violation when trying to access the attribute set to get the value of mana.

I decided to create a new map that’s empty and load that instead of creating it during the test, The begin play for my character appears to run now as I’m getting the log (so maybe I’m just creating the maps incorrectly for my tests) unfortunately I’m still getting an access violation and the attribute set for the character is still Null. Whilst this was compiling I did find another way to get the attribute value so I’m going to try that way( GetNumericAttribute function through the ability system component), I also came across “AbilitySystemComponentTest.cpp” so I’m going to have a read through that and see if there’s anything useful in there. Well GetNumericAttribute worked :).

[ZRogers-US]

So I managed to sort the test which checks the initial mana value but whilst trying to check it after an attack I’m unable to get the attack function to be called. I first tried finding the BP function and calling that.

Unfortunately, this wasn’t erroring but also didn’t seem to be calling the functions, at least my logs were not displaying. So I though I’d try implement a function in CPP that I could override in BP and just call the function that way unfortunately this also didn’t work

Sadly I think my days of training are numbered, if I find time at home to continue I will though would be on a different account. Might be a little while though as I’m creating my own game in Unity which I’m planning to be doing TDD on. Also the slate UI and driver test are added to my GitHub repo if I haven’t already mentioned this.