Hi there, I’ve recent started looking into automation testing in unreal engine 5.3.
I was hoping that someone could point me towards some resources for learning this as I cant make heads or tails of the documentation and it seems to be missing important information.
I have managed to implement a Functional test using blueprints and a Functional Test Actor but ideally I would also like to learn how to implement it in C++ too. I have attempted this and to start with the engine was just crashing whilst trying to run the test, managed to fix that issue but the test doesn’t actually run and times out, am I supposed to me overriding specific functions where the testing code should be located? unfortunately the only thing I can see in the documentation for this is implemented in blueprints.
I decided to take a break from that and write a spec test but have run into a couple of issue with this too. the first being, I’m assuming that the test class should be appearing in the content browser which is isn’t (likely cause I’ve messed up the code I expect). what class should these, if any, tests inherit from? one tutorial I came across talks about ‘Fixtures’ are these still required as I’ve seen no mention of them in the unreal documentation.
I also tried taking a look at the Automation Driver but again I’m unsure if the class needs to inherit from another class or just start with a blank class. I have since seen an example of the driver being used in ‘Engine/Plugins/Tests/AutomationDriverTests/Source/AutomationDriverTests/Private/AutomationDriver.spec.cpp’ on GitHub so I’m assuming it needs to be implemented into a spec test.
I’m only learned the basics of ue’s autotesting, but here is a small example i put together while trying to make it work. It’s not the complete project for reasons, but it does contains links for relevant docs.
Hope it helps you get started
A few notes about content:
it implements the simple class to test, as well as two ways to test this class. Which way to use - afaik it’s only about your preferences
Automation Driver
iirc this one is for something like ui testing. It’s also can be autotested, but since you talking about functional testing, my example should be relevant
Hi thankyou for the reply your examples have helped me a bit however, I’m still running into an issue when trying to test a component.
I have added an ammo component to a character that can shoot, when the character fires a shot a function is called on the ammo component to reduce the ammo count.
The tests that I’m trying to preform are:
Should start with a max ammo count of 30.
Should start with an ammo count of 30.
Should have an ammo count of 29 after shooting once.
I have a BeforeEach where I am getting my character and assigning it to a variable successfully and the first two test work correctly, I am able to get the max ammo count and ammo count from the ammo component and test their values.
The problem I’m having is with the final test, I’m able to call the fire function but the ammo count fails to update so the test fails. I have added some debug logs, I have a one in the Fire function which isn’t displaying but if i change the call to the Fire function in the test to straight call the reduce ammo count function I can see that is working correctly.
I did wonder if it possibly had to be a latent It so have also tried implementing one of those several different ways unsuccessfully, including your example one, but the tests keeps running until the time out time expires.
Okay so after adding some more logs I’m finding that my Fire function gets called but fails to enter the if statement ensuring that the projectile class exists. ( which is set in a BP).
I though the penny had dropped when i wrote than im assigning the projectile in the bp yet im spawning the C++ class. i’ve tried to change this to spawn the blueprint class instead but its currently crashing when i run the test.
So i ended up getting and assigning the projectile class in the C++ class for the character and the test is successful but i still would like to implement it with the blueprint class as this might be how they are implemented in some cases.
has anyone got an idea on how to spawn a blueprint class in a Spec test?
HI all I just wanted to update this so if anyone else is struggling with Spec tests my experience might help. My code will probably not be the best way but it worked for me (in UE 5.3)
So for my test I had create a character C++ class that can shoot and an ammoComponent C++ class which is attached to character. I then made a blueprint of the character and assigned my projectile in there.
The tests i wanted to run were:
What’s the ammoComponent’s max ammo at the start? should be 30
What’s the ammoComponent’s ammo count at the start? should be 30
What’s the ammoComponent’s ammo count after a shoot has been fired? should be 29
To start with I was trying to spawn my character class during the test and the first two tests would pass but the last would fail, it turned out this was because I was spawning my C++ class where the projectile wasn’t assigned (doh). After realising this I assigned my projectile class in the character’s constructor and success.
I still wanted to try this out with the projectile assigned in blueprint so commented out where I was assigning it in the C++ class. After a lot of trial and error and help from Francesco over at Automated Testing With Specs in Unreal Engine | Francesco’s GameDev Corner (minifloppy.it). I managed to spawn in the blueprint class using a soft class pointer and the FSoftClassPath function in my BeforeEach function.
I then spawned the BP character
For my first test (with the C++ character) I was creating the world using Francesco’s World Fixture but in the second (with the BP character) I created the world with UWorld::CreateWorld and then destroyed the world in the AfterEach function. Both of these way seemed to work without issue.
Fortunately I haven’t run into any issues with a simple automation test and will add my example to the git repo soon
The complex example from the documentation appears to be out of date however the example has been updated in the source code (EditorAutomationTests.cpp Line 180 - 273):
I decided to try write my own complex test to find all enemies on a map and check their max health values depending on their type. I did realise during implementing this that it could probably just be done with a simple test but at least its some experience on a complex test. Unfortunately, after writing the test it wasn’t displaying in the session frontend. As I wasn’t too sure I’d written it correctly I though I would implement the load all maps example exactly as it is in the engine source but this still isn’t displaying in the session front end .
Okay I got it showing (the load all maps example) I don’t if I want to admit it but was cause I wasn’t on the right dropdown filter.
I then decided to comment out the code for that test and add my code in, re-compiled and was no longer visible so I’m assuming my test code isn’t correct.
I was correct I remembered reading that someone’s tests weren’t showing because they weren’t adding anything to the arrays, I tested this but just adding a “test” string to both arrays and it displays in the session frontend. progress this also now allows me to debug through my code
Right so the issue is that its not finding my world so then not finding the actor and adding the strings to the arrays.
I should probably mention that in my GetTests function I’m trying to load a map and find all the enemy actors in the map and then adding a string to the arrays based off them. the plan to then check each of those enemies in the run test function to check there health values
so I got my complex test working in the end, had to get the world through the world context. my codes not the best but I have also uploaded the simple and complex tests to the git repo incase anyone wants to look at my examples. Unfortunately, ive been moved onto implementing some tests into a sample project but hopefully can go back to learning the other test types shortly and will likely keep updating this
So I ran out of things that I could work out how to easily test in the sample project (I’m using the Cropout sample project) so thought id try out a Automation Driver test for the main menu. after alot of faffing and intelisence moaning at me I worked out I need to use the AutomationDriver module, thanks Epic for mentioning this in your documentation. Unfortunately, my simple test of loading the main menu map and finding and selecting the new game button cause unreal to just freeze :/. I commented out all the code and uncommented bit by bit as it wouldn’t even let me debug it and narrowed it down to the click function, using the Exists function I found that my button was not found. The documentation mentions UI element IDs however I was unsure what they were as I’ve never seen mention of them having an id before, as they don’t have a specific ID in the details i went with this. Maybe this was wrong?
yer it was re-reading the documentation is mentions adding a explicit Automation Driver metadata Id tag to the widget, so just trying to work out how to do this in blueprints/graph.
Okay i might be wrong here but i believe you have to create the widgets in c++ in order to be able to add metadata to it so looking like i’m putting this on hold again for now as the sample project im using is made with blueprints. If i am wrong please correctly me and point me in the right direction thanks
In case anyone is interested in my automation tests for the sample project, I’m using the Cropout Sample Project - Unreal Engine project which is created in blueprints. As a result I used the blueprint functional Actor to preform a bunch of tests, I haven’t built on the game at all just added some different tests.
The tests I currently have are for checking:
That each of the resource points starts with the correct number of resources.
The amount of each resource that the user starts with.
Adding resources and spawning a villager, ensuring the values correctly update.
Removing resources to ensure the player cant go into the negative.
Removing food to 0 to test the lose state.
Spawning a monument to test the win state.
I grouped different related tests that wont interfere with each other together in test maps and had to temporally disable the auto load feature as the autosave/autoload feature was causing issues when tests where run one after the other and unfortunately I’ve been unable to ensure it loads a fresh save before running a test so far.
I’ll post some images of the different blueprint tests below
I think I forgot to mention for the win and lose test I had to set the “Log Error Handling” to “Output Ignored” as I was experiencing errors that I believe were related to the input, this probably is not advised but the errors where the only reason the test was failing and was unrelated to what I was testing.
I’ve added some more tests to the sample project this time testing the job assigning functionality.
For these tests I had to create a separate map to ensure the resource was near the villager and that the villager didn’t have time to unload their resources, in the map I added the resource source for the job I was testing, a villager and a townhall. I also had to include the navmesh so the villager was able to walk around. I started with a parent class and created a child for each job/resource.
I started by finding the villager and assigning the new job, then there’s a delay for the amount of time it takes to gather the resource and an extra 4 second to account for walking to the resource. Once the resource had been gathered I checked the resources held by the villager match the expected resource and if so passes otherwise fails.
Last lot of tests then moving onto the lyra sample project
Testing that the resources are destroyed once they are depleted, I started with a parent class like with the other tests. During the test preparation I find the specified resource and update the collection amount to match the total amount to speed the test up. during the test I then assign the required job to the villager and then wait some time for the villager the gather the resource. Finally, I then check if the resource actor is still valid and fail if it is and pass if its not. I then placed the function test actor in a test world with the one resource of the type that is being tested and the town hall.
I have realised after completing these that I could probably improve the testing time by performing the test check in the tick instead of having a delay.
I started off trying to write a C++ test however was having an issue where when looking through all actors I couldn’t find the player, map appeared to be loading fine and all other actors where found except the player’s. So I took a break and create a blueprint functional test instead which I had more success with, and possibly worked out why my C++ test was working before when I created a map which was that there was no set experience ill be trying to add an experience after creating the world next time.
In the mean time I’ve created three tests to check the powerups in the top down arena game mode. I started with a blank map and added a navmesh and the powerup actor. Like my previous functional tests I created a parent class which could then have a child class for each of the powerups. However, I created the parent class from a base functional test actor that implemented the prepare test event which got the player pawn and assigned to a variable.
So back to the C++ test, plan for this test is to load a test map, move the player towards a B_TeleportToUserFacingExperience which is a teleport to load an experience, then wait for the map to load and test the map name to confirm the correct map has been loaded.
Before I though that the fact there was no experience selected on the test map so the map wasn’t loading correctly however since instead of creating the map in C++ I created a test map where I set the experience in the world settings. Unfortunately I was still unable to locate the player using GetPlayerPawn. Currently I’m trying to load a test map that I’ve already implemented a successful function test on to see if I can get the player pawn on that map. This also did not work.
To check this was just an issue with my implementation in lyra I went back to a previous project and implemented the same test in there. Hmmm so not working on the other project either, going to change it from GetPlayerPawn to GetActorOfClass instead. No luck there either.
I have also tried creating a new map and spawning the required actors and assigning them to a variable, this also produced a null value and the actors were unable to spawn due to not specifiying a class, turns out I missed the _C from the end of the reference. Adding that fixed that issue, so I’m able to spawn the actor into a newly created world. Just have to try add the experience to the teleport now.
One post I’ve come across whilst trying to get this test working mentioned not mixing latent and non-latent commands so I’ll try implement each step as a latent command in case its trying to get the actor whilst still performing the latent load map command.
Will be trying to implementing this and adding the experience to the teleport tomorrow so if anyone is able to point out something else I might have overlooked or point me in the right direction would be much appreciated.
I was unable to add the experience to the teleport during the test setup so instead I’ve duplicated the original and manually assigned it instead and then spawning that one during the test instead. Not ideal but I just want to get it working atm.
So I’ve created a couple of latent commands for moving the player, checking the map name and finally destroying the world. So my test currently creates a world, spawns the two actors sets them to two specific positions then preforms the latent commands .
So this was just looping through the Latent movement command so I’ve added a log to ensure the actor is actually moving. Well great new it is moving but I had a set location with no tolerance so I’ve adjusted that now, however I’m a bit concerned as the collision should have been hit anyway so the new map should have loaded but didn’t appear to.
New movement latent command is now working but its now looping in the WaitForMapToLoad command which I was kind of expecting. Now just working out how to check if the player is colliding with the teleport. So i was thinking i could use the IsOverlappingActor function however this requires a AActor* where as I’m using a TWeakObjectPtr. Well this will continue tomorow.
so I implemented the CheckPlayerIsColliding latent command using the IsOverlappingActor function however this is just looping and so the player doesn’t appear to be overlapping the teleport collider. I decided to go back into my teleport blueprint class and adjust the collider to be 10x bigger and reran the test with no change. Currently I’m adding some break points to the blueprint class to see if the overlap event is called there. So they aren’t triggering either but if I place it manually in a map there’s no issue I overlap, breakpoint is hit, map gets loaded.
I tried copying my test over to a non lyra project in case there’s something in the lyra set up that I’m missing but the same issue occurs in that project, manually it works but fails to detect the collision during the test.
So I realised I wasn’t performing a sweep when setting the actor location, now the actor isn’t getting close enough to the teleport but using a FHitResult I can see that the player is colliding with the teleport. I’ve now adjusted the movement latent command to continue (return true) if the character is colliding with an actor, wasn’t working but by adding a breakpoint I’ve realise I was being stupid and resetting it before the check . oh well tomorrows problem
I recently found the UnrealSlackers discord but unfortunately I cant use that at work so going to be posting in there after work hopefully someone there might be able to help with the joys of automation testing. Also found another automation in game dev server so going to join that too Automated Testing In Gamedev. I do apologies for this tread to anyone that might pop over from there to give a hand .
So having fixed the hitResult the latent command is now completing but I cant tell if the new map is being loaded, my check for the checking the name of the new map is just displaying the name of the map in the editor at the time the test is running. So instead I’m getting the world from the character and its displaying the correct test map (untitled) unfortunately it does not appear to be attempting to load the map after colliding with the teleport. I’m currently trying to load it manually in the test to confirm that my debugging is updating when the new map loads or not.
So I’ve read a forum post that was saying that the world create doesn’t tick I assumed that the latent commands Update function ran off tick, but having manually load the map I noticed it said the new map had a max tick rate of 0. I’m assuming this means its not ticking and I believe the onOverlap runs off of tick?
So I tried to enable tick on the world after creating it but unfortunately no change. So I suppose my questions today would be, does a map/actor tick during automation tests? are you able to run tests over multiple maps? (I assume so as my manual map loading was working) has anyone got a clue why my map loading isn’t working? (triggered through collision with the teleport actor) also out of curiosity what’s the difference with a custom simple/complex test?
So I’ve come to the conclusion that automation tests do not run tick, I added a print string to the teleports tick and can see it in the log when PIE but not when running the test. So I though I’d try broadcast the Begin Overlap Event after checking that the character is colliding with the teleport. Unfortunately, that still fails to run the code in the teleport’s overlap function which is supposed to load the new map . Using HitResult->GetNumOverlapHits() is telling me that although the character is colliding with the teleport apparently its not overlapping with the teleport?
I decided to finish off the rest of my test to check the map name which is now implemented and the test passes with me manually loading the map. However, I have notice that when the test has finished running through, the main menu is loaded but it still takes 60 seconds for the test to complete, With the blueprint functional test I know I can adjust the timeout time or use the finish test function but I’m not too sure how to do this with a simple test.
So going back to the the onActorOverlapBegin broadcast in debug I can see it states that bIsBound is false, despite there being functionality in the Teleport’s blueprint class. I tried adding the load level from the overlap event into a function so I could try calling that instead to see if its the event or the blueprint code that was failing to run, this also didn’t work. I tried again but with a test function that just prints to the log and that doesn’t display either so might be my way of calling the function is incorrect.
I though it had been going so well, decided to add my test back into my Lyra project and now the two actors don’t even collide so stuck in the Latent move command.
So today I decided to take a break and try work on a automation driver test, I believe I understand how to setup elements for a Slate UI, does anyone know if it is possible to also use it on UMG, I feel like you should be able to as its built off Slate. But I’m unsure how to go about locating the elements as I can’t/don’t know how too set an Automation Driver Id metadata or locate the element via the path. tomorrow I’m going to try implement a slate UI to perform the test on but would like to know about UMG too.
To start with I was trying to find an element in a UMG UI via the hierarchy path, not sure if this is possible/how it should be done but I thought id give it a go. This was just crashing every time the test would try to find the element due to it being inaccessible, putting this down to me not trying to find the element correctly I decided to start with Slate as I knew how to add the ID tag so could find it that way.
I’ve written my first slate widget, just a simple button which has no functionality atm, I just want to be able to find the element for now. I added the metadata tag id and then tried to find the element by ID, unfortunately this didn’t find the button but at least it didn’t crash this time. after trying to find the element I check whether it exists before trying to click it and unfortunately it apparently doesn’t exists, so I’m currently trying to find it via the path but using the ‘#’ and ID. this didn’t work either.
I have to say there seems like there is very little out there on the Automation Driver but looking at the Automation Driver spec test and the spec suite I’m not too sure what I’ve done wrong here. I am currently doing my test as a simple test and the documentation does recommend a spec type test so you can ensure its off of the game thread so I guess I’ll try and implement it that way and see if it makes a difference. Currently this is building but wont be finished before I leave work so will have to wait until Monday to find out if that helped.
Whilst working on the other test types I came across BenUI’s website where he has some examples of a simple and complex test, I was having a little nose around his website and a few of his youtube videos When I noticed that he works at epic on UI so was thinking I might be a little cheeky and drop him an email to pick his brain.