Automated test crash game while loading level on packaged build on device

Development Programming & Scripting C++
, , , , , , ,
1

I am trying to learn how to test UE games
I have a default Third Person project template, configurated for mobile platforms, on which I try to implement tests, with my Android smartphone connected to a laptop via USB Debug mode
But I`m stuck on this point for a few days:
When I am trying to test gameplay features, even fundamental ones- like spawning Actor, my tests crash the game instance on the Level loading.
When I run autotest through Automation Framework, in Engine Editor- all works correctly, with no crashes.
It crashes only on game build - both Windows and Android.
Example of the simple test that I try to run:

#include "CoreMinimal.h"
#include "Andro/AndroCharacter.h"
#include "Misc/AutomationTest.h"
#include "Tests/AutomationCommon.h"
#include "AITestsCommon.h"


BEGIN_DEFINE_SPEC(FBasicTestsSet, "MyOwn",
                  EAutomationTestFlags::ApplicationContextMask/*ClientContext*/ |
                  EAutomationTestFlags::ProductFilter |
                  EAutomationTestFlags::HighPriorityAndAbove);
UWorld* TestWorld;
AAndroCharacter* Character;

END_DEFINE_SPEC(FBasicTestsSet);
void FBasicTestsSet::Define()
{
	Describe("Testing basic Gameplay elements", [this]
	{
		BeforeEach([this]{});
		It("Autotest should to spawn PlayerCharacter", [this]
		{
			const char* LevelToLoad = "/Maps/TestLevel";
			TestWorld = FAITestHelpers::GetWorld();
			
			auto LoadLevel = AutomationOpenMap(LevelToLoad);
			
			
			const char* BPName = "/Game/ThirdPerson/Blueprints/BP_ThirdPersonCharacter";
			const UBlueprint* ObjectToSpawn = LoadObject<UBlueprint>(nullptr, *FString(BPName));
            			
			const FVector Location(0.0f,0.0f,1000.0f);
			const FRotator Rotation(0.0f,0.0f,0.0f);
			//FActorSpawnParameters SpawnParameters;
			
			Character = TestWorld->GetWorld()->SpawnActor<AAndroCharacter>(ObjectToSpawn->GeneratedClass, Location, Rotation);
            
			ADD_LATENT_AUTOMATION_COMMAND(FEngineWaitLatentCommand(10.0f));
			
			
            
			return TestNotNull("Player is spawned",Character);//true;
		});

Crash memory damp and logs show only Access Violation to UWorld() as Context to load level

Also, some simple tests without gameplay- like this, work fine on the devices

#include "HelloWorld_SpecsStyle.h"
#include "CoreMinimal.h"
#include "Misc/AutomationTest.h"

DEFINE_SPEC(FSpecBooleanEqualiation, "MyOwn.Hello World",
																		EAutomationTestFlags::ApplicationContextMask	|
																		EAutomationTestFlags::ProductFilter				|
																		EAutomationTestFlags::HighPriorityAndAbove		);

void FSpecBooleanEqualiation::Define()
{
	Describe("Testing equaliation of two numbers in  New SPEC API", [this]
	{
		BeforeEach([this]
		{
			
		});
		
		It("Should test biggest number of pair", [this]()
		{
			AddInfo("Testing equality of two numbers");

			struct Dataset
			{
				int32 A;
				int32 B;
				int32 CorrectResult;
			};

			const TArray<Dataset> TestDataset {{1,2,2},
												{-3,-5,-3},
												{0, 1, 1},
												{0,-1,0} };

			for (const auto Data: TestDataset)
			{
				const FString InfoString = FString::Printf(TEXT("expected that in pair of %i and %i correct result will be %i "), Data.A, Data.B, Data.CorrectResult);
				if (TestEqual(InfoString,FMath::Max(Data.A,Data.B), Data.CorrectResult))
				{
					return true;
				}
				else
				{
					return false;
				}
			}

			return true;
		});

		It("", [this]
		{
			return true;
		});

		AfterEach([this]
		{
			
		});
	});
}

So, the problem is actually in the loading level into autotest. Also, some standard ways like UGameplayStatics::OpenLevel() don’t work in Automation tests
Does someone have any idea what’s wrong and how to fix it?

2

Unfortunately I don’t have any answers (although you probably want to look in the logs, or run it in debugger, if possible to find out why it’s crashing) … but I do have questions. From the documentation, I didn’t see an obvious way to use the regular unit testing functionality on a device outside of editor. Could you tell me how you go about performing a test on device like that?

3

Hello
Sorry, I didn’t see this reply
The whole process looks like this:
You build and package your build on the target platform via this section
image
For Desktop, your OS version- works natively, for mobile- check this guide: Setting Up an Unreal Engine Project for Mobile Platforms | Unreal Engine 5.0 Documentation
And then, when it runs on your platform, check Unreal Frontend, which you use for tests- your device and game sessions will be shown in the “My Session” tab. You could choose the session, and device that you need and run the tests that you need

image
image860×215 34.5 KB

In that case, it will be run exactly on your target platform, and you will see test processing on your phone, or wherever you run it

4

UPD for the main topic:
I found strange things. In the source code, I see that function

AutomationOpenMap();

contains string

ADD_LATENT_AUTOMATION_COMMAND(FWaitForMapToLoadCommand());

That must mean that after calling of that function, the whole functionality must be written in latent style, isn’t it?
But it didn’t work with SpecStyle (which whole written on latent commands)…but, it works, works correctly with ordinary old tests style API, which doesn’t use latent commands:

IMPLEMENT_SIMPLE_AUTOMATION_TEST(FTestOpenMap, "Autotests.1 Testing to load level",																		EAutomationTestFlags::ApplicationContextMask | EAutomationTestFlags::ProductFilter);

And I don’t find a reason for that behavior or how to use it n Spec Style yet