Automated Tests are Broken? (UE 5.3)

[Facer144]

In unreal engine 5.1 there was a latent automated test set up in the project designed to run each time the project was built, essentially a unit test. It works perfectly in 5.1 and follows the small pages of the documentation related to automated tests.

It uses the LatentIt() function with a lamda callback. The lambda broadcast’s it’s completion using the FDoneDelegate::Execute() function and it also receives an FDoneDelegate object as a parameter.

After migrating to 5.3, several issues were fixed except that the unit tests always timed out.
By placing a breakpoint in the FDoneDelegate::Execute() function, I have found that there is a class that periodically checks for the completion of the test called FUntilDoneLatentCommand and there was a member function that sets the test as done:

void Done() { if (bIsRunning) { bDone = true; } }

The thing that catch my attention was that bIsRunning was never true, and the test was never marked with bDone = true, eventually timing out.

So I checked the 5.1 source code for the same class and indeed they differ:

5.3 (This is an engine header file, I’ve only added comments. Censored part of the comments is nothing relevant)

5.1

In 5.1 there is no check in order to set the test as “done”.
In 5.3, the Predicate variable runs the Done function, but bIsRunning is false (it’s only set to true after the line where Done() gets called) so it won’t set the test as done.

Then the Update() function gets called again, but bIsRunning is true, so it won’t enter the scope of the if() statement, where the Done() function is called, so it won’t ever be able to set the test as Done, eventually timing out.

Checklist:

• bDone is a private variable only accessed in these two functions.
• Done() function is a private function only called by update.
• Verified the change notes and nothing contained information about changes in the Lantent Automated Tests.
• Searched google for people with similar problems, nothing was found (I guess automated tests are not used much)
• Make a dummy test where the lambda instantly broadcasts the FDoneDelegate (didn’t work)

I’m guessing this is an engine bug, let me know if I am missing anything or if you have more information about this.

[MichaelBCG]

I’ve experienced the same issue with UE5.3
What I’ve ended up with is using the latent functions (LatentBeforeEach, LatentBeforeEach and LatentAfterEach) with EAsyncExecution parameter. This way it creates a properly working FAsyncUntilDoneLatentCommand object instead of (presumably faulty) FUntilDoneLatentCommand object.

For example, like this:

            LatentBeforeEach(
                EAsyncExecution::ThreadPool,
                [this](const FDoneDelegate& TestDone)
                {
                    AsyncTask(ENamedThreads::GameThread,
                        [&]()
                        {
							// beautiful code 
                        });

                    TestDone.Execute();
                });

[Cpt_Marsh]

I want to follow up on this, I have a related question that no one knows the answer to.
I can’t get LatentIts to work with Tick. Here’s an example:

LatentIt("should tick", EAsyncExecution::ThreadPool, [this](const FDoneDelegate& Done)
  {
    AsyncTask(ENamedThreads::GameThread, [&]()
    {
      World->GetTimerManager().SetTimerForNextTick([&]()
      {
        TestTrue("Test ticked", true);
        Done.Execute();
      });
    });
  });

Where World is:

UWorld* World = UWorld::CreateWorld(EWorldType::Game, false, "Hello UWorld");

The lambda never gets called, no matter what I do.

Any ideas?
Thanks!

[Facer144]

I’m going to guess it might be due to registration of the world object within the engine. I know there’s a flag to notify the engine of the world. Or maybe you could try to register that directly within the FTSTicker class.

Another thing preventing all that from working is that the FTSTicker might not be instanced in the AutomatedTest, you’d have to do it manually but it might be hard to figure out the steps to get it working properly.

I’m also not entirely certain if the LatentTests just put the main thread to sleep until it receives a value from another thread or they use another method that voids the requirement for ticking.

[Facer144]

I haven’t tried it but thanks! I’m not going to mark it as a solution because it’s more of a workaround rather than a fix, hopefully they will see this and fix it in 5.4

[sg-mg-42]

Is there no other solution? The above does not seem to work for me as it crashes when attempting to call Execute() on the FDoneDelegate.

		LatentIt("Waits for success message", FTimespan::FromSeconds(1), [&](const FDoneDelegate& MessageDone)
		{
			ConnectSuccessHandle = MessageSubsystem->RegisterListener<FOnConnectToMasterServerSuccessMessage>(
				TAG_OnConnectToMasterServerSuccess,
				[this, &MessageDone](FGameplayTag GameplayTag, FOnConnectToMasterServerSuccessMessage Message)
				{
					UE_LOG(LogTemp, Display, TEXT("[][] GOOD [][]"));
					TestTrue(TEXT("Should be connected to Master Server"), UEEBlueprintFunctionLibrary::IsMasterServerConnected(World));
					MessageDone.Execute();
				}
			);

			EEWrapper->ConnectToMasterServer("127.0.0.1", 12345);
		});

The listener handler does infact execute and print the log message, however the test still fails due to “Latent command timeout”. Changing the timeout does not change the result.

My project is currently on UE5.3.2 where it is broken.

I tested UE5.4.3 and it is still broken there.

I also tried UE5.1.1 and it works fine there.

Shame because this would be extremely helpful but we can’t change our project version.

[sg-mg-42]

I can’t seem to edit my last reply but here are the links to the source code where the originally-mentioned issue lies.

Working Versions:

• 5.1

• 5.2

Broken Versions:

• 5.3

• 5.4

I submitted a bug ticket for the engine. I will also probably submit a merge request once the ticket is officially opened.

[Elewyth]

Hi, I’ve just run into the same problem, and wanted to share my solution:

Apparently, the World does not tick during AutomationTests. This means that the TimerManager does not tick either, so the Timers obviously do not advance. My solution to this issue is to manually tick the world in my test:

GFrameCounter++; // artificially increase the frame counter
GetWorld()->Tick(LEVELTICK_All, DeltaTime); // tick the world (and anything in it)
// assert result of "async" task

Incrementing the GFrameCounter is important, because the TimerManager checks that it does not tick multiple times within the same frame. By incrementing the frame counter, the TimerManager can then actually tick multiple times.

Note that the very first tick on the TickManager will not actually execute your task (even if start delay is set to 0). Instead, the first tick is always used to move the task from status “Pending” to “Active”. Only on the following tick will the TickManager then process “Active” tasks. So you will likely need to call Tick on the world at least twice. On the second tick, you can specify an arbitrarily large DeltaTime, and your task might then be called multiple times, if it has a lower rate and is configured to loop.

The next important thing to realize, is that manually calling Tick() will (synchronously!) call the TimerManager and have it process any submitted tasks. This means we actually DON’T want to use LatentIt. You can simply do your assertions right after the Tick() call in a regular It!

Hope this helps :).