Automated Tests are Broken? (UE 5.3)

[Facer144]

In unreal engine 5.1 there was a latent automated test set up in the project designed to run each time the project was built, essentially a unit test. It works perfectly in 5.1 and follows the small pages of the documentation related to automated tests.

It uses the LatentIt() function with a lamda callback. The lambda broadcast’s it’s completion using the FDoneDelegate::Execute() function and it also receives an FDoneDelegate object as a parameter.

After migrating to 5.3, several issues were fixed except that the unit tests always timed out.
By placing a breakpoint in the FDoneDelegate::Execute() function, I have found that there is a class that periodically checks for the completion of the test called FUntilDoneLatentCommand and there was a member function that sets the test as done:

void Done() { if (bIsRunning) { bDone = true; } }

The thing that catch my attention was that bIsRunning was never true, and the test was never marked with bDone = true, eventually timing out.

So I checked the 5.1 source code for the same class and indeed they differ:

5.3 (This is an engine header file, I’ve only added comments. Censored part of the comments is nothing relevant)

5.1

In 5.1 there is no check in order to set the test as “done”.
In 5.3, the Predicate variable runs the Done function, but bIsRunning is false (it’s only set to true after the line where Done() gets called) so it won’t set the test as done.

Then the Update() function gets called again, but bIsRunning is true, so it won’t enter the scope of the if() statement, where the Done() function is called, so it won’t ever be able to set the test as Done, eventually timing out.

Checklist:

• bDone is a private variable only accessed in these two functions.
• Done() function is a private function only called by update.
• Verified the change notes and nothing contained information about changes in the Lantent Automated Tests.
• Searched google for people with similar problems, nothing was found (I guess automated tests are not used much)
• Make a dummy test where the lambda instantly broadcasts the FDoneDelegate (didn’t work)

I’m guessing this is an engine bug, let me know if I am missing anything or if you have more information about this.

[MichaelBCG]

I’ve experienced the same issue with UE5.3
What I’ve ended up with is using the latent functions (LatentBeforeEach, LatentBeforeEach and LatentAfterEach) with EAsyncExecution parameter. This way it creates a properly working FAsyncUntilDoneLatentCommand object instead of (presumably faulty) FUntilDoneLatentCommand object.

For example, like this:

            LatentBeforeEach(
                EAsyncExecution::ThreadPool,
                [this](const FDoneDelegate& TestDone)
                {
                    AsyncTask(ENamedThreads::GameThread,
                        [&]()
                        {
							// beautiful code 
                        });

                    TestDone.Execute();
                });

[Cpt_Marsh]

I want to follow up on this, I have a related question that no one knows the answer to.
I can’t get LatentIts to work with Tick. Here’s an example:

LatentIt("should tick", EAsyncExecution::ThreadPool, [this](const FDoneDelegate& Done)
  {
    AsyncTask(ENamedThreads::GameThread, [&]()
    {
      World->GetTimerManager().SetTimerForNextTick([&]()
      {
        TestTrue("Test ticked", true);
        Done.Execute();
      });
    });
  });

Where World is:

UWorld* World = UWorld::CreateWorld(EWorldType::Game, false, "Hello UWorld");

The lambda never gets called, no matter what I do.

Any ideas?
Thanks!

[Facer144]

I’m going to guess it might be due to registration of the world object within the engine. I know there’s a flag to notify the engine of the world. Or maybe you could try to register that directly within the FTSTicker class.

Another thing preventing all that from working is that the FTSTicker might not be instanced in the AutomatedTest, you’d have to do it manually but it might be hard to figure out the steps to get it working properly.

I’m also not entirely certain if the LatentTests just put the main thread to sleep until it receives a value from another thread or they use another method that voids the requirement for ticking.

[Facer144]

I haven’t tried it but thanks! I’m not going to mark it as a solution because it’s more of a workaround rather than a fix, hopefully they will see this and fix it in 5.4