# Automated Tests are Broken? (UE 5.3) **URL:** \ **Category:** Programming & Scripting\ **Tags:** bug, automated-testing, unreal-engine, Automation, bug-report, CPP, question\ **Created:** [November 7, 2023, 3:33pm UTC](https://forums.unrealengine.com/t/automated-tests-are-broken-ue-5-3/1371033 "2023-11-07T15:33:12Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Facer144](https://d3kjluh73b9h9o.cloudfront.net/original/4X/2/8/5/28502927f1d97ba1051b3b47ccfe17233bf41ef0.png) [@Facer144](https://forums.unrealengine.com/u/Facer144)\ **Post date:** [November 7, 2023, 3:33pm UTC](https://forums.unrealengine.com/t/automated-tests-are-broken-ue-5-3/1371033/1 "2023-11-07T15:33:12Z")
In unreal engine 5.1 there was a latent automated test set up in the project designed to run each time the project was built, essentially a unit test. It works perfectly in 5.1 and follows the small pages of the documentation related to automated tests. It uses the `LatentIt()` function with a lamda callback. The lambda broadcast’s it’s completion using the `FDoneDelegate::Execute()` function and it also receives an `FDoneDelegate` object as a parameter. After migrating to 5.3, several issues were fixed except that the unit tests always timed out. By placing a breakpoint in the `FDoneDelegate::Execute()` function, I have found that there is a class that periodically checks for the completion of the test called `FUntilDoneLatentCommand` and there was a member function that sets the test as done: ` void Done() { if (bIsRunning) { bDone = true; } } ` The thing that catch my attention was that bIsRunning was never true, and the test was never marked with bDone = true, eventually timing out. So I checked the 5.1 source code for the same class and indeed they differ: ![AutomationTests_ue53.png](https://d3kjluh73b9h9o.cloudfront.net/original/4X/6/d/f/6df0263a2b693c503b206a402ae4611441f99750.png) **5.3** (This is an engine header file, I’ve only added comments. Censored part of the comments is nothing relevant) ![AutomationTests_ue51](https://d3kjluh73b9h9o.cloudfront.net/original/4X/0/9/4/094094ec29fd53b44efbd1e85379db2545a453d6.png) **5.1** In 5.1 there is no check in order to set the test as “done”. In 5.3, the `Predicate` variable runs the Done function, but bIsRunning is false (it’s only set to true after the line where `Done()` gets called) so it won’t set the test as done. Then the `Update()` function gets called again, but `bIsRunning` is true, so it won’t enter the scope of the `if() statement`, where the `Done()` function is called, so it won’t ever be able to set the test as Done, eventually timing out. Checklist: - `bDone` is a private variable only accessed in these two functions. - `Done()` function is a private function only called by update. - Verified the change notes and nothing contained information about changes in the Lantent Automated Tests. - Searched google for people with similar problems, nothing was found (I guess automated tests are not used much) - Make a dummy test where the lambda instantly broadcasts the `FDoneDelegate` (didn’t work) I’m guessing this is an engine bug, let me know if I am missing anything or if you have more information about this. ---
**Author:** ![MichaelBCG](https://d1ap1mz92jnks1.cloudfront.net/user_avatar/forums.unrealengine.com/michaelbcg/32/903972_2.png) [@MichaelBCG](https://forums.unrealengine.com/u/MichaelBCG)\ **Post date:** [January 21, 2024, 12:21am UTC](https://forums.unrealengine.com/t/automated-tests-are-broken-ue-5-3/1371033/2 "2024-01-21T00:21:51Z")
I’ve experienced the same issue with UE5.3 What I’ve ended up with is using the latent functions (_LatentBeforeEach_, _LatentBeforeEach_ and _LatentAfterEach_) with _EAsyncExecution_ parameter. This way it creates a properly working _FAsyncUntilDoneLatentCommand_ object instead of (presumably faulty) _FUntilDoneLatentCommand_ object. For example, like this: ```auto LatentBeforeEach( EAsyncExecution::ThreadPool, [this](const FDoneDelegate& TestDone) { AsyncTask(ENamedThreads::GameThread, [&]() { // beautiful code }); TestDone.Execute(); }); ``` ---
**Author:** ![Cpt\_Marsh](https://d3kjluh73b9h9o.cloudfront.net/original/4X/2/8/5/28502927f1d97ba1051b3b47ccfe17233bf41ef0.png) [@Cpt\_Marsh](https://forums.unrealengine.com/u/Cpt_Marsh)\ **Post date:** [February 17, 2024, 10:27am UTC](https://forums.unrealengine.com/t/automated-tests-are-broken-ue-5-3/1371033/3 "2024-02-17T10:27:19Z")
I want to follow up on this, I have a related question that no one knows the answer to. I can’t get `LatentIt`s to work with `Tick`. Here’s an example: ```auto LatentIt("should tick", EAsyncExecution::ThreadPool, [this](const FDoneDelegate& Done) { AsyncTask(ENamedThreads::GameThread, [&]() { World->GetTimerManager().SetTimerForNextTick([&]() { TestTrue("Test ticked", true); Done.Execute(); }); }); }); ``` Where `World` is: ```auto UWorld* World = UWorld::CreateWorld(EWorldType::Game, false, "Hello UWorld"); ``` The lambda never gets called, no matter what I do. Any ideas? Thanks! ---
**Author:** ![Facer144](https://d3kjluh73b9h9o.cloudfront.net/original/4X/2/8/5/28502927f1d97ba1051b3b47ccfe17233bf41ef0.png) [@Facer144](https://forums.unrealengine.com/u/Facer144)\ **Post date:** [February 21, 2024, 1:21pm UTC](https://forums.unrealengine.com/t/automated-tests-are-broken-ue-5-3/1371033/4 "2024-02-21T13:21:09Z")
I’m going to guess it might be due to registration of the world object within the engine. I know there’s a flag to notify the engine of the world. Or maybe you could try to register that directly within the FTSTicker class. Another thing preventing all that from working is that the FTSTicker might not be instanced in the AutomatedTest, you’d have to do it manually but it might be hard to figure out the steps to get it working properly. I’m also not entirely certain if the LatentTests just put the main thread to sleep until it receives a value from another thread or they use another method that voids the requirement for ticking. ---
**Author:** ![Facer144](https://d3kjluh73b9h9o.cloudfront.net/original/4X/2/8/5/28502927f1d97ba1051b3b47ccfe17233bf41ef0.png) [@Facer144](https://forums.unrealengine.com/u/Facer144)\ **Post date:** [February 21, 2024, 1:22pm UTC](https://forums.unrealengine.com/t/automated-tests-are-broken-ue-5-3/1371033/5 "2024-02-21T13:22:44Z")
I haven’t tried it but thanks! I’m not going to mark it as a solution because it’s more of a workaround rather than a fix, hopefully they will see this and fix it in 5.4 ---
**Author:** ![sg-mg-42](https://d3kjluh73b9h9o.cloudfront.net/original/4X/2/8/5/28502927f1d97ba1051b3b47ccfe17233bf41ef0.png) [@sg-mg-42](https://forums.unrealengine.com/u/sg-mg-42)\ **Post date:** [July 23, 2024, 3:06am UTC](https://forums.unrealengine.com/t/automated-tests-are-broken-ue-5-3/1371033/6 "2024-07-23T03:06:02Z")
Is there no other solution? The above does not seem to work for me as it crashes when attempting to call `Execute()` on the `FDoneDelegate`. ```auto LatentIt("Waits for success message", FTimespan::FromSeconds(1), [&](const FDoneDelegate& MessageDone) { ConnectSuccessHandle = MessageSubsystem->RegisterListener( TAG_OnConnectToMasterServerSuccess, [this, &MessageDone](FGameplayTag GameplayTag, FOnConnectToMasterServerSuccessMessage Message) { UE_LOG(LogTemp, Display, TEXT("[][] GOOD [][]")); TestTrue(TEXT("Should be connected to Master Server"), UEEBlueprintFunctionLibrary::IsMasterServerConnected(World)); MessageDone.Execute(); } ); EEWrapper->ConnectToMasterServer("127.0.0.1", 12345); }); ``` The listener handler does infact execute and print the log message, however the test still fails due to “Latent command timeout”. Changing the timeout does not change the result. My project is currently on UE5.3.2 where it is broken. I tested UE5.4.3 and it is still broken there. I also tried UE5.1.1 and it works fine there. Shame because this would be extremely helpful but we can’t change our project version. ---
**Author:** ![sg-mg-42](https://d3kjluh73b9h9o.cloudfront.net/original/4X/2/8/5/28502927f1d97ba1051b3b47ccfe17233bf41ef0.png) [@sg-mg-42](https://forums.unrealengine.com/u/sg-mg-42)\ **Post date:** [July 24, 2024, 3:54am UTC](https://forums.unrealengine.com/t/automated-tests-are-broken-ue-5-3/1371033/7 "2024-07-24T03:54:45Z")
I can’t seem to edit my last reply but here are the links to the source code where the originally-mentioned issue lies. Working Versions: - [5.1](https://github.com/EpicGames/UnrealEngine/blob/5.1/Engine/Source/Runtime/Core/Public/Misc/AutomationTest.h#L2159) - [5.2](https://github.com/EpicGames/UnrealEngine/blob/5.2/Engine/Source/Runtime/Core/Public/Misc/AutomationTest.h#L2212) Broken Versions: - [5.3](https://github.com/EpicGames/UnrealEngine/blob/5.3/Engine/Source/Runtime/Core/Public/Misc/AutomationTest.h#L2320) - [5.4](https://github.com/EpicGames/UnrealEngine/blob/5.4/Engine/Source/Runtime/Core/Public/Misc/AutomationTest.h#L2513) I submitted a bug ticket for the engine. I will also probably submit a merge request once the ticket is officially opened. ---
**Author:** ![Elewyth](https://d3kjluh73b9h9o.cloudfront.net/original/4X/2/8/5/28502927f1d97ba1051b3b47ccfe17233bf41ef0.png) [@Elewyth](https://forums.unrealengine.com/u/Elewyth)\ **Post date:** [November 2, 2024, 9:33am UTC](https://forums.unrealengine.com/t/automated-tests-are-broken-ue-5-3/1371033/8 "2024-11-02T09:33:40Z")
Hi, I’ve just run into the same problem, and wanted to share my solution: Apparently, the World does not tick during AutomationTests. This means that the TimerManager does not tick either, so the Timers obviously do not advance. My solution to this issue is to manually tick the world in my test: ```cpp GFrameCounter++; // artificially increase the frame counter GetWorld()->Tick(LEVELTICK_All, DeltaTime); // tick the world (and anything in it) // assert result of "async" task ``` Incrementing the `GFrameCounter` is important, because the TimerManager checks that it does not tick multiple times within the same frame. By incrementing the frame counter, the TimerManager can then actually tick multiple times. Note that the very first tick on the TickManager will not actually execute your task (even if start delay is set to 0). Instead, the first tick is always used to move the task from status “Pending” to “Active”. Only on the following tick will the TickManager then process “Active” tasks. So you will likely need to call Tick on the world at least twice. On the second tick, you can specify an arbitrarily large `DeltaTime`, and your task might then be called multiple times, if it has a lower rate and is configured to loop. The next important thing to realize, is that manually calling `Tick()` will (synchronously!) call the TimerManager and have it process any submitted tasks. This means we actually DON’T want to use `LatentIt`. You can simply do your assertions right after the `Tick()` call in a regular `It`! Hope this helps :).