Hot-reload does not re-compile Automation Tests

HateDread · January 12, 2016, 5:51pm

I have also tested this in 4.9.

Repro:

Create C++ project
Create a simple automation test as per this gist.
Run the test via the in-editor Frontend and observe the console output.
Change the UE_LOG output to some other phrase.
Hit the ‘compile’ button in-editor.
Run the test again and notice that it has the same output as before the hot-reload.

You can try this with any sort of change if for some reason console outputs aren’t hot-reloaded by default. The results are the same.

This is a huge problem for the speed and practicality of conducting regular unit tests. At the moment, I have to reboot the editor every single time I change a test. It takes too long to be practical or useful.

Thanks!

HateDread · January 13, 2016, 2:42pm

Ahh, sorry ! I’ve been looking at the Automation system so much that I had just presumed the universe knew what I was talking about.

That goes into its own .cpp file in the Private/Tests directory, as per [this][1] page. FHotReloadExampleTest was just a name I came up with for this unit test. It should show up in the Frontend → Automation section, with each word in the 2nd parameter of the marco separated by the ‘.’ character being a deeper level in the test hierarchy.

For example, I have the following macro in my Private/Tests/ViewTests.cpp file:

IMPLEMENT_SIMPLE_AUTOMATION_TEST(FViewLocationIsConsistent, "RTG.Time.ViewLocationIsConsistent", (EAutomationTestFlags::ATF_ApplicationMask | EAutomationTestFlags::ATF_SmokeTest))

And this shows up in the Frontend like so:

anonymous_user_2b03d27c · January 13, 2016, 2:36pm

Hey -

Where exactly are you adding this code? Is this going into an Actor class or the game mode class or somewhere else? What exactly is FHotReloadExampleTest? Can you provide more information on how you setup your step 2?

Cheers

HateDread · January 13, 2016, 2:56pm

Let me know if there’s anything else you need I’ve probably glossed over a detail or two!

anonymous_user_2b03d27c · January 13, 2016, 3:05pm

After further editing the code you provided I was able to get the project to compile and had the same outcome when running the automation test. I have entered a report (UE-25350) for further investigation.

Cheers

.

anonymous_user_2b03d27c · January 19, 2016, 7:19pm

Hey -

It was brought to my attention that any code that is not a UCLASS type, including automation code, is not hot-reloaded. The ticket I entered regarding this issue is still open in case this can be addressed in the future however there is no timeline for when that may be. For now what you’re experiencing is the expected result.

anonymous_user_7e341a86 · March 9, 2016, 8:55pm

Hi ,

I’m a game developer trying to write quality code using automation testing. I’m running into the same issue as .

This is obviously not a feature, it is bug. The current state of UE4’s automation testing framework makes it exceedingly difficult for game developers to write tested, defect free code. If this is the expected result, please exceed my expectations.

For more information on the problem, see this forum post: Automation Testing Should be on the Roadmap - Feedback & Requests - Epic Developer Community Forums

Looking forward to hearing you,

Stephen_Ellis · March 21, 2016, 3:56pm

Hi all,

Thanks for the feedback. The automation systems seem to be an area that could benefit from some additional attention, and I’ll bring up the topic with our engineers soon to discuss any plans.

Cheers

anonymous_user_c983727a · April 28, 2016, 2:41am

Hi, does this work for you in 4.11.2? This is what I have: , it’s inside my project’s source/project_name/private/Test folder. It does not show up in frontend… not sure what I’m missing?

anonymous_user_704f7aff · August 13, 2016, 3:26pm

This issue also affects UE 4.12.

anonymous_user_33e74471 · December 13, 2016, 5:24pm

Haya,
Ran into this problem also and just wanted to add a bit more data that I didn’t notice in the bug report. The tests do recompile, they just don’t get reloaded. not sure if that’s any help at all.

Also just kinda wanted to +1 this bug because it does make establishing new unit tests a little more painful than it needs to be. Not a huge issue, but a big enough issue that it makes me frown a little every time I have to restart to update the session frontend with my newly built tests.

anonymous_user_7615687d1 · June 28, 2017, 6:37pm

Just in case anyone is looking for a sorta workaround to this: I played around a bit with hot reloading modules that contain tests and while you cannot change an existing test, adding a new one seems to work fine (recompile module from the editor - or, if the test is in your main module, it also works when building from VS - then hit Refresh Tests in the session frontend).

So when you want to change an existing test to make it pass or whatever, you can change its name from Test to something else (I use Test2, Test3, etc.). Unfortunately, the old test class stays around as a ghost until you restart the editor, so it messes up the “Run All tests in a category” functionality. Note that you can have multiple test classes in a single source file so genuinely adding new tests is not a big issue.

While this is obviously not a perfect solution, it’s somewhat workable and is at least better than having to restart the editor all the time. Once in a while, you can clean up all the numbers and just restart the editor.

Squareys · March 2, 2018, 7:46pm

I found a workaround that allows you to enable hot reloading per test at least. The nature of the bug and the workaround are described in my blog post about it: UE4 Code: Fixing Hot Reload of AutomationTests | Vhite Rabbit Website

jimmyt1988 · April 16, 2019, 7:32pm

and 4.22

PygmyShrew127 · August 6, 2019, 8:41pm

If anyone is still looking for a solution, a simple engine change worked for me.

In …/Engine/Source/Runtime/Core/Private/Misc/AutomationTest.cpp, in the FAutomationTestFramework::RegisterAutomationTes(…) function, you just have to make AutomationTestClassNameToInstanceMap be updated with the new FAutomationTestBase* pointer even if an entry exists and return false to tell the calling function that it’s always a new test. I.e. change to this

bool FAutomationTestFramework::RegisterAutomationTest( const FString& InTestNameToRegister, class FAutomationTestBase* InTestToRegister )
{
	const bool bAlreadyRegistered = AutomationTestClassNameToInstanceMap.Contains( InTestNameToRegister );
	if ( !bAlreadyRegistered )
	{
		AutomationTestClassNameToInstanceMap.Add( InTestNameToRegister, InTestToRegister );
	}
    else
    {
        AutomationTestClassNameToInstanceMap[InTestNameToRegister] = InTestToRegister;
    }

    return false;
}

If you’ve put your tests in a separate module, you’ll probably also have to put a UCLASS() somewhere as described.

jmgomez.me · June 26, 2020, 9:48pm

I cant believe that you have to build the engine from scratch to have the hot reloading working

huehuehue · November 12, 2022, 7:11pm

For UE 4.27:

I got hot reloading for tests working without extra code.

The solution is to

Use live coding instead of hot reload
Not inline tests in header files

Here is my test code:

Test.h

#pragma once

#include "CoreMinimal.h"
#include "FunctionalTest.h"

IMPLEMENT_SIMPLE_AUTOMATION_TEST(FMirrorStringTest, "MyGame.MirrorString Test",EAutomationTestFlags::EditorContext | EAutomationTestFlags::EngineFilter)

Test.cpp

#include "Test.h"

bool FMirrorStringTest::RunTest(const FString& Parameters)
{
	return true;
}

NorthWeapon · March 25, 2023, 12:41pm

Confirmed working for IMPLEMENT_SIMPLE_AUTOMATION_TEST when doing Live Coding.

This does not work for the alternative automation test DEFINE_SPEC, however. If you are using the new Automation Spec way of creating unit tests with Define Describe and It, then live coding refreshing the frontend does not update the unit test.

It is upsetting, how despite 8 years of requesting Epic to add more unit testing support, that we still have bugs for basic unit testing functionality (not being able to run unit tests quickly!)