Hot-reload does not re-compile Automation Tests

I have also tested this in 4.9.

Repro:

  1. Create C++ project
  2. Create a simple automation test as per this gist.
  3. Run the test via the in-editor Frontend and observe the console output.
  4. Change the UE_LOG output to some other phrase.
  5. Hit the ‘compile’ button in-editor.
  6. Run the test again and notice that it has the same output as before the hot-reload.

You can try this with any sort of change if for some reason console outputs aren’t hot-reloaded by default. The results are the same.

This is a huge problem for the speed and practicality of conducting regular unit tests. At the moment, I have to reboot the editor every single time I change a test. It takes too long to be practical or useful.

Thanks!

1 Like

Ahh, sorry Doug! I’ve been looking at the Automation system so much that I had just presumed the universe knew what I was talking about.

That goes into its own .cpp file in the Private/Tests directory, as per [this][1] page. FHotReloadExampleTest was just a name I came up with for this unit test. It should show up in the Frontend → Automation section, with each word in the 2nd parameter of the marco separated by the ‘.’ character being a deeper level in the test hierarchy.

For example, I have the following macro in my Private/Tests/ViewTests.cpp file:

IMPLEMENT_SIMPLE_AUTOMATION_TEST(FViewLocationIsConsistent, "RTG.Time.ViewLocationIsConsistent", (EAutomationTestFlags::ATF_ApplicationMask | EAutomationTestFlags::ATF_SmokeTest))

And this shows up in the Frontend like so:

Hey HateDread-

Where exactly are you adding this code? Is this going into an Actor class or the game mode class or somewhere else? What exactly is FHotReloadExampleTest? Can you provide more information on how you setup your step 2?

Cheers

Doug Wilson

Let me know if there’s anything else you need :slight_smile: I’ve probably glossed over a detail or two!

After further editing the code you provided I was able to get the project to compile and had the same outcome when running the automation test. I have entered a report (UE-25350) for further investigation.

Cheers

Doug Wilson.

Hey HateDread-

It was brought to my attention that any code that is not a UCLASS type, including automation code, is not hot-reloaded. The ticket I entered regarding this issue is still open in case this can be addressed in the future however there is no timeline for when that may be. For now what you’re experiencing is the expected result.

Hi Doug,

I’m a game developer trying to write quality code using automation testing. I’m running into the same issue as HateDread.

This is obviously not a feature, it is bug. The current state of UE4’s automation testing framework makes it exceedingly difficult for game developers to write tested, defect free code. If this is the expected result, please exceed my expectations.

For more information on the problem, see this forum post: Automation Testing Should be on the Roadmap - Feedback for Unreal Engine team - Unreal Engine Forums

Looking forward to hearing you,
Clarke

Hi all,

Thanks for the feedback. The automation systems seem to be an area that could benefit from some additional attention, and I’ll bring up the topic with our engineers soon to discuss any plans.

Cheers

Hi, does this work for you in 4.11.2? This is what I have: gist:e8a6c08940d911f34c2dd440dc98ffbe · GitHub, it’s inside my project’s source/project_name/private/Test folder. It does not show up in frontend… not sure what I’m missing?

This issue also affects UE 4.12.

Haya,
Ran into this problem also and just wanted to add a bit more data that I didn’t notice in the bug report. The tests do recompile, they just don’t get reloaded. not sure if that’s any help at all.

Also just kinda wanted to +1 this bug because it does make establishing new unit tests a little more painful than it needs to be. Not a huge issue, but a big enough issue that it makes me frown a little every time I have to restart to update the session frontend with my newly built tests.

Just in case anyone is looking for a sorta workaround to this: I played around a bit with hot reloading modules that contain tests and while you cannot change an existing test, adding a new one seems to work fine (recompile module from the editor - or, if the test is in your main module, it also works when building from VS - then hit Refresh Tests in the session frontend).

So when you want to change an existing test to make it pass or whatever, you can change its name from Test to something else (I use Test2, Test3, etc.). Unfortunately, the old test class stays around as a ghost until you restart the editor, so it messes up the “Run All tests in a category” functionality. Note that you can have multiple test classes in a single source file so genuinely adding new tests is not a big issue.

While this is obviously not a perfect solution, it’s somewhat workable and is at least better than having to restart the editor all the time. Once in a while, you can clean up all the numbers and just restart the editor.

I found a workaround that allows you to enable hot reloading per test at least. The nature of the bug and the workaround are described in my blog post about it: UE4 Code: Fixing Hot Reload of AutomationTests | Vhite Rabbit Website

and 4.22 :frowning:

If anyone is still looking for a solution, a simple engine change worked for me.

In …/Engine/Source/Runtime/Core/Private/Misc/AutomationTest.cpp, in the FAutomationTestFramework::RegisterAutomationTes(…) function, you just have to make AutomationTestClassNameToInstanceMap be updated with the new FAutomationTestBase* pointer even if an entry exists and return false to tell the calling function that it’s always a new test. I.e. change to this

bool FAutomationTestFramework::RegisterAutomationTest( const FString& InTestNameToRegister, class FAutomationTestBase* InTestToRegister )
{
	const bool bAlreadyRegistered = AutomationTestClassNameToInstanceMap.Contains( InTestNameToRegister );
	if ( !bAlreadyRegistered )
	{
		AutomationTestClassNameToInstanceMap.Add( InTestNameToRegister, InTestToRegister );
	}
    else
    {
        AutomationTestClassNameToInstanceMap[InTestNameToRegister] = InTestToRegister;
    }

    return false;
}

If you’ve put your tests in a separate module, you’ll probably also have to put a UCLASS() somewhere as Squareys described.

I cant believe that you have to build the engine from scratch to have the hot reloading working :confused:

For UE 4.27:

I got hot reloading for tests working without extra code.

The solution is to

  1. Use live coding instead of hot reload
  2. Not inline tests in header files

Here is my test code:

Test.h

#pragma once

#include "CoreMinimal.h"
#include "FunctionalTest.h"

IMPLEMENT_SIMPLE_AUTOMATION_TEST(FMirrorStringTest, "MyGame.MirrorString Test",EAutomationTestFlags::EditorContext | EAutomationTestFlags::EngineFilter)

Test.cpp

#include "Test.h"

bool FMirrorStringTest::RunTest(const FString& Parameters)
{
	return true;
}