Multiple issues running complete unit test suite

Development Pipeline & Plugins
, ,
1

Hello!

In our CI pipeline we use running complete unit test suite (all available unit tests, see repro steps). We use the entire suite as a safeguard because of the modification we do occasionally.

We noticed that release versions of UE5.6 and UE5.7 (without any modifications) have issues running all unit tests. There are multiple issues, involving crashes and failed tests. Prominent examples are crashing tests from ShaderPreprocessorTests.cpp (both UE5.6 and 5.7) failing TestFramework.Validation.TestInequalityStringNulls (UE5.7) or failing some of the CoreUObject::NativeInstancedSubobjects test. There are quite a few more failing/crashing tests. I’m assuming some of the tests failing/crashing in unattended commandline run won’t be failing when executed from editor frontend, but at this moment I don’t believe there will be many of such cases.

Due to the fact that there are failing tests in release versions, I came to the conclusion that theres a gap in my understanding of how Epic incorporate and handle the test suite - I would assume that there wouldn’t be failing tests in release versions of the engine. I would also assume that Epic incorporate running complete test suite in it’s CI it order to detect and fix regressions.

In this regard, I have two questions:

  1. Does Epic use some sort of a standard filter for subset of executed test on daily basis?
  2. In your opinion, should we be worried that some seemingly core unit tests are failing in clean release version and pursue a cherrypick fix from the main branch (//UE5/Main)? Or do you think this issue can be safely skipped?

Best regards

[Attachment Removed]

2

Steps to Reproduce

  1. Download //UE5/Release-5.7.4 sources from Perforce repository.
  2. Compile the engine.
  3. Create an empty project on this engine.
  4. Run complete unit test suite in commandline. Example: “Engine\Binaries\Win64\UnrealEditor-Cmd.exe E:\proj\Sandbox\Sandbox\Sandbox.uproject -ExecCmds=“Automation RunAll” -TestExit=“Automation Test Queue Empty” -Unattended -NoPause -NullRHI -NoSplash -NoSound -Log -ReportExportPath=“E:\Report””
    [Attachment Removed]
3

Hello!

You should expect all tests to be successful in the released versions. Any test failing is a bug. I can’t go into details but we do have some internal factors that can hide some failures,

Do not hesitate to report tests that are failing. They will be logged as bugs and will get proper attention from the development team.

I recommend that you remove some of the arguments from the command line you shared (-NullRHI & -NoSound). Some tests will fail with those as they are relying on some systems that are turned off. If you want the tests to run in a headless mode, you should prefer the -RenderOffscreen argument. Unlike -nullrhi which disables rendering entirely, -RenderOffscreen still renders but without displaying a visible window so tests that rely on rendering will not fail.

I reached out to our release management team to discuss how we can improve our processes to prevent shipping with failing tests in future release.

Regards,

Martin

[Attachment Removed]