We’re curious how this is intended to be handled, or what Epic’s best practises around this are. Mindful that the Fortnite workflow may actual be different.
When using functional tests, especially when running them in cooked/packaged builds (consoles, multiplayer environments, etc) there is a requirement to cook and package the test assets (maps, custom blueprints, code, etc). Preferably this data would not make it into a shipping build, but the obvious ways of attempting this means you either end up cooking a different flavour of build for shipping, making the tests potentially nonsensical, or you actually just ship the test assets (not great, noticed Fortnite has a few that sneak in but definitely not all testing assets), or you use something like modular gameplay/separately packaged plugins so you can just omit the packages with test assets selectively without having to redo the rest of the build.
Any recommendations/pointers as to what Epic’s thinking is here?
Blueprint functional tests are meant to test Blueprint components.
The Screenshot BP functional tests are meant to test Rendering features.
One could test maps by loading maps through another map and have the BP functional tests do specialize testing on the loaded map. But there is no pipeline setup for this in open source code base.
FN tests is mostly using the Replay pipeline which is linked to an in-house performance tracking system.
The LLTs can be compiled independently of the Game App as long as they are not tightly coupled with the Engine systems.
We do recommend tests to be separated in their own plugins. It is easier to manage at cook time. Also related assets should be part of the plugins, it is easier to manage inclusion of said test assets.
The main way is through plugins enabling. Plugins can be opt in or out at the cook command. Also plugins can be packaged separately and loaded at the app command line.
For FN we have a few old replays that are used to consistency, but we frequently rotate what is being use for replay testing based on features and season releases.
In our case we are using functional test to actual test gameplay logic, e.g. players spawning and being able to pick up items, navigate to certain points in the world, that kind of stuff. We tend to have custom test maps for this when not utilising actual game maps.
It’s these custom test maps, and related blueprints for functional tests, that we would rather not make into the shipping version of the game. When you say to put these in plugins and then “it is easier to manage inclusion of said test assets” - what are the recommended ways to manage this inclusion? They are setup to be packaged separately and then in the shipping build some packages are just omitted?
Curious btw, for Fortnite, how do the replays get generated? Do they regularly get regenerated/re-recorded?