I was hoping to get some guidance on the Automated Performance Tests plugin. The plugin is clearly geared towards Tech Art performance analysis and less about the overall game thread performance. Is the expectation that this will grow into a more general performance testing suite or should we still be building that out with gauntlet for setting up tests representing the player?
Also it warns about being experimental a lot, I assume this is just the framework portion, but all the stat capturing and analysis is done using the current and stable stat systems?
Hey there! Sorry for the delay getting back to you. I wanted to approach this from a few angles:
Experimental
The framework ties together a lot of the disparate systems necessary for truly automated performance testing. BuildGraph, UAT, Gauntlet, CSV Profiler and PerfReportTool, Unreal Insights. Those individual systems aren’t themselves experimental. What’s experimental is the glue holding it all together, which are the BuildGraph includes, the UnrealTestNodes to manage the sessions, and the Gauntlet Test Controllers provided in the plugin itself.
Target Audience
The framework will gather both CSV Profiler and Insights traces (depending on the commandline arguments) and those are included in the artifacts of a run just like any other Gauntlet test. It’ll gather both GPU and CPU stats, and in fact we used the results of one of our tests to catch a small CPU performance regression before 5.6 released!
What’s Getting Tested
This is the part where I’d tend to agree with you that the things that get tested are, by and large, things that tech artists will care about for GPU performance. Static Camera and Sequencer tests out of the box will likely be focusing on the environment. That said, CitySample uses the Sequencer test controller and that’s where we saw that CPU regression. It’s really up to you what you point the camera at.
The ReplayRun test controller is more well suited to testing gameplay, but only if your game is set up for replication. Otherwise it’d be more of a challenge. We’ve heard of some studios who’ve built a whole data driven automated game playing framework which plays their game nightly and they gather the performance results from that. We’ve discussed some options for this, but that’s as far as we’ve gotten.
I’d expect in your situation you’ll likely want to explore building your own test controllers to drive a representative playthrough. However, we also haven’t spent much time exploring how project-specific test controllers would integrate into the framework. In an earlier phase it was a little easier, but it also required quite a bit more boilerplate. If you’re keen on leveraging the framework to handle most of the glue, I’d be really interested to get your feedback and questions on what it’d look like to add a project-specific controller!
Yes indeed, there’s still some open questions about the “Epic Way” of visualizing the results of performance tests, but as Julian mentioned we’ve got some ideas for how to do that directly in Horde.
Please feel free to email me directly if you’ve got feedback or questions as you’re looking to extend the framework with additional test controllers: [Content removed]
Actually we did end up hitting an issue with tool once we started to get it running in the cloud. I’ve opened a new question for it, but it seems like the horde reporting is currently reliant on some internal tooling still?
Ya, being able to store the data in horde would be nice as we already send our telemetry data there. I’ll reach out once we’ve used more thoroughly, but seems like a good starting point.