When running AutomatedPerfTest.SequenceTest on a project with multiple map+sequence combinations without specifying one combination, APT runs all the combinations defined in the projects settings. When the CSVOutputMode is set to `Separate` (the default), it writes out one CSV per combination.
When the tests end, AutomatedPerfTestNode copies the files from the GauntletTemp artifacts dir to the output dir in CopyPerfFilesToOutputDir. This function has some logic to only copy out the latest CSV written, which is meant to avoid copying files from previous iterations. When every iteration outputs multiple CSVs this logic fails to copy all the relevant files. This is repro’d on 5.6.1 but latest ue5-main seems to have the same logic.
Is this an oversight or by design, for example am I supposed/encouraged to only run one sequence per `RunUnreal` execution rather than all together?
I have also tried to switch to the `Single` CSVOutputMode but it doesn’t seem to work properly for SequenceTest at least - it outputs a single file with incomplete data (2 out of 3 sequences in my test) and no parseable metadata.
this is indeed an oversight, perhaps because we don’t run all of the sequences in one invocation (this is how uploading to our report server works). I filed a JIRA ticket (UE-348061) to look into this. As for whether or not to run them all at once - I’d be wary that 2nd and later sequences may have a different LLM / Memory stats than when run standalone. You may want to check that and see if that discrepancy is something you don’t care about.
Agreed that side effects can be an issue when running them all at once, but there are still legitimate cases for it I believe. E.g. if only looking at perf/framerate I’d imagine that results would be similar.
Most likely yes, but in the presence of various streaming systems that retain state and caches (e.g. most notably PSO cache) even that statement is hard to ascertain. But I absolutely agree that it might be not something that matters for a specific use case or metric, so it’s a valid mode to support.
I have tried to check the status for UE-348061 on https://issues.unrealengine.com/ but couldn’t find any match. Could you confirm that the ID is correct? Or is the issue not on the tracker for some reason?