The issue turned out to be a bug in one of the tests that was hidden by a bug in the automation framework.
The test bug was that one of the completion paths did not end in a “Finish Test” node. So the test never completed. This was hidden by the fact that the all tests were showing as having succeed. Basically, the automation framework can show a test as successful even when the test is cancelled by closing the game preview window before the test is marked as finished.
One way to avoid running into this issue is to make sure the timeouts on all tests are reasonable. (The default timeout is 60 seconds.) Then always wait for the timeout so you don’t risk triggering the framework bug.