In our game, we have a workflow where we use MRQ to render out cinematics, and then pass these renders to audio designers to make STEMs. We then bring these STEMs back into Sequencer and use that for the final game audio.
We noticed that a cinematic with a lot of time dilation in it got its audio out of sync very quickly, and it got worse and worse as the cinematic played on.
After eliminating a bunch of technical reasons for this, I compared the MRQ render used by the audio team to a gameplay recording of me playing that same cinematic. I lined up both videos in a video editing program, ensuring that they started in sync, and noticed that the sections slowed down via time dilation lasted longer in the render than they did during the gameplay. We tried tweaking various settings and things, but couldn’t get the length of the runtime cinematic to match the length of the rendered version. In the end, the audio team re-conformed the audio to my playthrough video, and this ended up working perfectly. However, this is NOT an ideal workflow, and we’d like to figure out how to get the MRQ renders to be the correct length.
Some notes:
We discovered this in 5.5, but I did the same test with 5.6 this morning and the issue is still occurring. It was actually slightly worse in 5.6 -- the slo-mo shots in the MRQ version were a couple of frames longer in 5.6 than they were in 5.5.
We are using the workflow where we have a master scene, and every shot is in it is own subsequence, so all the time dilation is nested. However, during our testing we had a poor cinematic artist pull everything out the subsequences and into the master sequence, and we got the same result. (I probably owe the guy a beverage.) This test was done in 5.5.
Audio is not supported with Time Dilation in general and should not be used as a reliable ground truth. The audio will keep playing at speed no matter the time dialition in Sequencer. To be perfectly honest you can’t do frame-perfect audio on client computers. Conceptually, Sequencer can evaluate on sub-frames and evaluation isn’t locked to whole frames. If time dilation is on a curve, or even changed at all within a sequence, gameplay clients can fall on different parts of the curve which will get different time multiplication values, which will cause their next frame to evaluate at a different duration.
ie: in normal gameplay, if you have a sequence that has 1x time dilation from frames 0-5, then slows down to 0.5x in a non-linear fashion from 5-10, then stays low, a machine at low framerates might evaluate frame 0 (1x), then frame 3 (1x), then frame 7 (partial slowdown), the “slow motion” time represented by frames 5 and 6 is just… gone, it was never evaluated and never taken into account. While a fast machine might evaluate frame 0, 0.5, 1, 1.5, 2, 2.5, 3, 3.5, 4, 4.5, 5, 5.5, 6, 6.5, etc. and closely follows the curve because the whole sequence is being evaluated more often.
The small difference in total frames between Sequencer and MRQ is expected as there is an additional frame delay in MRQ before it detects time dilation and can be properly handled in sampling. MRQ doesn’t know time dilation has happened until the sequence has been evaluated, we don’t evaluate the sequence until after we calculate the time for a frame, therefor it’s one frame behind every time dilation changes, but it’s also much more accurate about when the ‘end’ of a sequence is (and will throw out a partial frame at the end if there’s not enough time left for 1/24th of time), which is also different behavior than realtime playback in Sequencer.