Hello again,
Great question!
It’s possible to use replays but there are some major challenges. I’ve tried it - not with Lyra though - so this is coming from 1st-hand experience:
- The UE built in replay system does not directly record ground-truth actions, i.e. user inputs. For example, rotations of the player pawns will be the final result synced over the network, so if you have multiple things manipulating the rotations, it’s challenging to untangle them. For example, weapon recoil to pitch a gun up vs user moved the mouse to pitch up. Depending on the problem you’re trying to solve, this may not be a issue for you.
- You could try to solve this by learning an inverse dynamics model or by pushing up more data into the replays (I’m not an expert on how to do this - yet)
- The replays are not a deterministic simulation of the game, so querying the game state can be challenging as well. For example, at game time you might call “IsFiring()” on a Pawn to see if its currently firing its weapon, but at replay time this function might always return false because the backing code doesn’t need to run (only the special effects like bullet tracers actually get played back at replay time)
- You can get creative with how you query game state but it’s a major pain.
How could you do it though?
Assuming you still want to try, you simply spawn a learning agents manager in the replay level and have it query the stuff you want through the interactor and save off the data with the LA Data Recorder object. I haven’t done this recently so it’s possible there is some regression in the LA design that makes this no longer work (please let me know if you try and run into problems).
I’m working on internally “solving” this issue and hope to be able to port the solution into LA/replay system but I have absolutely no idea on the time frame or if it’ll ever come!
Hope that answer your question!
Brendan