Learning Agents Imitation Learning Recording logic question

Development Programming & Scripting
, , ,
1

Hello. I have a question about Imitation Learning Recording logic. So I basically went by “Learn to drive” LA tutorial. I am working on a stock UE 5.7.2.
I have an IL recording manager that in each update calls
ULearningAgentsController::RunController
ULearningAgentsRecorder::AddExperience

I also use a player imitator character as actions producer, so I manually play during IL recording sessions and create actions via ULearningAgentsActions::Make_X_Action.

and ULearningAgentsController::RunController calls
Interactor->GatherObservations();
EvaluateController();
Interactor->PerformActions();

THE PROBLEM:
And the function call “Interactor->PerformActions()” is a problem for me, because in my interactor PerformAgentAction_Implementation logic it is expected that certain components exist on agent actor (to be specific, components like “do X agent action” so that I don’t cast to a character class directly) and

  1. My imitator character class doesn’t have these components because it is not intended to be a LA agent. So no components → need to add redundant null checks in the interactor
  2. I don’t really understand why IL controller calls PerformActions after I report already made actions in ULearningAgentsController::EvaluateAgentController_Implementation. I looked into the code starting from PerformActions and I can’t really tell if it does anything significant for recording and/or training process so I assume the PerformActions doesn’t have to be called, but I’m not 100% sure.

THE QUESTION:
So what can/should I do in this situation? I mean I can think of 2 options:

  1. Add a custom RunControllerWithoutSamplingActions function in my ULearningAgentsController child class that calls only
    Interactor->GatherObservations();
    EvaluateController();
    without a Interactor->PerformActions call

  2. I could add a boolean “bImitationRecordingMode” to my Interactor and set it in my IL Recording Manager class and in interactors PerformActions_Implementation just do an early return if it’s in IL recording mode

But maybe I just don’t understand something about IL training process and calling PerformActions is actually essential here and actions MUST be sampled even if the agent is an imitator character controlled by a human player?

2

Controller→EvaluateController is public and so is Interactor→GatherObservations (and both are exposed to BP), so you can just call these functions directly if you like to do something unique. Controller→RunController is merely a convenience function to wrap the typical functionality together.

Normally we run PerformActions because we don’t have anything else driving the agent’s behavior, i.e. the controller shouldn’t be calling functions on the agent to enact the action, that’s the interactor’s job. And obviously we want to take actions so we can get to the next game state.

For times when there is an actual human player in the loop, we are still re-using the interactor between the human player data collection mode and the AI controlled inference mode. We simply just have a check inside the PerformActions and skip it, but again this could have been done at the manager level by using the non-convenience API.

3

So, if I understood you right, both of my assumed options in the topic post are viable. And the call to Interactor→PerformActions isn’t required for a proper IL recording workflow, obviously as long as my agent keeps doing things by getting actions from whatever external source.

Okay, thanks for clarifying this.

1 Like
4

Yes, basically both are viable.

In hindsight, I’m not sure if we should have included PerformActions inside of RunController. It makes sense when you are imitating another AI agent, but when imitating humans it doesn’t make sense.