How to use a behavior tree based AI (or player controlled pawn) to generate a recording for imitation learning

[mirrorclimb]

I have followed the tutorial on reinforcement learning and have it working decently in my game in a simple use case, but for more complex use cases I have having issues with the AI learning very slowly, or getting stuck in a local maximum.

In some of these cases, I think starting with some initial imitation learning would drastically accelerate the process and I have behavior tree based AIs that can execute portions of the behavior I am trying to have my learning agents do. I would like to use the behavior tree based AIs (or potentially a player controlled pawn) to record data for imitiation learning.

I am trying to set up the recording like this (here the agent being added is a pawn controlled by a behavior tree):

learning agents recorder setup posted by anonymous | blueprintUE | PasteBin...

then once my actors are all initialized and ready for training I am doing this

Begin Imitation Record posted by anonymous | blueprintUE | PasteBin For...

However, I am not clear on what the learning manager class on tick should look like.

I can do this

learning manager recording tick posted by anonymous | blueprintUE | PasteBin...

If I print the output of Get Observation Vector everything looks good and it appears to be correctly gathering observations from the agent that is controlled by a behavior tree.

I then added code to call “setFloatAction” for each of the interactors actions and used game data to calculate a float action that is equivalent to what the behavior tree decided to do (such that if a learning agent called getActions with the float actions set to those values then it would do exactly what the behavior tree controlled agent did). I can also print those float actions values and they look proper and match what the behavior tree is doing.

I believe the next step should be to call Add Experience like this:

However, when I make that call I get console errors like this:

LogLearning: Warning: BP_FighterRecorder: Agent with id 0 has not made observations and taken actions so experience will not be recorded for it.

If I try to call “getActionVector” I get a related (I think) error

LogLearning: Error: BP_FighterInteractor: Agent with id 0 has not yet computed an action vector. Have you run EvaluatePolicy or EncodeActions?

However, as far as I can tell there is no EncodeActions function and I don’t think calling EvaluatePolicy makes sense (there is no policy, I am just trying to record…

Any help is greatly appreciated, thanks!!!

[mirrorclimb]

Hah I figured out my own problem, but I’ve leave this up incase others are confused by it in the future.

You need to also include a LearningAgentsController component which you can then setup like this:

The controller is then the object that lets you call EncodeActions

For some reason I had expected that EncodeActions and EncodeObservations would both be on the interactor.

With that addition the AddExperience function no longer gives errors and the recording works as expected: