How to use custom model in Learning Agents?

Development Programming & Scripting AI
, , , , ,
1

I am planning to conduct research on reinforcement learning and need to gather information and create a simple demo. I am currently planning to use ‘Learning Agents’ and would like to know how to apply a custom model. It seems that I have to modify python file in learning agents but I’m not sure. Is there anyone knowledgeable or able to provide some tips on this?

2

For example, I assume that I want to add cumstom layers to my policy network. I found that in train_ppo.py there is a line defining policy netowrk using class NeuralNetwork which is defined in nne_runtime_basic_cpu_pytorch.py. Then it seems that policy network is updated to trainer using ‘recv_policy’ function. Then if I want to add some layer, can I just add layer to policy network after respone = trainer.recv_policy(policy_network) line? If not, what should I do? If so, what do I have to do next? I’m sorry that I’m not familiar using Unreal Engine with python files.

3

Yes, essentially you would have to modify the python code. You would need to make corresponding changes on the C++ side as well or code in a way to use ONNX. This is because the model is evaluated in UE for doing rollouts in the environment (we don’t receive actions from the python side, only model weights). This is a bit inconvenient for researchers but it makes LA a much easier transition from development to deployment for game developers.

Eventually I plan to have a “bring your own algorithm” functionality (working on it now actually) but not sure when it will land. Perhaps part of UE 5.5 but no guarantee on that.

Brendan

4

Don’t you also have to edit the python code to adjust the number of epochs to run in training?

5

I have confirmed that the update from 5.4 to 5.5 was made, making it more convenient to use custom models. Could you explain how to apply custom models in this context, or do you have any plans to create a tutorial for this?

6

If I have time, I would like to make a detailed tutorial, but it’s likely I won’t find the time until we can get Learning Agents from Experimental → Beta.

Here’s the brief strategy:

  1. Make a new Trainer class derived from LearningAgentsTrainer or Listener
  2. Make a new python class, similar to train_ppo.py
  3. Put the python file into {your-game-project}\Content\Python
  4. Adjust the trainer settings to pass in the file name of your new trainer
  5. ???
  6. PROFIT!

I have used this internally to make a new variant of PPO which seems to unblock me. I’m hoping to have time to work on an implementation of offline SAC which should help push this forward.

If I have a magic wand, there would only be one C++ or Python class, and not have to maintain two “sources of truth” but software design is hard lol and we need to be pragmatic.

Thanks,
Brendan

7

Unfortunately there is still quite a bit of janky bits when it comes to training and python, hence the experimental status. We have plans to make these things better with time but we got bigger fish to fry and this whole project is made by a small team.

Thanks!

8

I forgot I did a write-up over here with how to use ONNX if you want. The code I provided should be thought of as pseudo-code. I wrote it up in a notepad and did not try to compile it. The idea is you make a sub-class of Policy and add some bits which let LA work with NNE & ONNX:

We haven’t made this part of the LA “official” yet because it has a lot of rough edges that need to be fixed up before it will be “good” for our definition of good. Eventually it will probably come… but for now you need to do a little C++ dev in your game project.

Thanks,
Brendan