A brief announcement of Learning Agents: a machine learning plugin for AI bots. Learning Agents allows you to train your NPCs via reinforcement & imitation learning. It aims to be useful in the creation of game-playing agents, physics-based animations, automated QA bots, and much more!
https://dev.epicgames.com/community/learning/tutorials/8OWY/unreal-engine-learning-agents-introduction
16 Likes
Great work! I am so pumped about this. I’ve been waiting a long time for native support of learning agents.
5 Likes
Is there a github repo or associated plugin for this?
2 Likes
This sounds super interesting! Is there anything that you can share right now that describes how to get started with this? E.g. a simple example that we can build upon?
It’s in UE5-Main on GitHub. The current preview of UE5.2 doesn’t have it.
1 Like
I was able to find the plugin in the Experimental folder. However i cannot find the python example scripts (the post states there are some). Where can i find them? Thanks!
As @order66 has already discovered, this plugin is currently only available in a source build of the engine under (https://github.com/EpicGames/UnrealEngine/tree/ue5-main/Engine/Plugins/Experimental/LearningAgents). The plugin will be coming in a later UE release (not 5.2 - I marked the tutorial as 5.2 because it’s a limitation of the developer site, i.e. you have to select a version). Sorry if this post caused any confusion and thanks for your patience!
@TRoman07 Thanks for pointing this out:
I realize what is up now. The Content folder isn’t normally distributed via regular git since it is typically binary files. If you follow the instructions to build from source (https://github.com/EpicGames/UnrealEngine), at some point post fetch, you will pull down the Content/Python directory and have the full plugin. Keep in mind you have to be on the ue5-main branch, which can be unstable.
Otherwise, wait until later when the plugin becomes available via your normal Epic Games Launcher install 
Yep after building it all I see the Python folder with the examples. It is located in Engine/Plugins/Experimental/LearningAgents/Content/Python.
Thanks!
Is it possible to still use this plugin with the Gymnasium RL framework (used to be OpenAI gym)? If so, could you maybe add an example of it to the Python code? Compared to this new version of the plugin, the ML-Adapter plugin seemed to be quite a bit easier to use.
Thanks for the question!
No, Learning Agents does not work with Open AI Gym. We feel like the design of Learning Agents is much better for someone focused on shipping ML in working games. There is a very natural path to go from training to inference in games, whereas ML Adapter didn’t lend itself to that very well.
In Learning Agents, the game itself is responsible for gathering data and pushing it over to the python process for training, vs with ML Adapter that relationship was flipped: the python process would call the UE process. The design in Learning Agents is much more flexible as you are in control of both the details of experience gathering, as well as the full training loop. For example, if you have a turn-based game where you only want to run inference once when the AI’s turn starts, this is super easy to do in Learning Agents, vs ML Adapter would require you to do something incredibly hacky.
I hope you will give Learning Agents a shot and provide us more feedback after you have some experience with it.
2 Likes
Is there anything that you can share right now that describes how to get started with this? E.g. a simple example that we can build upon?
I think the only examples that are available currently can be obtained by building the ue5-main branch from source. During the building process it will add Python files to the Plugin that demonstrate how to use the plugin. However, at least for me, these examples did not really help understand how to get started.
@Deathcalibur Is there any ETA on the learning course and more examples?
We are currently working on a course with examples. I wouldn’t expect it to be available very soon. At the latest, it will come around the time the plugin is actually released with UE.
Sorry to be vague but we don’t have a firm timeline yet.
1 Like
I just published a brief tutorial on how to get started with the plugin. I got it up and running, and if you follow the same steps, you should too!
Link to my tutorial: Early Explorations of Learning Agents in Unreal Engine | by Jonathan Jørgensen | Apr, 2023 | Medium
I hope you find this useful!
5 Likes
Awesome work Jonathan! You figured out how most of everything works. A couple changes needed in your implementation:
-
Your inference loop should call EncodeObservations before EvaluatePolicy otherwise your observations will be a frame delayed
-
Call the trainer methods (rewards/completions/iterate) before the Agent Type’s Encode/Evaluate/Decode (and skip trainer stuff on the first iteration of the loop)
Some notes:
-
Policies are going to move out of the agent type soon to become their own object (for more flexibility)
-
We’re going to remove the notion of AgentTypeComponents and instead you would compose your inference and training loops explicitly on an object, very similar to your “LearningManager”. We’re going to take away most of the callbacks (OnAgentSetup and OnAgentAdded) because we think the approach you are using is less error prone than using these callbacks. The major change to your implementation would be calling AddAgent(s) the trainer + calling SetupTrainer manually.
-
You can make your LearningManager Tick less than every frame if you want your actions to have some time to affect the world before you take your next observations.
2 Likes
Thanks alot! I did wonder about whether eval policy would be able to use freshly encoded observations before the iteration call is made. I’ll continue to toy around, and when everything is neat and pretty, I can make a public repo and link it here. I suppose most people would prefer to clone and edit, instead of following clumsy steps from a medium article in order to try for themselves.
Gotta leave for easter now, but I’m excited to get my hands on this as soon as I’m back!
2 Likes
Allot of this machine learning functionality and an expanded set of reinforcement learning functionality is already available with Mindmaker ML plugin MindMaker Machine Learning AI Plugin in Code Plugins - UE Marketplace
1 Like
Big fan of this movement from Epic! Hype!
I have been working on projects with Unity’s ML-agents for nearly 2 years now and recently becoming very disappointed on their progress & support on the library. Its still way ahead of this UE learning agents in terms of features etc., but if Epic puts enough efforts and resources into this, it could catch up soon enough and hopefully continue aligning with the achievements in Reinforcement learning research and deliver good support.
Best of luck. I am fallowing
3 Likes
In case anyone is playing around with the plugin from source, this morning I pushed out what might be the last major update for the near future (there is still plenty of small issues and potential bug fixes coming in the next weeks).
If you have checked it out, here are some changes which will impact you:
- ULearningAgentsManager - we made a new actor that is sort of the core of the system. We split this out from Agent Type. @jonathaj - this is similar to the manager you made (sort of). It contains the code responsible for add/remove agent, and you attach the other components to this actor. You can have multiple of them if you need/want.
- ULearningAgentsInteractor - This is what the rest of Agent Type became. The interactor is responsible for defining & implementing how the agents interact with the game world, i.e. observations and actions.
We also added some convenience methods like “RunInference” on the ULearningAgentsPolicy and “RunTraining” on the ULearningAgentsTrainer (again @jonathaj - might be useful in your playing around).
I will post a course similar to Jonathan’s to the forums eventually (at the very latest when the plugin becomes available in your normal UE download). I’ll link to it here when it’s available.
Thanks! Looking forward to any feedback and really excited to see some learning agents in the wild!
3 Likes