Tutorial: Learning to Drive

Deathcalibur · January 8, 2024, 2:46pm

An episode will accumulate experience for an agent if that agent’s Observations/Actions/Rewards were all able to run. I would ensure that all these are running for your agent before your completion. Try setting break points and also make sure to pass along the correct agent ID by wiring that up (assuming you are using blueprints, but generally same advice for C++).

I can’t debug your whole setup for you but this Warning is trying to help you not waste any time when you think you are collecting data but you are actually not!

huangzike · January 19, 2024, 3:05pm

Hi Brendan I am new to ML in Unreal and want to check the reward curve, but it seems like my TensorBoard isn’t working.

LogLearning: Display: BP_RLTrainer: Sending / Receiving initial policy...
LogLearning: Display: Training Process: Failed to Load TensorBoard. Could not be found in site-packages.
LogLearning: Display: Training Process: {
LogLearning: Display: Training Process:     "TaskName": "BP_RLTrainer",
LogLearning: Display: Training Process:     "TrainerMethod": "PPO",
LogLearning: Display: Training Process:     "TrainerType": "SharedMemory",
LogLearning: Display: Training Process:     "TimeStamp": "2024-01-19_22-50-03",
LogLearning: Display: Training Process:     "SitePackagesPath": "D:/ue 5.3/UE_5.3/Engine/Plugins/Experimental/PythonFoundationPackages/Content/Python/Lib/Win64/site-packages",
LogLearning: Display: Training Process:     "IntermediatePath": "D:/ue 5.3/project/RL_ship/Intermediate/LearningAgents",

Do you know what might be the issue？

Deathcalibur · February 14, 2024, 9:59pm

TensorBoard can be installed by going to your Unreal Editor Python Binaries directory (e.g. “\Engine\Binaries\ThirdParty\Python3\Win64”) and running python -m pip install tensorboard

Unfortunately this is a manual step at this point in time but something we can hopefully address automatically later.

Confeather · February 20, 2024, 3:41pm

Hi Brendan, thanks your tutorial.
I want to train the NeuralNetwork in EWorldType::Game, not EWorldType::PIE.
How can I save the LearningAgentsNeuralNetwork(named “DA_DrivingNetwork” in the tutorial) in the game world after package project?
I tried a lot, but failed.

Deathcalibur · February 20, 2024, 4:31pm

Hey, thanks for your question. We don’t have a tutorial about how to do this currently but basically you need to:

Adjust the Trainer Path Settings to work outside the editor
Enabled Snapshots in Trainer Training Settings
After training, call “LoadNetworkFromSnapshot” on the Policy component after Setup

I don’t have the time at the moment to do a full write-up but I intend to add more tutorials to accompany the next release of Learning Agents.

tomhalpin8 · February 20, 2024, 6:21pm

Hey @Deathcalibur Thanks to you and your team for getting this to Unreal! I was wondering if there are any updates we could be looking forward to in UE 5.4? Possible Apple Silicon integration?

Confeather · February 21, 2024, 2:07am

Thanks for help.
But I can’t find the UPROPERTY named “Enabled Snapshots” in Trainer Training Settings.
What does it mean to “Enabled Snapshots” in tips?

Deathcalibur · February 21, 2024, 2:41pm

Oh sorry I’ve been using the latest version of the code on ue5-main that I forgot what the API looked like in 5.3.

Look at these functions on the Policy:

	/**
	 * Load a snapshots weights into this policy.
	 * @param File The snapshot file.
	 */
	UFUNCTION(BlueprintCallable, Category = "LearningAgents", meta = (RelativePath))
	void LoadPolicyFromSnapshot(const FFilePath& File);

	/**
	 * Save this policys weights into a snapshot.
	 * @param File The snapshot file.
	 */
	UFUNCTION(BlueprintCallable, BlueprintPure = false, Category = "LearningAgents", meta = (RelativePath))
	void SavePolicyToSnapshot(const FFilePath& File) const;

Deathcalibur · February 26, 2024, 2:44pm

We will have a big update coming with 5.4!

The most substantial change is that we completely reworked the Interactor. You now define a schema object which is shared with the training process behind the scenes. This allows us to do much more sophisticated observations including Attention over sets of objects, optional observations, etc.

Here’s a sneak peek in blueprint form (the C++ API is delightful as well):

Expect completely breaking changes, but the upside is you get a much more flexible/powerful API and it is actually simpler as well.

Unfortunately, we were unable to get Mac/Linux support working as it was lower priority for us at the moment. There is a known issue with spawning the python training process right now.

Thanks for your interest!
Brendan

Mahzad · February 28, 2024, 8:47pm

Great! Have you planned a tutorial for the RL and also one for imitation learning with this new release?

Deathcalibur · February 29, 2024, 2:29pm

Hello,

Here’s my intention with tutorials. I can’t promise I will be able to get them all done, but if no major surprises come up:

Learning to Drive
- Racing well (btw behavior looks 10x better after taking some inspiration from the GT Sophy paper)
- Driving with static obstacle avoidance (i.e. avoiding pillars/cones/etc.)
- Car-on-car collision avoidance would be great
  - This is more challenging to train as agents’ behavior is influenced by other agents. So if one agent goes haywire it can have a cascading effect on the training & inference
  - Works decently with ray tracing observation, but would be nice to have Attention version working well too (or at least a solid understanding of why the attention has more trouble)
- Imitation Learning
  - How to create datasets
  - How to save/load policies, continue training from a trained model
Line Up Four (Connect Four-inspired game)
- Demonstrate how discrete actions work
- Illustrate how competitive self-play can be setup
Brief Tensorboard tutorial and resuming from snapshots

A couple questions for you (and anyone!):

Is there anything else you think is more crucial than the above? i.e. should be prioritized over the above.
Is there anything that would be nice to have if there is extra time?

I can’t promise that I will act on your suggestions but I will carefully consider them!

Thanks for your time,
Brendan

Mahzad · February 29, 2024, 3:10pm

Geat news thanks for your work!
My suggestion would be that for imitation learning it could be based on a third person view, for example teaching the agent to go from point A to point B.

Deathcalibur · February 29, 2024, 5:11pm

Sure, that should be simple enough! We have tests like that internally they just aren’t polished up for a tutorial haha. I will think about this more and see what I can do.

Thanks,
Brendan

Mahzad · March 20, 2024, 5:13pm

Hi! With the announcement of UE 5.4, will there be a specific announcement for Learning Agents in the next few days?

Deathcalibur · March 20, 2024, 6:58pm

We will post an announcement and tutorials closer to the full release of 5.4, which should be in about a month or so.

Thanks!

Jeebius · March 29, 2024, 2:41pm

@Deathcalibur awesome stuff, I’ve been playing around the past few days. I’m trying to create a ray observation and I wanted to know if this made sense.

I have a few ray traces and I am feeding the hit locations into a position observation array with the agents location and rotation fed into the relative position and rotation.

My hope was that the agents would be able to detect some distance and avoid obstacles.

Deathcalibur · March 29, 2024, 3:41pm

That should work. You can also add the type of object hit if you have a good way to enumerate them.

Brendan

Jeebius · March 30, 2024, 8:46am

Oh awesome! So in theory, if the enumeration is in a separate observation, the agents should be able to correlate the object type to the hit distance?

Sengchor · April 12, 2024, 2:35pm

Thank you for the tutorial. I want the car to avoid collisions with other cars and get racing behavior. Can you provide some ideas on how to achieve that?

Deathcalibur · April 12, 2024, 2:37pm

I have more tutorials coming with the imminent release of 5.4. The new Learning Agents Interactor API makes this 100x easier (IMO).

Thanks!