# Tutorial: Learning to Drive

**URL:** <https://forums.unrealengine.com/t/tutorial-learning-to-drive/1275490>\
**Category:** Tutorial & Course Discussions\
**Tags:** unreal-engine\
**Created:** [August 28, 2023, 3:43pm UTC](https://forums.unrealengine.com/t/tutorial-learning-to-drive/1275490 "2023-08-28T15:43:16Z")\
**Posts on this page:** 20\
**Page:** 6

<div class="post-metadata">

**Author:** ![Deathcalibur](https://d1ap1mz92jnks1.cloudfront.net/user_avatar/forums.unrealengine.com/deathcalibur/32/1037610_2.png) [@Deathcalibur](https://forums.unrealengine.com/u/Deathcalibur)\
**Post date:** [January 8, 2024, 2:46pm UTC](https://forums.unrealengine.com/t/tutorial-learning-to-drive/1275490/110 "2024-01-08T14:46:34Z")

</div>

An episode will accumulate experience for an agent if that agent’s Observations/Actions/Rewards were all able to run. I would ensure that all these are running for your agent before your completion. Try setting break points and also make sure to pass along the correct agent ID by wiring that up (assuming you are using blueprints, but generally same advice for C++).

I can’t debug your whole setup for you but this **Warning** is trying to help you not waste any time when you think you are collecting data but you are actually not!

---

<div class="post-metadata">

**Author:** ![huangzike](https://d3kjluh73b9h9o.cloudfront.net/original/4X/2/8/5/28502927f1d97ba1051b3b47ccfe17233bf41ef0.png) [@huangzike](https://forums.unrealengine.com/u/huangzike)\
**Post date:** [January 19, 2024, 3:05pm UTC](https://forums.unrealengine.com/t/tutorial-learning-to-drive/1275490/111 "2024-01-19T15:05:25Z")

</div>

Hi Brendan I am new to ML in Unreal and want to check the reward curve, but it seems like my TensorBoard isn’t working.

```auto
LogLearning: Display: BP_RLTrainer: Sending / Receiving initial policy...
LogLearning: Display: Training Process: Failed to Load TensorBoard. Could not be found in site-packages.
LogLearning: Display: Training Process: {
LogLearning: Display: Training Process: "TaskName": "BP_RLTrainer",
LogLearning: Display: Training Process: "TrainerMethod": "PPO",
LogLearning: Display: Training Process: "TrainerType": "SharedMemory",
LogLearning: Display: Training Process: "TimeStamp": "2024-01-19_22-50-03",
LogLearning: Display: Training Process: "SitePackagesPath": "D:/ue 5.3/UE_5.3/Engine/Plugins/Experimental/PythonFoundationPackages/Content/Python/Lib/Win64/site-packages",
LogLearning: Display: Training Process: "IntermediatePath": "D:/ue 5.3/project/RL_ship/Intermediate/LearningAgents",

```

Do you know what might be the issue？

---

<div class="post-metadata">

**Author:** ![Deathcalibur](https://d1ap1mz92jnks1.cloudfront.net/user_avatar/forums.unrealengine.com/deathcalibur/32/1037610_2.png) [@Deathcalibur](https://forums.unrealengine.com/u/Deathcalibur)\
**Post date:** [February 14, 2024, 9:59pm UTC](https://forums.unrealengine.com/t/tutorial-learning-to-drive/1275490/112 "2024-02-14T21:59:41Z")

</div>

TensorBoard can be installed by going to your Unreal Editor Python Binaries directory (e.g. “\Engine\Binaries\ThirdParty\Python3\Win64”) and running `python -m pip install tensorboard`

Unfortunately this is a manual step at this point in time but something we can hopefully address automatically later.

---

<div class="post-metadata">

**Author:** ![Confeather](https://d3kjluh73b9h9o.cloudfront.net/original/4X/2/8/5/28502927f1d97ba1051b3b47ccfe17233bf41ef0.png) [@Confeather](https://forums.unrealengine.com/u/Confeather)\
**Post date:** [February 20, 2024, 3:41pm UTC](https://forums.unrealengine.com/t/tutorial-learning-to-drive/1275490/113 "2024-02-20T15:41:02Z")

</div>

Hi Brendan, thanks your tutorial.  
I want to train the NeuralNetwork in EWorldType::Game, not EWorldType::PIE.  
How can I save the LearningAgentsNeuralNetwork(named “DA\_DrivingNetwork” in the tutorial) in the game world after package project?  
I tried a lot, but failed.

---

<div class="post-metadata">

**Author:** ![Deathcalibur](https://d1ap1mz92jnks1.cloudfront.net/user_avatar/forums.unrealengine.com/deathcalibur/32/1037610_2.png) [@Deathcalibur](https://forums.unrealengine.com/u/Deathcalibur)\
**Post date:** [February 20, 2024, 4:31pm UTC](https://forums.unrealengine.com/t/tutorial-learning-to-drive/1275490/114 "2024-02-20T16:31:04Z")

</div>

Hey, thanks for your question. We don’t have a tutorial about how to do this currently but basically you need to:

1. Adjust the Trainer Path Settings to work outside the editor
2. Enabled Snapshots in Trainer Training Settings
3. After training, call “LoadNetworkFromSnapshot” on the Policy component after Setup

I don’t have the time at the moment to do a full write-up but I intend to add more tutorials to accompany the next release of Learning Agents.

---

<div class="post-metadata">

**Author:** ![tomhalpin8](https://d3kjluh73b9h9o.cloudfront.net/original/4X/2/8/5/28502927f1d97ba1051b3b47ccfe17233bf41ef0.png) [@tomhalpin8](https://forums.unrealengine.com/u/tomhalpin8)\
**Post date:** [February 20, 2024, 6:21pm UTC](https://forums.unrealengine.com/t/tutorial-learning-to-drive/1275490/115 "2024-02-20T18:21:19Z")

</div>

Hey @Deathcalibur Thanks to you and your team for getting this to Unreal! I was wondering if there are any updates we could be looking forward to in UE 5.4? Possible Apple Silicon integration? 😃

---

<div class="post-metadata">

**Author:** ![Confeather](https://d3kjluh73b9h9o.cloudfront.net/original/4X/2/8/5/28502927f1d97ba1051b3b47ccfe17233bf41ef0.png) [@Confeather](https://forums.unrealengine.com/u/Confeather)\
**Post date:** [February 21, 2024, 2:07am UTC](https://forums.unrealengine.com/t/tutorial-learning-to-drive/1275490/116 "2024-02-21T02:07:54Z")

</div>

Thanks for help.  
But I can’t find the UPROPERTY named “Enabled Snapshots” in Trainer Training Settings.  
What does it mean to “Enabled Snapshots” in tips?

 ![image](https://d3kjluh73b9h9o.cloudfront.net/original/4X/9/7/a/97aa3be79db4dbde03c21a6be3e9f1dbd5f98cba.png)

---

<div class="post-metadata">

**Author:** ![Deathcalibur](https://d1ap1mz92jnks1.cloudfront.net/user_avatar/forums.unrealengine.com/deathcalibur/32/1037610_2.png) [@Deathcalibur](https://forums.unrealengine.com/u/Deathcalibur)\
**Post date:** [February 21, 2024, 2:41pm UTC](https://forums.unrealengine.com/t/tutorial-learning-to-drive/1275490/117 "2024-02-21T14:41:40Z")

</div>

Oh sorry I’ve been using the latest version of the code on ue5-main that I forgot what the API looked like in 5.3.

Look at these functions on the Policy:

```auto
	/**
	 * Load a snapshots weights into this policy.
	 * @param File The snapshot file.
	 */
	UFUNCTION(BlueprintCallable, Category = "LearningAgents", meta = (RelativePath))
	void LoadPolicyFromSnapshot(const FFilePath& File);

	/**
	 * Save this policys weights into a snapshot.
	 * @param File The snapshot file.
	 */
	UFUNCTION(BlueprintCallable, BlueprintPure = false, Category = "LearningAgents", meta = (RelativePath))
	void SavePolicyToSnapshot(const FFilePath& File) const;

```

---

<div class="post-metadata">

**Author:** ![Deathcalibur](https://d1ap1mz92jnks1.cloudfront.net/user_avatar/forums.unrealengine.com/deathcalibur/32/1037610_2.png) [@Deathcalibur](https://forums.unrealengine.com/u/Deathcalibur)\
**Post date:** [February 26, 2024, 2:44pm UTC](https://forums.unrealengine.com/t/tutorial-learning-to-drive/1275490/118 "2024-02-26T14:44:34Z")

</div>

We will have a big update coming with 5.4!

The most substantial change is that we completely reworked the **Interactor**. You now define a schema object which is shared with the training process behind the scenes. This allows us to do much more sophisticated observations including Attention over sets of objects, optional observations, etc.

Here’s a sneak peek in blueprint form (the C++ API is delightful as well):

 ![image](https://d3kjluh73b9h9o.cloudfront.net/original/4X/c/1/f/c1f307a0e603715bc8d362a91c5a9731aa8afe65.jpeg)

Expect completely breaking changes, but the upside is you get a much more flexible/powerful API and it is actually simpler as well.

Unfortunately, we were unable to get Mac/Linux support working as it was lower priority for us at the moment. There is a known issue with spawning the python training process right now.

Thanks for your interest!  
Brendan

---

<div class="post-metadata">

**Author:** ![Mahzad](https://d1ap1mz92jnks1.cloudfront.net/user_avatar/forums.unrealengine.com/mahzad/32/745429_2.png) [@Mahzad](https://forums.unrealengine.com/u/Mahzad)\
**Post date:** [February 28, 2024, 8:47pm UTC](https://forums.unrealengine.com/t/tutorial-learning-to-drive/1275490/119 "2024-02-28T20:47:48Z")

</div>

Great! Have you planned a tutorial for the RL and also one for imitation learning with this new release?

---

<div class="post-metadata">

**Author:** ![Deathcalibur](https://d1ap1mz92jnks1.cloudfront.net/user_avatar/forums.unrealengine.com/deathcalibur/32/1037610_2.png) [@Deathcalibur](https://forums.unrealengine.com/u/Deathcalibur)\
**Post date:** [February 29, 2024, 2:29pm UTC](https://forums.unrealengine.com/t/tutorial-learning-to-drive/1275490/120 "2024-02-29T14:29:21Z")

</div>

Hello,

Here’s my intention with tutorials. I can’t promise I will be able to get them all done, but if no major surprises come up:

- Learning to Drive
  - Racing well (btw behavior looks 10x better after taking some inspiration from the GT Sophy paper)
  - Driving with static obstacle avoidance (i.e. avoiding pillars/cones/etc.)
  - Car-on-car collision avoidance would be great
    - This is more challenging to train as agents’ behavior is influenced by other agents. So if one agent goes haywire it can have a cascading effect on the training & inference
    - Works decently with ray tracing observation, but would be nice to have Attention version working well too (or at least a solid understanding of why the attention has more trouble)

  - Imitation Learning
    - How to create datasets
    - How to save/load policies, continue training from a trained model

- Line Up Four (Connect Four-inspired game)
  - Demonstrate how discrete actions work
  - Illustrate how competitive self-play can be setup

- Brief Tensorboard tutorial and resuming from snapshots

**A couple questions for you (and anyone!):**

1. Is there anything else you think is more crucial than the above? i.e. should be prioritized over the above.
2. Is there anything that would be nice to have if there is extra time?

I can’t promise that I will act on your suggestions but I will carefully consider them!

Thanks for your time,  
Brendan

---

<div class="post-metadata">

**Author:** ![Mahzad](https://d1ap1mz92jnks1.cloudfront.net/user_avatar/forums.unrealengine.com/mahzad/32/745429_2.png) [@Mahzad](https://forums.unrealengine.com/u/Mahzad)\
**Post date:** [February 29, 2024, 3:10pm UTC](https://forums.unrealengine.com/t/tutorial-learning-to-drive/1275490/121 "2024-02-29T15:10:44Z")

</div>

Geat news thanks for your work!  
My suggestion would be that for imitation learning it could be based on a third person view, for example teaching the agent to go from point A to point B.

---

<div class="post-metadata">

**Author:** ![Deathcalibur](https://d1ap1mz92jnks1.cloudfront.net/user_avatar/forums.unrealengine.com/deathcalibur/32/1037610_2.png) [@Deathcalibur](https://forums.unrealengine.com/u/Deathcalibur)\
**Post date:** [February 29, 2024, 5:11pm UTC](https://forums.unrealengine.com/t/tutorial-learning-to-drive/1275490/122 "2024-02-29T17:11:26Z")

</div>

Sure, that should be simple enough! We have tests like that internally they just aren’t polished up for a tutorial haha. I will think about this more and see what I can do.

Thanks,  
Brendan

---

<div class="post-metadata">

**Author:** ![Mahzad](https://d1ap1mz92jnks1.cloudfront.net/user_avatar/forums.unrealengine.com/mahzad/32/745429_2.png) [@Mahzad](https://forums.unrealengine.com/u/Mahzad)\
**Post date:** [March 20, 2024, 5:13pm UTC](https://forums.unrealengine.com/t/tutorial-learning-to-drive/1275490/123 "2024-03-20T17:13:52Z")

</div>

Hi! With the announcement of UE 5.4, will there be a specific announcement for Learning Agents in the next few days?

---

<div class="post-metadata">

**Author:** ![Deathcalibur](https://d1ap1mz92jnks1.cloudfront.net/user_avatar/forums.unrealengine.com/deathcalibur/32/1037610_2.png) [@Deathcalibur](https://forums.unrealengine.com/u/Deathcalibur)\
**Post date:** [March 20, 2024, 6:58pm UTC](https://forums.unrealengine.com/t/tutorial-learning-to-drive/1275490/124 "2024-03-20T18:58:42Z")

</div>

We will post an announcement and tutorials closer to the full release of 5.4, which should be in about a month or so.

Thanks!

---

<div class="post-metadata">

**Author:** ![Jeebius](https://d3kjluh73b9h9o.cloudfront.net/original/4X/2/8/5/28502927f1d97ba1051b3b47ccfe17233bf41ef0.png) [@Jeebius](https://forums.unrealengine.com/u/Jeebius)\
**Post date:** [March 29, 2024, 2:41pm UTC](https://forums.unrealengine.com/t/tutorial-learning-to-drive/1275490/125 "2024-03-29T14:41:00Z")

</div>

@Deathcalibur awesome stuff, I’ve been playing around the past few days. I’m trying to create a ray observation and I wanted to know if this made sense.

I have a few ray traces and I am feeding the hit locations into a position observation array with the agents location and rotation fed into the relative position and rotation.

My hope was that the agents would be able to detect some distance and avoid obstacles.

---

<div class="post-metadata">

**Author:** ![Deathcalibur](https://d1ap1mz92jnks1.cloudfront.net/user_avatar/forums.unrealengine.com/deathcalibur/32/1037610_2.png) [@Deathcalibur](https://forums.unrealengine.com/u/Deathcalibur)\
**Post date:** [March 29, 2024, 3:41pm UTC](https://forums.unrealengine.com/t/tutorial-learning-to-drive/1275490/126 "2024-03-29T15:41:49Z")

</div>

That should work. You can also add the type of object hit if you have a good way to enumerate them.

Brendan

---

<div class="post-metadata">

**Author:** ![Jeebius](https://d3kjluh73b9h9o.cloudfront.net/original/4X/2/8/5/28502927f1d97ba1051b3b47ccfe17233bf41ef0.png) [@Jeebius](https://forums.unrealengine.com/u/Jeebius)\
**Post date:** [March 30, 2024, 8:46am UTC](https://forums.unrealengine.com/t/tutorial-learning-to-drive/1275490/127 "2024-03-30T08:46:28Z")

</div>

Oh awesome! So in theory, if the enumeration is in a separate observation, the agents should be able to correlate the object type to the hit distance?

---

<div class="post-metadata">

**Author:** ![Sengchor](https://d1ap1mz92jnks1.cloudfront.net/user_avatar/forums.unrealengine.com/sengchor/32/1214574_2.png) [@Sengchor](https://forums.unrealengine.com/u/Sengchor)\
**Post date:** [April 12, 2024, 2:35pm UTC](https://forums.unrealengine.com/t/tutorial-learning-to-drive/1275490/128 "2024-04-12T14:35:50Z")

</div>

Thank you for the tutorial. I want the car to avoid collisions with other cars and get racing behavior. Can you provide some ideas on how to achieve that?

---

<div class="post-metadata">

**Author:** ![Deathcalibur](https://d1ap1mz92jnks1.cloudfront.net/user_avatar/forums.unrealengine.com/deathcalibur/32/1037610_2.png) [@Deathcalibur](https://forums.unrealengine.com/u/Deathcalibur)\
**Post date:** [April 12, 2024, 2:37pm UTC](https://forums.unrealengine.com/t/tutorial-learning-to-drive/1275490/129 "2024-04-12T14:37:31Z")

</div>

I have more tutorials coming with the imminent release of 5.4. The new Learning Agents Interactor API makes this 100x easier (IMO).

Thanks!

[Previous page](https://forums.unrealengine.com/t/tutorial-learning-to-drive/1275490.md?page=5)

[Next page](https://forums.unrealengine.com/t/tutorial-learning-to-drive/1275490.md?page=7)
