Let’s walk through a quick example that shows how each piece of Learning Agents works in practice. During this tutorial, we will train a neural network to drive a sports car around a race track.
https://dev.epicgames.com/community/learning/tutorials/qj2O/unreal-engine-learning-to-drive
3 Likes
awesome. thanks, got my ai to learn how to drive.
one thing i don’t understand is the “fixed time step frequency” setting". it’s supposed to match the inference rate? but by default it’s 60hz and the inference rate in this example was 10hz right?
Hey, good question and good to hear that the tutorial worked out for you.
As you probably saw from the comments, that frequency changes the Unreal game loop so that it will run with fixed time steps instead of variable time steps. I would say that you want the frequency to match what you anticipate your game to run at. If the game runs at 60 FPS, then you want the fixed time steps to be 60 Hz. If your game runs at 30 FPS, then you would want it to be 30 Hz.
I can see how the existing comment could be unclear. We will consider rephrasing it.
Thanks!
I’m getting a warning during training
LogLearning: Display: Training Process: Creating Replay Buffer…
LogLearning: Display: Training Process: Creating Networks…
LogLearning: Display: Training Process: Sending Policy…
LogLearning: Display: Training Process: Creating Optimizer…
LogLearning: Display: Training Process: Creating PPO Policy…
LogLearning: Display: Training Process: Opening TensorBoard…
LogLearning: Display: Training Process: Begin Training…
LogLearning: Display: Training Process: Profile| Pull Experience 19580ms
LogLearning: Display: Training Process: Traceback (most recent call last):
LogLearning: Display: Training Process: File “D:\EPIC\UE_5.3\Engine\Plugins\Experimental\LearningAgents\Content\Python\train_ppo.py”, line 361, in
LogLearning: Display: Training Process: train_ppo(config, trainer)
LogLearning: Display: Training Process: File “D:\EPIC\UE_5.3\Engine\Plugins\Experimental\LearningAgents\Content\Python\train_ppo.py”, line 199, in train_ppo
LogLearning: Display: Training Process: assert response == UE_RESPONSE_SUCCESS
LogLearning: Display: Training Process: AssertionError
LogLearning: Warning: Training Process finished with warnings or errors
Then after a minute of getting this, this shows up and everything stops training
LogLearning: Error: BP_DrivingRLTrainer: Error waiting for policy from trainer. Check log for errors.
LogLearning: Display: BP_DrivingRLTrainer: Stopping training…
LogLearning: Error: BP_DrivingRLTrainer: Training has failed. Check log for errors.
LogLearning: Error: BP_DrivingRLTrainer: Training has failed. Check log for errors.
LogLearning: Error: BP_DrivingRLTrainer: Training has failed. Check log for errors.
Any ideas?
Can you see if you’re running into the same issue as this post? Learning Agents fails - #2 by Deathcalibur
Might be timing out for some reason. Unfortunately the assertion error message is not outputting the response type so need to print it.
2 Likes
If I wanted it to train all night, what would be some good settings so it doesn’t quit out too early?
I got it to stop quitting out of training by increasing The Maximum Recorded Episodes and
The Maximum Recorded Steps. Now it seems to be going, but not seeing a improvement in driving 
Edit: Also notice my DA_DrivingNetwork only saves 72 kb of information then stops there no matter how long it trains.
Really excited to learn more about the imitation learning as that seems like it could be perfect for my use cases. Any idea when this tutorial will be released?
1 Like
Ah ok that might be the cause of your timeouts. The Maximum Recorded Episodes and
Maximum Recorded Steps are per training iteration. You don’t want these values to be too large otherwise you are going to be collecting a lot of data for a specific policy version and not iterating on new weights quickly enough. The default settings of 1000 and 10000 were fine for the driving demo.
To get it to train all night, you merely should need to leave it running. In the TrainerTrainingSettings (which you pass into RunTraining) there is a property “Number of Iterations” which controls how long you would train for, but the default is 1 million iterations. If the entire iteration took 10 seconds to run, this would run for 70 days (which is super overkill).
1 Like
Thank you for being patient with me. I did the print(response) on line 195.
Reset the setting to what you said.
The training is still failing after two iterations.
Here are the logs
[2023.09.07-17.28.45:640][171]LogDebuggerCommands: Repeating last play command: Selected Viewport
[2023.09.07-17.28.45:646][171]LogContentBundle: [VehicleExampleMap(Standalone)] Generating Streaming for 0 Content Bundles.
[2023.09.07-17.28.45:646][171]LogWorldPartition: Display: GenerateStreaming for 'VehicleExampleMap' started...
[2023.09.07-17.28.45:648][171]LogWorldPartition: Display: GenerateStreaming for 'VehicleExampleMap' took 1 ms
[2023.09.07-17.28.45:648][171]LogPlayLevel: PlayLevel: No blueprints needed recompiling
[2023.09.07-17.28.45:649][171]PIE: New page: PIE session: VehicleExampleMap (Sep 7, 2023, 3:28:45 AM)
[2023.09.07-17.28.45:652][171]LogOnline: OSS: Created online subsystem instance for: NULL
[2023.09.07-17.28.45:652][171]LogOnline: OSS: TryLoadSubsystemAndSetDefault: Loaded subsystem for type [NULL]
[2023.09.07-17.28.45:652][171]LogPlayLevel: Creating play world package: /Game/VehicleTemplate/Maps/UEDPIE_0_VehicleExampleMap
[2023.09.07-17.28.45:917][171]LogPlayLevel: PIE: StaticDuplicateObject took: (0.265737s)
[2023.09.07-17.28.45:917][171]LogPlayLevel: PIE: Created PIE world by copying editor world from /Game/VehicleTemplate/Maps/VehicleExampleMap.VehicleExampleMap to /Game/VehicleTemplate/Maps/UEDPIE_0_VehicleExampleMap.VehicleExampleMap (0.265775s)
[2023.09.07-17.28.45:922][171]LogUObjectHash: Compacting FUObjectHashTables data took 0.45ms
[2023.09.07-17.28.45:922][171]LogWorldSubsystemInput: UEnhancedInputDeveloperSettings::bEnableWorldSubsystem is false, the world subsystem will not be created!
[2023.09.07-17.28.45:922][171]LogChaos: FPhysicsSolverBase::AsyncDt:-1.000000
[2023.09.07-17.28.45:923][171]LogAIModule: Creating AISystem for world VehicleExampleMap
[2023.09.07-17.28.45:923][171]LogWorldPartition: ULevel::OnLevelLoaded(VehicleExampleMap)(bIsOwningWorldGameWorld=1, bIsOwningWorldPartitioned=1, InitializeForMainWorld=1, InitializeForEditor=0, InitializeForGame=1)
[2023.09.07-17.28.45:923][171]LogWorldPartition: Display: WorldPartition initialize started...
[2023.09.07-17.28.45:923][171]LogWorldPartition: UWorldPartition::Initialize : World = VehicleExampleMap, World Type = PIE, IsMainWorldPartition = 1, Location = V(0), Rotation = R(0), IsEditor = 0, IsGame = 0, IsPIEWorldTravel = 0, IsCooking = 0
[2023.09.07-17.28.45:923][171]LogWorldPartition: UWorldPartition::Initialize Context : World NetMode = Standalone, IsServer = 0, IsDedicatedServer = 0, IsServerStreamingEnabled = 0, IsServerStreamingOutEnabled = 0, IsUsingMakingVisibleTransaction = 0, IsUsingMakingInvisibleTransaction = 0
[2023.09.07-17.28.45:943][171]LogContentBundle: [VehicleExampleMap(Standalone)] Creating new contrainer.
[2023.09.07-17.28.45:943][171]LogWorldPartition: Display: WorldPartition initialize took 19 ms (total: 384 ms)
[2023.09.07-17.28.45:943][171]LogPlayLevel: PIE: World Init took: (0.020554s)
[2023.09.07-17.28.45:943][171]LogAudio: Display: Creating Audio Device: Id: 2, Scope: Unique, Realtime: True
[2023.09.07-17.28.45:943][171]LogAudioMixer: Display: Audio Mixer Platform Settings:
[2023.09.07-17.28.45:943][171]LogAudioMixer: Display: Sample Rate: 48000
[2023.09.07-17.28.45:943][171]LogAudioMixer: Display: Callback Buffer Frame Size Requested: 1024
[2023.09.07-17.28.45:943][171]LogAudioMixer: Display: Callback Buffer Frame Size To Use: 1024
[2023.09.07-17.28.45:943][171]LogAudioMixer: Display: Number of buffers to queue: 1
[2023.09.07-17.28.45:943][171]LogAudioMixer: Display: Max Channels (voices): 32
[2023.09.07-17.28.45:943][171]LogAudioMixer: Display: Number of Async Source Workers: 4
[2023.09.07-17.28.45:943][171]LogAudio: Display: AudioDevice MaxSources: 32
[2023.09.07-17.28.45:943][171]LogAudio: Display: Audio Spatialization Plugin: None (built-in).
[2023.09.07-17.28.45:943][171]LogAudio: Display: Audio Reverb Plugin: None (built-in).
[2023.09.07-17.28.45:944][171]LogAudio: Display: Audio Occlusion Plugin: None (built-in).
[2023.09.07-17.28.45:945][171]LogAudioMixer: Display: Initializing audio mixer using platform API: 'XAudio2'
[2023.09.07-17.28.45:969][171]LogAudioMixer: Display: Using Audio Hardware Device LC49G95T (NVIDIA High Definition Audio)
[2023.09.07-17.28.45:969][171]LogAudioMixer: Display: Initializing Sound Submixes...
[2023.09.07-17.28.45:969][171]LogAudioMixer: Display: Creating Master Submix 'MasterSubmixDefault'
[2023.09.07-17.28.45:969][171]LogAudioMixer: Display: Creating Master Submix 'MasterReverbSubmixDefault'
[2023.09.07-17.28.45:970][171]LogAudioMixer: FMixerPlatformXAudio2::StartAudioStream() called. InstanceID=2
[2023.09.07-17.28.45:970][171]LogAudioMixer: Display: Output buffers initialized: Frames=1024, Channels=2, Samples=2048, InstanceID=2
[2023.09.07-17.28.45:971][171]LogAudioMixer: Display: Starting AudioMixerPlatformInterface::RunInternal(), InstanceID=2
[2023.09.07-17.28.45:971][171]LogInit: FAudioDevice initialized with ID 2.
[2023.09.07-17.28.45:971][171]LogAudioMixer: Display: FMixerPlatformXAudio2::SubmitBuffer() called for the first time. InstanceID=2
[2023.09.07-17.28.45:971][171]LogAudio: Display: Audio Device (ID: 2) registered with world 'VehicleExampleMap'.
[2023.09.07-17.28.45:971][171]LogAudioMixer: Initializing Audio Bus Subsystem for audio device with ID 2
[2023.09.07-17.28.45:974][171]LogLoad: Game class is 'VehicleAdvGameMode_C'
[2023.09.07-17.28.46:001][171]LogWorld: Bringing World /Game/VehicleTemplate/Maps/UEDPIE_0_VehicleExampleMap.VehicleExampleMap up for play (max tick rate 0) at 2023.09.07-10.28.46
[2023.09.07-17.28.46:004][171]LogWorld: Bringing up level for play took: 0.029811
[2023.09.07-17.28.46:006][171]LogOnline: OSS: Created online subsystem instance for: :Context_1
[2023.09.07-17.28.46:007][171]LogSpawn: Warning: SpawnActor failed because no class was specified
[2023.09.07-17.28.46:011][171]r.MotionBlur.Amount = "0"
[2023.09.07-17.28.46:011][171]LogLearning: Display: BP_RLTrainingManager_C_UAID_2CF05DAF86939DA601_1666759087: Adding Agent SportsCar_Pawn_C_UAID_2CF05DAF8693A2A601_1835287986 with id 0.
[2023.09.07-17.28.46:012][171]LogLearning: Display: BP_RLTrainingManager_C_UAID_2CF05DAF86939DA601_1666759087: Adding Agent SportsCar_Pawn_C_UAID_2CF05DAF8693A2A601_1829836981 with id 1.
[2023.09.07-17.28.46:012][171]LogLearning: Display: BP_RLTrainingManager_C_UAID_2CF05DAF86939DA601_1666759087: Adding Agent SportsCar_Pawn_C_UAID_2CF05DAF8693A2A601_1848134999 with id 2.
[2023.09.07-17.28.46:012][171]LogLearning: Display: BP_RLTrainingManager_C_UAID_2CF05DAF86939DA601_1666759087: Adding Agent SportsCar_Pawn_C_UAID_2CF05DAF8693A2A601_1825199978 with id 3.
[2023.09.07-17.28.46:012][171]LogLearning: Display: BP_RLTrainingManager_C_UAID_2CF05DAF86939DA601_1666759087: Adding Agent SportsCar_Pawn_C_UAID_2CF05DAF8693A2A601_1841329991 with id 4.
[2023.09.07-17.28.46:012][171]LogLearning: Display: BP_RLTrainingManager_C_UAID_2CF05DAF86939DA601_1666759087: Adding Agent SportsCar_Pawn_C_UAID_2CF05DAF8693A2A601_1844611994 with id 5.
[2023.09.07-17.28.46:012][171]LogLearning: Display: BP_RLTrainingManager_C_UAID_2CF05DAF86939DA601_1666759087: Adding Agent SportsCar_Pawn_C_UAID_2CF05DAF8693A2A601_1844619996 with id 6.
[2023.09.07-17.28.46:012][171]LogLearning: Display: BP_RLTrainingManager_C_UAID_2CF05DAF86939DA601_1666759087: Adding Agent SportsCar_Pawn_C_UAID_2CF05DAF8693A2A601_1819807975 with id 7.
[2023.09.07-17.28.46:012][171]LogLearning: Display: BP_RLTrainingManager_C_UAID_2CF05DAF86939DA601_1666759087: Adding Agent SportsCar_Pawn_C_UAID_2CF05DAF8693A2A601_1848125997 with id 8.
[2023.09.07-17.28.46:012][171]LogLearning: Display: BP_RLTrainingManager_C_UAID_2CF05DAF86939DA601_1666759087: Adding Agent SportsCar_Pawn_C_UAID_2CF05DAF8693A2A601_1784027972 with id 9.
[2023.09.07-17.28.46:013][171]LogLearning: Display: BP_RLTrainingManager_C_UAID_2CF05DAF86939DA601_1666759087: Adding Agent SportsCar_Pawn_C_UAID_2CF05DAF8693A2A601_1819803974 with id 10.
[2023.09.07-17.28.46:013][171]LogLearning: Display: BP_RLTrainingManager_C_UAID_2CF05DAF86939DA601_1666759087: Adding Agent SportsCar_Pawn_C_UAID_2CF05DAF8693A2A601_1778131970 with id 11.
[2023.09.07-17.28.46:013][171]LogLearning: Display: BP_RLTrainingManager_C_UAID_2CF05DAF86939DA601_1666759087: Adding Agent SportsCar_Pawn_C_UAID_2CF05DAF8693A2A601_1829841982 with id 12.
[2023.09.07-17.28.46:013][171]LogLearning: Display: BP_RLTrainingManager_C_UAID_2CF05DAF86939DA601_1666759087: Adding Agent SportsCar_Pawn_C_UAID_2CF05DAF8693A2A601_1825208980 with id 13.
[2023.09.07-17.28.46:013][171]LogLearning: Display: BP_RLTrainingManager_C_UAID_2CF05DAF86939DA601_1666759087: Adding Agent SportsCar_Pawn_C_UAID_2CF05DAF8693A2A601_1835293987 with id 14.
[2023.09.07-17.28.46:013][171]LogLearning: Display: BP_RLTrainingManager_C_UAID_2CF05DAF86939DA601_1666759087: Adding Agent SportsCar_Pawn_C_UAID_2CF05DAF8693A2A601_1844606993 with id 15.
[2023.09.07-17.28.46:013][171]LogLearning: Display: BP_RLTrainingManager_C_UAID_2CF05DAF86939DA601_1666759087: Adding Agent SportsCar_Pawn_C_UAID_2CF05DAF8693A2A601_1848130998 with id 16.
[2023.09.07-17.28.46:013][171]LogLearning: Display: BP_RLTrainingManager_C_UAID_2CF05DAF86939DA601_1666759087: Adding Agent SportsCar_Pawn_C_UAID_2CF05DAF8693A2A601_1829845983 with id 17.
[2023.09.07-17.28.46:014][171]LogLearning: Display: BP_RLTrainingManager_C_UAID_2CF05DAF86939DA601_1666759087: Adding Agent SportsCar_Pawn_C_UAID_2CF05DAF8693A2A601_1819799973 with id 18.
[2023.09.07-17.28.46:014][171]LogLearning: Display: BP_RLTrainingManager_C_UAID_2CF05DAF86939DA601_1666759087: Adding Agent SportsCar_Pawn_C_UAID_2CF05DAF8693A2A601_1829849984 with id 19.
[2023.09.07-17.28.46:014][171]LogLearning: Display: BP_RLTrainingManager_C_UAID_2CF05DAF86939DA601_1666759087: Adding Agent SportsCar_Pawn_C_UAID_2CF05DAF8693A2A601_1780584971 with id 20.
[2023.09.07-17.28.46:014][171]LogLearning: Display: BP_RLTrainingManager_C_UAID_2CF05DAF86939DA601_1666759087: Adding Agent SportsCar_Pawn_C_UAID_2CF05DAF8693A2A601_1841333992 with id 21.
[2023.09.07-17.28.46:014][171]LogLearning: Display: BP_RLTrainingManager_C_UAID_2CF05DAF86939DA601_1666759087: Adding Agent SportsCar_Pawn_C_UAID_2CF05DAF8693A2A601_1841325990 with id 22.
[2023.09.07-17.28.46:014][171]LogLearning: Display: BP_RLTrainingManager_C_UAID_2CF05DAF86939DA601_1666759087: Adding Agent SportsCar_Pawn_C_UAID_2CF05DAF8693A2A601_1766075969 with id 23.
[2023.09.07-17.28.46:014][171]LogLearning: Display: BP_RLTrainingManager_C_UAID_2CF05DAF86939DA601_1666759087: Adding Agent SportsCar_Pawn_C_UAID_2CF05DAF8693A2A601_1844615995 with id 24.
[2023.09.07-17.28.46:015][171]LogLearning: Display: BP_RLTrainingManager_C_UAID_2CF05DAF86939DA601_1666759087: Adding Agent SportsCar_Pawn_C_UAID_2CF05DAF8693A2A601_1835281985 with id 25.
[2023.09.07-17.28.46:015][171]LogLearning: Display: BP_RLTrainingManager_C_UAID_2CF05DAF86939DA601_1666759087: Adding Agent SportsCar_Pawn_C_UAID_2CF05DAF8693A2A601_1825195977 with id 26.
[2023.09.07-17.28.46:015][171]LogLearning: Display: BP_RLTrainingManager_C_UAID_2CF05DAF86939DA601_1666759087: Adding Agent SportsCar_Pawn_C_UAID_2CF05DAF8693A2A601_1825204979 with id 27.
[2023.09.07-17.28.46:015][171]LogLearning: Display: BP_RLTrainingManager_C_UAID_2CF05DAF86939DA601_1666759087: Adding Agent SportsCar_Pawn_C_UAID_2CF05DAF8693A2A601_1841320989 with id 28.
[2023.09.07-17.28.46:015][171]LogLearning: Display: BP_RLTrainingManager_C_UAID_2CF05DAF86939DA601_1666759087: Adding Agent SportsCar_Pawn_C_UAID_2CF05DAF8693A2A601_1819811976 with id 29.
[2023.09.07-17.28.46:015][171]LogLearning: Display: BP_RLTrainingManager_C_UAID_2CF05DAF86939DA601_1666759087: Adding Agent SportsCar_Pawn_C_UAID_2CF05DAF8693A2A601_1835297988 with id 30.
[2023.09.07-17.28.46:016][171]LogLearning: Display: BP_RLTrainingManager_C_UAID_2CF05DAF86939DA601_1666759087: Adding Agent SportsCar_Pawn_C_0 with id 31.
[2023.09.07-17.28.46:016][171]PIE: Server logged in
[2023.09.07-17.28.46:017][171]PIE: Play in editor total start time 0.371 seconds.
[2023.09.07-17.28.46:028][171]LogLearning: Display: BP_DrivingRLTrainer: Sending / Receiving initial policy...
[2023.09.07-17.28.47:020][171]LogLearning: Display: Training Process: {
[2023.09.07-17.28.47:020][171]LogLearning: Display: Training Process: "TaskName": "BP_DrivingRLTrainer",
[2023.09.07-17.28.47:020][171]LogLearning: Display: Training Process: "TrainerMethod": "PPO",
[2023.09.07-17.28.47:020][171]LogLearning: Display: Training Process: "TrainerType": "SharedMemory",
[2023.09.07-17.28.47:020][171]LogLearning: Display: Training Process: "TimeStamp": "2023-09-07_10-28-46",
[2023.09.07-17.28.47:020][171]LogLearning: Display: Training Process: "SitePackagesPath": "D:/EPIC/UE_5.3/Engine/Plugins/Experimental/PythonFoundationPackages/Content/Python/Lib/Win64/site-packages",
[2023.09.07-17.28.47:020][171]LogLearning: Display: Training Process: "IntermediatePath": "D:/Unreal/LearningAgents/Intermediate/LearningAgents",
[2023.09.07-17.28.47:020][171]LogLearning: Display: Training Process: "PolicyGuid": "{280EAD29-41D1-B46C-A492-EEA5CB279674}",
[2023.09.07-17.28.47:020][171]LogLearning: Display: Training Process: "ControlsGuid": "{A0E3F191-4B53-1304-3FB6-1C9496F7F7CD}",
[2023.09.07-17.28.47:020][171]LogLearning: Display: Training Process: "EpisodeStartsGuid": "{9C3DC187-4CC9-DF5A-C82F-98814EDB4A3A}",
[2023.09.07-17.28.47:020][171]LogLearning: Display: Training Process: "EpisodeLengthsGuid": "{4E878D45-4BDF-8CB9-F162-AEAE3F053BFC}",
[2023.09.07-17.28.47:020][171]LogLearning: Display: Training Process: "EpisodeCompletionModesGuid": "{F19BB3C7-488F-1B5F-0D1B-0AACD55E2C5A}",
[2023.09.07-17.28.47:020][171]LogLearning: Display: Training Process: "EpisodeFinalObservationsGuid": "{D6F2C79F-40FA-4134-5182-65A0A700650D}",
[2023.09.07-17.28.47:020][171]LogLearning: Display: Training Process: "ObservationsGuid": "{576CDED1-4AF5-F1EA-02C1-2DA7411F3FE9}",
[2023.09.07-17.28.47:020][171]LogLearning: Display: Training Process: "ActionsGuid": "{6D8D2EE1-4754-5B04-C143-789D600E72EA}",
[2023.09.07-17.28.47:020][171]LogLearning: Display: Training Process: "RewardsGuid": "{CFF7875C-4076-F442-B158-76B72D4E4AD3}",
[2023.09.07-17.28.47:020][171]LogLearning: Display: Training Process: "ObservationVectorDimensionNum": 8,
[2023.09.07-17.28.47:020][171]LogLearning: Display: Training Process: "ActionVectorDimensionNum": 2,
[2023.09.07-17.28.47:020][171]LogLearning: Display: Training Process: "MaxEpisodeNum": 1000,
[2023.09.07-17.28.47:020][171]LogLearning: Display: Training Process: "MaxStepNum": 10000,
[2023.09.07-17.28.47:020][171]LogLearning: Display: Training Process: "PolicyNetworkByteNum": 72788,
[2023.09.07-17.28.47:020][171]LogLearning: Display: Training Process: "PolicyHiddenUnitNum": 128,
[2023.09.07-17.28.47:020][171]LogLearning: Display: Training Process: "PolicyLayerNum": 3,
[2023.09.07-17.28.47:020][171]LogLearning: Display: Training Process: "PolicyActivationFunction": "ELU",
[2023.09.07-17.28.47:020][171]LogLearning: Display: Training Process: "PolicyActionNoiseMin": 0.25,
[2023.09.07-17.28.47:020][171]LogLearning: Display: Training Process: "PolicyActionNoiseMax": 0.25,
[2023.09.07-17.28.47:020][171]LogLearning: Display: Training Process: "CriticNetworkByteNum": 71240,
[2023.09.07-17.28.47:020][171]LogLearning: Display: Training Process: "CriticHiddenUnitNum": 128,
[2023.09.07-17.28.47:020][171]LogLearning: Display: Training Process: "CriticLayerNum": 3,
[2023.09.07-17.28.47:020][171]LogLearning: Display: Training Process: "CriticActivationFunction": "ELU",
[2023.09.07-17.28.47:020][171]LogLearning: Display: Training Process: "ProcessNum": 1,
[2023.09.07-17.28.47:020][171]LogLearning: Display: Training Process: "IterationNum": 1000000,
[2023.09.07-17.28.47:020][171]LogLearning: Display: Training Process: "LearningRatePolicy": 9.999999747378752e-05,
[2023.09.07-17.28.47:020][171]LogLearning: Display: Training Process: "LearningRateCritic": 0.0010000000474974513,
[2023.09.07-17.28.47:020][171]LogLearning: Display: Training Process: "LearningRateDecay": 0.9900000095367432,
[2023.09.07-17.28.47:020][171]LogLearning: Display: Training Process: "WeightDecay": 0.0010000000474974513,
[2023.09.07-17.28.47:020][171]LogLearning: Display: Training Process: "InitialActionScale": 0.10000000149011612,
[2023.09.07-17.28.47:020][171]LogLearning: Display: Training Process: "BatchSize": 128,
[2023.09.07-17.28.47:020][171]LogLearning: Display: Training Process: "EpsilonClip": 0.20000000298023224,
[2023.09.07-17.28.47:020][171]LogLearning: Display: Training Process: "ActionRegularizationWeight": 0.0010000000474974513,
[2023.09.07-17.28.47:020][171]LogLearning: Display: Training Process: "EntropyWeight": 0.009999999776482582,
[2023.09.07-17.28.47:020][171]LogLearning: Display: Training Process: "GaeLambda": 0.8999999761581421,
[2023.09.07-17.28.47:020][171]LogLearning: Display: Training Process: "ClipAdvantages": true,
[2023.09.07-17.28.47:020][171]LogLearning: Display: Training Process: "AdvantageNormalization": true,
[2023.09.07-17.28.47:020][171]LogLearning: Display: Training Process: "TrimEpisodeStartStepNum": 0,
[2023.09.07-17.28.47:021][171]LogLearning: Display: Training Process: "TrimEpisodeEndStepNum": 0,
[2023.09.07-17.28.47:021][171]LogLearning: Display: Training Process: "Seed": 1234,
[2023.09.07-17.28.47:021][171]LogLearning: Display: Training Process: "DiscountFactor": 0.9900000095367432,
[2023.09.07-17.28.47:021][171]LogLearning: Display: Training Process: "Device": "GPU",
[2023.09.07-17.28.47:021][171]LogLearning: Display: Training Process: "UseTensorBoard": false,
[2023.09.07-17.28.47:021][171]LogLearning: Display: Training Process: "UseInitialPolicyNetwork": true,
[2023.09.07-17.28.47:021][171]LogLearning: Display: Training Process: "UseInitialCriticNetwork": false,
[2023.09.07-17.28.47:021][171]LogLearning: Display: Training Process: "SynchronizeCriticNetwork": false,
[2023.09.07-17.28.47:021][171]LogLearning: Display: Training Process: "LoggingEnabled": true
[2023.09.07-17.28.47:021][171]LogLearning: Display: Training Process: }
[2023.09.07-17.28.48:493][171]LogLearning: Display: BP_RLTrainingManager_C_UAID_2CF05DAF86939DA601_1666759087: Resetting Agents [0 1 2 3 4 5 6 7 ... 31 30 29 28 27 26 25 24].
[2023.09.07-17.28.51:976][333]LogSlate: Updating window title bar state: overlay mode, drag disabled, window buttons hidden, title bar hidden
[2023.09.07-17.28.51:976][333]LogWorld: BeginTearingDown for /Game/VehicleTemplate/Maps/UEDPIE_0_VehicleExampleMap
[2023.09.07-17.28.51:976][333]LogLearning: Display: BP_DrivingRLTrainer: Stopping training...
[2023.09.07-17.28.51:982][333]LogLearning: Display: Training Process: Creating Replay Buffer...
[2023.09.07-17.28.51:982][333]LogLearning: Display: Training Process: Creating Networks...
[2023.09.07-17.28.51:982][333]LogLearning: Display: Training Process: Receiving Policy...
[2023.09.07-17.28.51:982][333]LogLearning: Display: Training Process: Creating Optimizer...
[2023.09.07-17.28.51:982][333]LogLearning: Display: Training Process: Creating PPO Policy...
[2023.09.07-17.28.51:982][333]LogLearning: Display: Training Process: Opening TensorBoard...
[2023.09.07-17.28.51:982][333]LogLearning: Display: Training Process: Begin Training...
[2023.09.07-17.28.51:982][333]LogLearning: Display: Training Process: Profile| Pull Experience 3482ms
[2023.09.07-17.28.51:982][333]LogLearning: Display: Training Process: 3
[2023.09.07-17.28.51:982][333]LogLearning: Display: Training Process: Done!
[2023.09.07-17.28.51:982][333]LogLearning: Display: Training Process: Exiting...
[2023.09.07-17.28.52:192][333]LogWorld: UWorld::CleanupWorld for VehicleExampleMap, bSessionEnded=true, bCleanupResources=true
[2023.09.07-17.28.52:193][333]LogSlate: InvalidateAllWidgets triggered. All widgets were invalidated
[2023.09.07-17.28.52:193][333]LogContentBundle: [VehicleExampleMap(Standalone)] Deleting container.
[2023.09.07-17.28.52:196][333]LogPlayLevel: Display: Shutting down PIE online subsystems
[2023.09.07-17.28.52:198][333]LogSlate: InvalidateAllWidgets triggered. All widgets were invalidated
[2023.09.07-17.28.52:203][333]LogAudio: Display: Audio Device unregistered from world 'None'.
[2023.09.07-17.28.52:203][333]LogAudioMixer: Deinitializing Audio Bus Subsystem for audio device with ID 2
[2023.09.07-17.28.52:204][333]LogAudioMixer: FMixerPlatformXAudio2::StopAudioStream() called. InstanceID=2
[2023.09.07-17.28.52:205][333]LogAudioMixer: FMixerPlatformXAudio2::StopAudioStream() called. InstanceID=2
[2023.09.07-17.28.52:209][333]LogSlate: Updating window title bar state: overlay mode, drag disabled, window buttons hidden, title bar hidden
[2023.09.07-17.28.52:216][333]LogUObjectHash: Compacting FUObjectHashTables data took 0.53ms
[2023.09.07-17.28.52:329][341]LogPlayLevel: Display: Destroying online subsystem :Context_1
[2023.09.07-17.28.58:867][780]LogDebuggerCommands: Repeating last play command: Selected Viewport
[2023.09.07-17.28.58:873][780]LogContentBundle: [VehicleExampleMap(Standalone)] Generating Streaming for 0 Content Bundles.
[2023.09.07-17.28.58:873][780]LogWorldPartition: Display: GenerateStreaming for 'VehicleExampleMap' started...
[2023.09.07-17.28.58:875][780]LogWorldPartition: Display: GenerateStreaming for 'VehicleExampleMap' took 2 ms (total: 3 ms)
[2023.09.07-17.28.58:875][780]LogPlayLevel: PlayLevel: No blueprints needed recompiling
[2023.09.07-17.28.58:875][780]PIE: New page: PIE session: VehicleExampleMap (Sep 7, 2023, 3:28:58 AM)
[2023.09.07-17.28.58:875][780]LogPlayLevel: Creating play world package: /Game/VehicleTemplate/Maps/UEDPIE_0_VehicleExampleMap
[2023.09.07-17.28.59:143][780]LogPlayLevel: PIE: StaticDuplicateObject took: (0.268329s)
[2023.09.07-17.28.59:143][780]LogPlayLevel: PIE: Created PIE world by copying editor world from /Game/VehicleTemplate/Maps/VehicleExampleMap.VehicleExampleMap to /Game/VehicleTemplate/Maps/UEDPIE_0_VehicleExampleMap.VehicleExampleMap (0.268364s)
[2023.09.07-17.28.59:147][780]LogUObjectHash: Compacting FUObjectHashTables data took 0.46ms
[2023.09.07-17.28.59:148][780]LogWorldSubsystemInput: UEnhancedInputDeveloperSettings::bEnableWorldSubsystem is false, the world subsystem will not be created!
[2023.09.07-17.28.59:148][780]LogChaos: FPhysicsSolverBase::AsyncDt:-1.000000
[2023.09.07-17.28.59:149][780]LogAIModule: Creating AISystem for world VehicleExampleMap
[2023.09.07-17.28.59:149][780]LogWorldPartition: ULevel::OnLevelLoaded(VehicleExampleMap)(bIsOwningWorldGameWorld=1, bIsOwningWorldPartitioned=1, InitializeForMainWorld=1, InitializeForEditor=0, InitializeForGame=1)
[2023.09.07-17.28.59:149][780]LogWorldPartition: Display: WorldPartition initialize started...
[2023.09.07-17.28.59:149][780]LogWorldPartition: UWorldPartition::Initialize : World = VehicleExampleMap, World Type = PIE, IsMainWorldPartition = 1, Location = V(0), Rotation = R(0), IsEditor = 0, IsGame = 0, IsPIEWorldTravel = 0, IsCooking = 0
[2023.09.07-17.28.59:149][780]LogWorldPartition: UWorldPartition::Initialize Context : World NetMode = Standalone, IsServer = 0, IsDedicatedServer = 0, IsServerStreamingEnabled = 0, IsServerStreamingOutEnabled = 0, IsUsingMakingVisibleTransaction = 0, IsUsingMakingInvisibleTransaction = 0
[2023.09.07-17.28.59:168][780]LogContentBundle: [VehicleExampleMap(Standalone)] Creating new contrainer.
[2023.09.07-17.28.59:168][780]LogWorldPartition: Display: WorldPartition initialize took 18 ms (total: 402 ms)
[2023.09.07-17.28.59:168][780]LogPlayLevel: PIE: World Init took: (0.019527s)
[2023.09.07-17.28.59:168][780]LogAudio: Display: Creating Audio Device: Id: 3, Scope: Unique, Realtime: True
[2023.09.07-17.28.59:168][780]LogAudioMixer: Display: Audio Mixer Platform Settings:
[2023.09.07-17.28.59:168][780]LogAudioMixer: Display: Sample Rate: 48000
[2023.09.07-17.28.59:168][780]LogAudioMixer: Display: Callback Buffer Frame Size Requested: 1024
[2023.09.07-17.28.59:168][780]LogAudioMixer: Display: Callback Buffer Frame Size To Use: 1024
[2023.09.07-17.28.59:168][780]LogAudioMixer: Display: Number of buffers to queue: 1
[2023.09.07-17.28.59:168][780]LogAudioMixer: Display: Max Channels (voices): 32
[2023.09.07-17.28.59:168][780]LogAudioMixer: Display: Number of Async Source Workers: 4
[2023.09.07-17.28.59:168][780]LogAudio: Display: AudioDevice MaxSources: 32
[2023.09.07-17.28.59:169][780]LogAudio: Display: Audio Spatialization Plugin: None (built-in).
[2023.09.07-17.28.59:169][780]LogAudio: Display: Audio Reverb Plugin: None (built-in).
[2023.09.07-17.28.59:169][780]LogAudio: Display: Audio Occlusion Plugin: None (built-in).
[2023.09.07-17.28.59:171][780]LogAudioMixer: Display: Initializing audio mixer using platform API: 'XAudio2'
[2023.09.07-17.28.59:199][780]LogAudioMixer: Display: Using Audio Hardware Device LC49G95T (NVIDIA High Definition Audio)
[2023.09.07-17.28.59:199][780]LogAudioMixer: Display: Initializing Sound Submixes...
[2023.09.07-17.28.59:199][780]LogAudioMixer: Display: Creating Master Submix 'MasterSubmixDefault'
[2023.09.07-17.28.59:199][780]LogAudioMixer: Display: Creating Master Submix 'MasterReverbSubmixDefault'
[2023.09.07-17.28.59:199][780]LogAudioMixer: FMixerPlatformXAudio2::StartAudioStream() called. InstanceID=3
[2023.09.07-17.28.59:199][780]LogAudioMixer: Display: Output buffers initialized: Frames=1024, Channels=2, Samples=2048, InstanceID=3
[2023.09.07-17.28.59:201][780]LogAudioMixer: Display: Starting AudioMixerPlatformInterface::RunInternal(), InstanceID=3
[2023.09.07-17.28.59:201][780]LogAudioMixer: Display: FMixerPlatformXAudio2::SubmitBuffer() called for the first time. InstanceID=3
[2023.09.07-17.28.59:201][780]LogInit: FAudioDevice initialized with ID 3.
[2023.09.07-17.28.59:201][780]LogAudio: Display: Audio Device (ID: 3) registered with world 'VehicleExampleMap'.
[2023.09.07-17.28.59:201][780]LogAudioMixer: Initializing Audio Bus Subsystem for audio device with ID 3
[2023.09.07-17.28.59:203][780]LogLoad: Game class is 'VehicleAdvGameMode_C'
[2023.09.07-17.28.59:230][780]LogWorld: Bringing World /Game/VehicleTemplate/Maps/UEDPIE_0_VehicleExampleMap.VehicleExampleMap up for play (max tick rate 0) at 2023.09.07-10.28.59
[2023.09.07-17.28.59:233][780]LogWorld: Bringing up level for play took: 0.029216
[2023.09.07-17.28.59:235][780]LogOnline: OSS: Created online subsystem instance for: :Context_2
[2023.09.07-17.28.59:235][780]LogSpawn: Warning: SpawnActor failed because no class was specified
[2023.09.07-17.28.59:238][780]r.MotionBlur.Amount = "0"
[2023.09.07-17.28.59:238][780]LogLearning: Display: BP_RLTrainingManager_C_UAID_2CF05DAF86939DA601_1666759087: Adding Agent SportsCar_Pawn_C_UAID_2CF05DAF8693A2A601_1835287986 with id 0.
[2023.09.07-17.28.59:238][780]LogLearning: Display: BP_RLTrainingManager_C_UAID_2CF05DAF86939DA601_1666759087: Adding Agent SportsCar_Pawn_C_UAID_2CF05DAF8693A2A601_1829836981 with id 1.
[2023.09.07-17.28.59:238][780]LogLearning: Display: BP_RLTrainingManager_C_UAID_2CF05DAF86939DA601_1666759087: Adding Agent SportsCar_Pawn_C_UAID_2CF05DAF8693A2A601_1848134999 with id 2.
[2023.09.07-17.28.59:238][780]LogLearning: Display: BP_RLTrainingManager_C_UAID_2CF05DAF86939DA601_1666759087: Adding Agent SportsCar_Pawn_C_UAID_2CF05DAF8693A2A601_1825199978 with id 3.
[2023.09.07-17.28.59:238][780]LogLearning: Display: BP_RLTrainingManager_C_UAID_2CF05DAF86939DA601_1666759087: Adding Agent SportsCar_Pawn_C_UAID_2CF05DAF8693A2A601_1841329991 with id 4.
[2023.09.07-17.28.59:238][780]LogLearning: Display: BP_RLTrainingManager_C_UAID_2CF05DAF86939DA601_1666759087: Adding Agent SportsCar_Pawn_C_UAID_2CF05DAF8693A2A601_1844611994 with id 5.
[2023.09.07-17.28.59:238][780]LogLearning: Display: BP_RLTrainingManager_C_UAID_2CF05DAF86939DA601_1666759087: Adding Agent SportsCar_Pawn_C_UAID_2CF05DAF8693A2A601_1844619996 with id 6.
[2023.09.07-17.28.59:238][780]LogLearning: Display: BP_RLTrainingManager_C_UAID_2CF05DAF86939DA601_1666759087: Adding Agent SportsCar_Pawn_C_UAID_2CF05DAF8693A2A601_1819807975 with id 7.
[2023.09.07-17.28.59:239][780]LogLearning: Display: BP_RLTrainingManager_C_UAID_2CF05DAF86939DA601_1666759087: Adding Agent SportsCar_Pawn_C_UAID_2CF05DAF8693A2A601_1848125997 with id 8.
[2023.09.07-17.28.59:239][780]LogLearning: Display: BP_RLTrainingManager_C_UAID_2CF05DAF86939DA601_1666759087: Adding Agent SportsCar_Pawn_C_UAID_2CF05DAF8693A2A601_1784027972 with id 9.
[2023.09.07-17.28.59:239][780]LogLearning: Display: BP_RLTrainingManager_C_UAID_2CF05DAF86939DA601_1666759087: Adding Agent SportsCar_Pawn_C_UAID_2CF05DAF8693A2A601_1819803974 with id 10.
[2023.09.07-17.28.59:239][780]LogLearning: Display: BP_RLTrainingManager_C_UAID_2CF05DAF86939DA601_1666759087: Adding Agent SportsCar_Pawn_C_UAID_2CF05DAF8693A2A601_1778131970 with id 11.
[2023.09.07-17.28.59:239][780]LogLearning: Display: BP_RLTrainingManager_C_UAID_2CF05DAF86939DA601_1666759087: Adding Agent SportsCar_Pawn_C_UAID_2CF05DAF8693A2A601_1829841982 with id 12.
[2023.09.07-17.28.59:239][780]LogLearning: Display: BP_RLTrainingManager_C_UAID_2CF05DAF86939DA601_1666759087: Adding Agent SportsCar_Pawn_C_UAID_2CF05DAF8693A2A601_1825208980 with id 13.
[2023.09.07-17.28.59:239][780]LogLearning: Display: BP_RLTrainingManager_C_UAID_2CF05DAF86939DA601_1666759087: Adding Agent SportsCar_Pawn_C_UAID_2CF05DAF8693A2A601_1835293987 with id 14.
[2023.09.07-17.28.59:239][780]LogLearning: Display: BP_RLTrainingManager_C_UAID_2CF05DAF86939DA601_1666759087: Adding Agent SportsCar_Pawn_C_UAID_2CF05DAF8693A2A601_1844606993 with id 15.
[2023.09.07-17.28.59:240][780]LogLearning: Display: BP_RLTrainingManager_C_UAID_2CF05DAF86939DA601_1666759087: Adding Agent SportsCar_Pawn_C_UAID_2CF05DAF8693A2A601_1848130998 with id 16.
[2023.09.07-17.28.59:240][780]LogLearning: Display: BP_RLTrainingManager_C_UAID_2CF05DAF86939DA601_1666759087: Adding Agent SportsCar_Pawn_C_UAID_2CF05DAF8693A2A601_1829845983 with id 17.
[2023.09.07-17.28.59:240][780]LogLearning: Display: BP_RLTrainingManager_C_UAID_2CF05DAF86939DA601_1666759087: Adding Agent SportsCar_Pawn_C_UAID_2CF05DAF8693A2A601_1819799973 with id 18.
[2023.09.07-17.28.59:240][780]LogLearning: Display: BP_RLTrainingManager_C_UAID_2CF05DAF86939DA601_1666759087: Adding Agent SportsCar_Pawn_C_UAID_2CF05DAF8693A2A601_1829849984 with id 19.
[2023.09.07-17.28.59:240][780]LogLearning: Display: BP_RLTrainingManager_C_UAID_2CF05DAF86939DA601_1666759087: Adding Agent SportsCar_Pawn_C_UAID_2CF05DAF8693A2A601_1780584971 with id 20.
[2023.09.07-17.28.59:240][780]LogLearning: Display: BP_RLTrainingManager_C_UAID_2CF05DAF86939DA601_1666759087: Adding Agent SportsCar_Pawn_C_UAID_2CF05DAF8693A2A601_1841333992 with id 21.
[2023.09.07-17.28.59:240][780]LogLearning: Display: BP_RLTrainingManager_C_UAID_2CF05DAF86939DA601_1666759087: Adding Agent SportsCar_Pawn_C_UAID_2CF05DAF8693A2A601_1841325990 with id 22.
[2023.09.07-17.28.59:240][780]LogLearning: Display: BP_RLTrainingManager_C_UAID_2CF05DAF86939DA601_1666759087: Adding Agent SportsCar_Pawn_C_UAID_2CF05DAF8693A2A601_1766075969 with id 23.
[2023.09.07-17.28.59:241][780]LogLearning: Display: BP_RLTrainingManager_C_UAID_2CF05DAF86939DA601_1666759087: Adding Agent SportsCar_Pawn_C_UAID_2CF05DAF8693A2A601_1844615995 with id 24.
[2023.09.07-17.28.59:241][780]LogLearning: Display: BP_RLTrainingManager_C_UAID_2CF05DAF86939DA601_1666759087: Adding Agent SportsCar_Pawn_C_UAID_2CF05DAF8693A2A601_1835281985 with id 25.
[2023.09.07-17.28.59:241][780]LogLearning: Display: BP_RLTrainingManager_C_UAID_2CF05DAF86939DA601_1666759087: Adding Agent SportsCar_Pawn_C_UAID_2CF05DAF8693A2A601_1825195977 with id 26.
[2023.09.07-17.28.59:241][780]LogLearning: Display: BP_RLTrainingManager_C_UAID_2CF05DAF86939DA601_1666759087: Adding Agent SportsCar_Pawn_C_UAID_2CF05DAF8693A2A601_1825204979 with id 27.
[2023.09.07-17.28.59:241][780]LogLearning: Display: BP_RLTrainingManager_C_UAID_2CF05DAF86939DA601_1666759087: Adding Agent SportsCar_Pawn_C_UAID_2CF05DAF8693A2A601_1841320989 with id 28.
[2023.09.07-17.28.59:241][780]LogLearning: Display: BP_RLTrainingManager_C_UAID_2CF05DAF86939DA601_1666759087: Adding Agent SportsCar_Pawn_C_UAID_2CF05DAF8693A2A601_1819811976 with id 29.
[2023.09.07-17.28.59:241][780]LogLearning: Display: BP_RLTrainingManager_C_UAID_2CF05DAF86939DA601_1666759087: Adding Agent SportsCar_Pawn_C_UAID_2CF05DAF8693A2A601_1835297988 with id 30.
[2023.09.07-17.28.59:242][780]LogLearning: Display: BP_RLTrainingManager_C_UAID_2CF05DAF86939DA601_1666759087: Adding Agent SportsCar_Pawn_C_0 with id 31.
[2023.09.07-17.28.59:242][780]PIE: Server logged in
[2023.09.07-17.28.59:243][780]PIE: Play in editor total start time 0.37 seconds.
[2023.09.07-17.28.59:271][781]LogLearning: Display: BP_DrivingRLTrainer: Sending / Receiving initial policy...
[2023.09.07-17.29.00:186][781]LogLearning: Display: Training Process: {
[2023.09.07-17.29.00:186][781]LogLearning: Display: Training Process: "TaskName": "BP_DrivingRLTrainer",
[2023.09.07-17.29.00:186][781]LogLearning: Display: Training Process: "TrainerMethod": "PPO",
[2023.09.07-17.29.00:186][781]LogLearning: Display: Training Process: "TrainerType": "SharedMemory",
[2023.09.07-17.29.00:186][781]LogLearning: Display: Training Process: "TimeStamp": "2023-09-07_10-28-59",
[2023.09.07-17.29.00:186][781]LogLearning: Display: Training Process: "SitePackagesPath": "D:/EPIC/UE_5.3/Engine/Plugins/Experimental/PythonFoundationPackages/Content/Python/Lib/Win64/site-packages",
[2023.09.07-17.29.00:186][781]LogLearning: Display: Training Process: "IntermediatePath": "D:/Unreal/LearningAgents/Intermediate/LearningAgents",
[2023.09.07-17.29.00:186][781]LogLearning: Display: Training Process: "PolicyGuid": "{15B0E014-4536-8275-0E81-CF9CF2B74B60}",
[2023.09.07-17.29.00:186][781]LogLearning: Display: Training Process: "ControlsGuid": "{2C0019DE-4775-E1E7-8F72-4883C7BEBF37}",
[2023.09.07-17.29.00:186][781]LogLearning: Display: Training Process: "EpisodeStartsGuid": "{0DAF694A-4DAC-4146-E03F-F29EB52B7DF2}",
[2023.09.07-17.29.00:186][781]LogLearning: Display: Training Process: "EpisodeLengthsGuid": "{B0164A60-41B6-D507-73FA-448C791ACA60}",
[2023.09.07-17.29.00:186][781]LogLearning: Display: Training Process: "EpisodeCompletionModesGuid": "{EE3BDE90-49FA-36D7-EAA0-B49FDC50DE73}",
[2023.09.07-17.29.00:186][781]LogLearning: Display: Training Process: "EpisodeFinalObservationsGuid": "{2EAB5A16-43CE-4361-CAF5-F2BA92DC3506}",
[2023.09.07-17.29.00:186][781]LogLearning: Display: Training Process: "ObservationsGuid": "{CD8400B0-4D72-2BFA-BB18-7996A9D9DAF8}",
[2023.09.07-17.29.00:186][781]LogLearning: Display: Training Process: "ActionsGuid": "{EBE55992-4FC4-A259-ABA0-C0BAE22B9785}",
[2023.09.07-17.29.00:186][781]LogLearning: Display: Training Process: "RewardsGuid": "{1B417F82-473D-34DD-C2CD-15B8A45E9BCF}",
[2023.09.07-17.29.00:186][781]LogLearning: Display: Training Process: "ObservationVectorDimensionNum": 8,
[2023.09.07-17.29.00:186][781]LogLearning: Display: Training Process: "ActionVectorDimensionNum": 2,
[2023.09.07-17.29.00:186][781]LogLearning: Display: Training Process: "MaxEpisodeNum": 1000,
[2023.09.07-17.29.00:186][781]LogLearning: Display: Training Process: "MaxStepNum": 10000,
[2023.09.07-17.29.00:186][781]LogLearning: Display: Training Process: "PolicyNetworkByteNum": 72788,
[2023.09.07-17.29.00:186][781]LogLearning: Display: Training Process: "PolicyHiddenUnitNum": 128,
[2023.09.07-17.29.00:186][781]LogLearning: Display: Training Process: "PolicyLayerNum": 3,
[2023.09.07-17.29.00:186][781]LogLearning: Display: Training Process: "PolicyActivationFunction": "ELU",
[2023.09.07-17.29.00:186][781]LogLearning: Display: Training Process: "PolicyActionNoiseMin": 0.25,
[2023.09.07-17.29.00:186][781]LogLearning: Display: Training Process: "PolicyActionNoiseMax": 0.25,
[2023.09.07-17.29.00:186][781]LogLearning: Display: Training Process: "CriticNetworkByteNum": 71240,
[2023.09.07-17.29.00:186][781]LogLearning: Display: Training Process: "CriticHiddenUnitNum": 128,
[2023.09.07-17.29.00:186][781]LogLearning: Display: Training Process: "CriticLayerNum": 3,
[2023.09.07-17.29.00:186][781]LogLearning: Display: Training Process: "CriticActivationFunction": "ELU",
[2023.09.07-17.29.00:186][781]LogLearning: Display: Training Process: "ProcessNum": 1,
[2023.09.07-17.29.00:186][781]LogLearning: Display: Training Process: "IterationNum": 1000000,
[2023.09.07-17.29.00:186][781]LogLearning: Display: Training Process: "LearningRatePolicy": 9.999999747378752e-05,
[2023.09.07-17.29.00:186][781]LogLearning: Display: Training Process: "LearningRateCritic": 0.0010000000474974513,
[2023.09.07-17.29.00:186][781]LogLearning: Display: Training Process: "LearningRateDecay": 0.9900000095367432,
[2023.09.07-17.29.00:186][781]LogLearning: Display: Training Process: "WeightDecay": 0.0010000000474974513,
[2023.09.07-17.29.00:186][781]LogLearning: Display: Training Process: "InitialActionScale": 0.10000000149011612,
[2023.09.07-17.29.00:186][781]LogLearning: Display: Training Process: "BatchSize": 128,
[2023.09.07-17.29.00:186][781]LogLearning: Display: Training Process: "EpsilonClip": 0.20000000298023224,
[2023.09.07-17.29.00:186][781]LogLearning: Display: Training Process: "ActionRegularizationWeight": 0.0010000000474974513,
[2023.09.07-17.29.00:186][781]LogLearning: Display: Training Process: "EntropyWeight": 0.009999999776482582,
[2023.09.07-17.29.00:186][781]LogLearning: Display: Training Process: "GaeLambda": 0.8999999761581421,
[2023.09.07-17.29.00:186][781]LogLearning: Display: Training Process: "ClipAdvantages": true,
[2023.09.07-17.29.00:186][781]LogLearning: Display: Training Process: "AdvantageNormalization": true,
[2023.09.07-17.29.00:186][781]LogLearning: Display: Training Process: "TrimEpisodeStartStepNum": 0,
[2023.09.07-17.29.00:186][781]LogLearning: Display: Training Process: "TrimEpisodeEndStepNum": 0,
[2023.09.07-17.29.00:186][781]LogLearning: Display: Training Process: "Seed": 1234,
[2023.09.07-17.29.00:186][781]LogLearning: Display: Training Process: "DiscountFactor": 0.9900000095367432,
[2023.09.07-17.29.00:186][781]LogLearning: Display: Training Process: "Device": "GPU",
[2023.09.07-17.29.00:186][781]LogLearning: Display: Training Process: "UseTensorBoard": false,
[2023.09.07-17.29.00:186][781]LogLearning: Display: Training Process: "UseInitialPolicyNetwork": true,
[2023.09.07-17.29.00:186][781]LogLearning: Display: Training Process: "UseInitialCriticNetwork": false,
[2023.09.07-17.29.00:186][781]LogLearning: Display: Training Process: "SynchronizeCriticNetwork": false,
[2023.09.07-17.29.00:186][781]LogLearning: Display: Training Process: "LoggingEnabled": true
[2023.09.07-17.29.00:186][781]LogLearning: Display: Training Process: }
[2023.09.07-17.29.01:700][781]LogLearning: Display: BP_RLTrainingManager_C_UAID_2CF05DAF86939DA601_1666759087: Resetting Agents [0 1 2 3 4 5 6 7 ... 31 30 29 28 27 26 25 24].
[2023.09.07-17.29.21:457][743]LogLearning: Display: Training Process: Creating Replay Buffer...
[2023.09.07-17.29.21:457][743]LogLearning: Display: Training Process: Creating Networks...
[2023.09.07-17.29.21:457][743]LogLearning: Display: Training Process: Receiving Policy...
[2023.09.07-17.29.21:457][743]LogLearning: Display: Training Process: Creating Optimizer...
[2023.09.07-17.29.21:457][743]LogLearning: Display: Training Process: Creating PPO Policy...
[2023.09.07-17.29.21:457][743]LogLearning: Display: Training Process: Opening TensorBoard...
[2023.09.07-17.29.21:457][743]LogLearning: Display: Training Process: Begin Training...
[2023.09.07-17.29.21:457][743]LogLearning: Display: Training Process: Profile| Pull Experience 19750ms
[2023.09.07-17.29.21:457][743]LogLearning: Display: Training Process: 4
[2023.09.07-17.29.21:457][743]LogLearning: Display: Training Process: Traceback (most recent call last):
[2023.09.07-17.29.21:457][743]LogLearning: Display: Training Process: File "D:\EPIC\UE_5.3\Engine\Plugins\Experimental\LearningAgents\Content\Python\train_ppo.py", line 361, in <module>
[2023.09.07-17.29.21:457][743]LogLearning: Display: Training Process: train_ppo(config, trainer)
[2023.09.07-17.29.21:457][743]LogLearning: Display: Training Process: File "D:\EPIC\UE_5.3\Engine\Plugins\Experimental\LearningAgents\Content\Python\train_ppo.py", line 199, in train_ppo
[2023.09.07-17.29.21:457][743]LogLearning: Display: Training Process: assert response == UE_RESPONSE_SUCCESS
[2023.09.07-17.29.21:457][743]LogLearning: Display: Training Process: AssertionError
[2023.09.07-17.29.21:697][754]LogLearning: Warning: Training Process finished with warnings or errors
[2023.09.07-17.29.38:367][556]LogLearning: Display: BP_RLTrainingManager_C_UAID_2CF05DAF86939DA601_1666759087: Resetting Agents [15].
[2023.09.07-17.29.39:587][616]LogLearning: Display: BP_RLTrainingManager_C_UAID_2CF05DAF86939DA601_1666759087: Resetting Agents [0 1 2 3 4 5 6 7 ... 31 30 29 28 27 26 25 24].
[2023.09.07-17.29.53:420][240]LogLearning: Display: BP_RLTrainingManager_C_UAID_2CF05DAF86939DA601_1666759087: Resetting Agents [15].
[2023.09.07-17.30.14:936][164]LogLearning: Display: BP_RLTrainingManager_C_UAID_2CF05DAF86939DA601_1666759087: Resetting Agents [26].
Unreal freezes for about a minute at this point
[2023.09.07-17.31.00:467][416]LogLearning: Error: BP_DrivingRLTrainer: Error waiting for policy from trainer. Check log for errors.
[2023.09.07-17.31.00:467][416]LogLearning: Display: BP_DrivingRLTrainer: Stopping training...
[2023.09.07-17.31.00:555][421]LogLearning: Error: BP_DrivingRLTrainer: Training has failed. Check log for errors.
[2023.09.07-17.31.00:680][424]LogLearning: Error: BP_DrivingRLTrainer: Training has failed. Check log for errors.
[2023.09.07-17.31.00:846][426]LogLearning: Error: BP_DrivingRLTrainer: Training has failed. Check log for errors.
[2023.09.07-17.31.00:880][427]LogLearning: Error: BP_DrivingRLTrainer: Training has failed. Check log for errors.
[2023.09.07-17.31.00:948][430]LogLearning: Error: BP_DrivingRLTrainer: Training has failed. Check log for errors.
[2023.09.07-17.31.01:057][434]LogLearning: Error: BP_DrivingRLTrainer: Training has failed. Check log for errors.
[2023.09.07-17.31.01:170][438]LogLearning: Error: BP_DrivingRLTrainer: Training has failed. Check log for errors.
[2023.09.07-17.31.01:255][441]LogLearning: Error: BP_DrivingRLTrainer: Training has failed. Check log for errors.
[2023.09.07-17.31.01:365][445]LogLearning: Error: BP_DrivingRLTrainer: Training has failed. Check log for errors.
[2023.09.07-17.31.01:450][448]LogLearning: Error: BP_DrivingRLTrainer: Training has failed. Check log for errors.
[2023.09.07-17.31.01:559][452]LogLearning: Error: BP_DrivingRLTrainer: Training has failed. Check log for errors.
[2023.09.07-17.31.01:668][456]LogLearning: Error: BP_DrivingRLTrainer: Training has failed. Check log for errors.
[2023.09.07-17.31.01:757][459]LogLearning: Error: BP_DrivingRLTrainer: Training has failed. Check log for errors.
[2023.09.07-17.31.01:848][462]LogLearning: Error: BP_DrivingRLTrainer: Training has failed. Check log for errors.
[2023.09.07-17.31.01:963][466]LogLearning: Error: BP_DrivingRLTrainer: Training has failed. Check log for errors.
[2023.09.07-17.31.02:143][476]LogSlate: Updating window title bar state: overlay mode, drag disabled, window buttons hidden, title bar hidden
[2023.09.07-17.31.02:143][476]LogWorld: BeginTearingDown for /Game/VehicleTemplate/Maps/UEDPIE_0_VehicleExampleMap
[2023.09.07-17.31.02:144][476]LogWorld: UWorld::CleanupWorld for VehicleExampleMap, bSessionEnded=true, bCleanupResources=true
[2023.09.07-17.31.02:145][476]LogSlate: InvalidateAllWidgets triggered. All widgets were invalidated
[2023.09.07-17.31.02:146][476]LogContentBundle: [VehicleExampleMap(Standalone)] Deleting container.
[2023.09.07-17.31.02:148][476]LogPlayLevel: Display: Shutting down PIE online subsystems
[2023.09.07-17.31.02:151][476]LogSlate: InvalidateAllWidgets triggered. All widgets were invalidated
[2023.09.07-17.31.02:156][476]LogSlate: Updating window title bar state: overlay mode, drag disabled, window buttons hidden, title bar hidden
[2023.09.07-17.31.02:158][476]LogAudioMixer: Deinitializing Audio Bus Subsystem for audio device with ID 3
[2023.09.07-17.31.02:158][476]LogAudioMixer: FMixerPlatformXAudio2::StopAudioStream() called. InstanceID=3
[2023.09.07-17.31.02:159][476]LogAudioMixer: FMixerPlatformXAudio2::StopAudioStream() called. InstanceID=3
[2023.09.07-17.31.02:169][476]LogUObjectHash: Compacting FUObjectHashTables data took 0.48ms
[2023.09.07-17.31.02:244][481]LogPlayLevel: Display: Destroying online subsystem :Context_2
[2023.09.07-17.31.03:119][530]LogDerivedDataCache: C:/Users/thomas.halpin/AppData/Local/UnrealEngine/Common/DerivedDataCache: Maintenance finished in +00:01:44.154 and deleted 0 files with total size 0 MiB and 0 empty folders. Scanned 56597 files in 52761 folders with total size 11372 MiB.
Can you tell what maybe happening?
Hello, can you try opening D:\EPIC\UE_5.3\Engine\Plugins\Experimental\LearningAgents\Content\Python\train_common.py and go to line 228 and increase the timeout from 10 to something larger?
Let me know if this works and by how much you increased it to get it to work.
2 Likes
I increased it to 50(don’t know if that’s over kill). Been running for 8 minutes so far and getting no errors! I think that fixed it! Thank you so much for your help!
I’ll edit this post if anything happens as this training continues.
Edit: Tested numbers 20, 30, 35, 40, 45, and 50. All numbers 40 and below failed.
45 and up worked correctly
1 Like
Hi Brendan, thank you for the excellent tutorial. I’ve encountered an issue after following your steps. Despite 2 hours of training, my cars are moving at only 3-4 km/h and are frequently switching between 1st gear and reverse. It seems like this behavior helps the cars stay on track, but I’m puzzled as to why I’m the only one facing this problem. Any insights?
1 Like
same for me…im not experienced enough to tell if this is normal and just needs more time 
i hope they at some point learn to step on the gas for some sweet rewards 
I think I might solve the problem by changing Reverse Gear Ratios from 4.04 to 0 in VehicleAdvPawn. It is still in the training step but cars just moving forward. So after 20 minutes of training, cars are not backing up but the speed does not exceed 5 km/s, and the gear is frequently changed between 1 to R and vice versa. I am going to check my code again and if every thing is good maybe I will disable the R gear or make a change at reward.
2 Likes
I turned off reverse is brake, then lowered reverse to 0.5
1 Like
“lowered reverse to 0.5” …the reverse gear ratio?
1 Like
I think so
it worked ?
nope,…they never really want to accelerate. i looked at all the autobrake options and set the velocity reward to 1 without any recognizable difference. they are just stuck in stutterstepping between braking and accelerating with never going above 5kmh (around 4h of training). my vehicle demo level is from ue5.0 and got converted to 5.3…did epic maybe change something in the meanwhile?..
lots of changes! inputsystem…new parent class. trying now with the 5.3 version of the vehicle demo.
yop already after 500 iterations i see solid movement! only change i did was to set “reverse as brake” to off
2 Likes
I checked that with Ue5.3 and it did not work, there should be another solution. I am exploring the other options.