I got snapshots working! I had to set the MaximumRecordedEpisodesPerIteration to a lower number to trigger snapshot saving at a more frequent interval for my setup.
I was overflowing memory at first when I got it working but now I’ve simplified/packed down my observation space. Do you have any insight on what might be causing this error? I get it sporadically now when training:
Engine\Plugins\Experimental\LearningAgents\Content\Python\ppo.py", line 531, in train
LogLearning: Display: Training Process: policy_batch = torch.randint(0, len(window_indices), size=[policy_batch_size])
LogLearning: Display: Training Process: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
LogLearning: Display: Training Process: RuntimeError: random_ expects 'from' to be less than 'to', but got from=0 >= to=0
I’ve been messing with the training settings for my use case, so likely something on my end. I imagine every use case requires different configurations but that there are certain principles (which I’m still learning) that hold true in every case for ML to be effective…