Off-by-one errors in ppo.py affect the last episode of a gather during training

Development Programming & Scripting AI
, , ,
1

Hello,

I think I have found two off-by-one errors in the LearningAgents PPO python trainer ppo.py

While comparing the LearningAgents Python PPO trainer to the CleanRL version I found two off-by-one errors in loop bounds. Both are in Content/Python/learning_agents/ppo.py. Each one makes the trainer drop the last step of the experience buffer.

1. GAE recursion never computes the last step’s advantage

In compute_advantages_and_returns, advantages are accumulated backwards over the buffer:

adv = np.zeros_like(rew)
for i in reversed(range(len(rew) - 1)):
adv[i] = delta[i] + discount[i] * adv[i+1]

adv[-1] is always 0 because range(len(rew) - 1) starts the loop at len - 2.

Since the loop reads adv[i+1] the missing value feeds back into every earlier step of the buffer’s final episode, until it hits an episode boundary. The advantage is incorrect for last episode of every gather. Episodes before the last one are correct.

The fix is to start at len - 1 and carry a running sum:

gae = 0.0
for i in reversed(range(len(rew))):
  gae = delta[i] + discount[i] * gae
  adv[i] = gae

The second occurrence is in the policy window loop. It stops one window short. In in ppo.py:train (line 403), fixed-length windows are built over the buffer for recurrent policy training:

for i in range(len(obs) - policy_window):
  window_indices.append(i + np.arange(policy_window))

The last valid window start is len(obs) - policy_window, and range excludes its upper bound, so that window is never built. As a result:

  • The last step of every gather is never part of any window, so it never gets a policy gradient.

  • When len(obs) == policy_window, no windows are built at all. Then torch.randint(0, len(window_indices), ...) at line 454 fails with random_ expects ‘from’ to be less than ‘to’, an error that doesn’t mention the window size.

The fix is range(len(obs) - policy_window + 1). You might also want to include a guard that raises a clear error when len(obs) < policy_window.

Both errors come from the same mistake: the upper bound passed to range is one too small. Here are two generic example in Python that demonstrate the general shape of the error:

# Issue 1: backward recursion skips the last element
delta = [1.0, 1.0, 1.0, 1.0]        # per-step values to accumulate
discount = [0.5, 0.5, 0.5, 0.0]     # 0.0 marks the end of an episode

buggy = [0.0] * len(delta)
for i in reversed(range(len(delta) - 1)):   # first i is len-2; buggy[-1] is never written
buggy[i] = delta[i] + discount[i] * buggy[i + 1]

fixed = [0.0] * len(delta)
running = 0.0
for i in reversed(range(len(delta))):       # first i is len-1
running = delta[i] + discount[i] * running
fixed[i] = running
print("buggy:", buggy)
print("fixed:", fixed)

Output:

buggy: [1.75, 1.5, 1.0, 0.0]
fixed: [1.875, 1.75, 1.5, 1.0]


#Issue 2: sliding windows stop one short
n, w = 8, 3
def covered(starts):
return sorted({s + k for s in starts for k in range(w)})
print("buggy:", len(range(n - w)), "windows, covers", covered(range(n - w)))
print("fixed:", len(range(n - w + 1)), "windows, covers", covered(range(n - w + 1)))
print("n == w -> buggy:", len(range(w - w)), "windows, fixed:", len(range(w - w + 1)), "window")

Output:

buggy: 5 windows, covers [0, 1, 2, 3, 4, 5, 6]
fixed: 6 windows, covers [0, 1, 2, 3, 4, 5, 6, 7]
n == w -> buggy: 0 windows, fixed: 1 window

P.S. Apologies if these have already been found and fixed. I could not figure out where the learning agents python code is in the GitHub repo.

1 Like