# Off-by-one errors in ppo.py affect the last episode of a gather during training

**URL:** <https://forums.unrealengine.com/t/off-by-one-errors-in-ppo-py-affect-the-last-episode-of-a-gather-during-training/2845303>\
**Category:** AI\
**Tags:** Reinforcement-Learning, unreal-engine, Learning-Agents, bug-report\
**Created:** [October 26, 2026, 2:08pm UTC](https://forums.unrealengine.com/t/off-by-one-errors-in-ppo-py-affect-the-last-episode-of-a-gather-during-training/2845303 "2026-10-26T14:08:11Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![MannyVindiola](https://d3kjluh73b9h9o.cloudfront.net/original/4X/2/8/5/28502927f1d97ba1051b3b47ccfe17233bf41ef0.png) [@MannyVindiola](https://forums.unrealengine.com/u/MannyVindiola)\
**Post date:** [October 26, 2026, 2:08pm UTC](https://forums.unrealengine.com/t/off-by-one-errors-in-ppo-py-affect-the-last-episode-of-a-gather-during-training/2845303/1 "2026-10-26T14:08:11Z")

</div>

Hello,

I think I have found two off-by-one errors in the LearningAgents PPO python trainer ppo.py

While comparing the LearningAgents Python PPO trainer to the CleanRL version I found two off-by-one errors in loop bounds. Both are in Content/Python/learning\_agents/ppo.py. Each one makes the trainer drop the last step of the experience buffer.

1. GAE recursion never computes the last step’s advantage

In compute\_advantages\_and\_returns, advantages are accumulated backwards over the buffer:

```auto
adv = np.zeros_like(rew)
for i in reversed(range(len(rew) - 1)):
  adv[i] = delta[i] + discount[i] * adv[i+1]

```

`adv[-1] is always 0 because range(len(rew) - 1)` starts the loop at `len - 2`.

Since the loop reads `adv[i+1]` the missing value feeds back into every earlier step of the buffer’s final episode, until it hits an episode boundary. The advantage is incorrect for last episode of every gather. Episodes before the last one are correct.

The fix is to start at `len - 1` and carry a running sum:

```auto
gae = 0.0
for i in reversed(range(len(rew))):
  gae = delta[i] + discount[i] * gae
  adv[i] = gae

```

The second occurrence is in the policy window loop. It stops one window short. In in ppo.py:train (line 403), fixed-length windows are built over the buffer for recurrent policy training:

```auto
for i in range(len(obs) - policy_window):
  window_indices.append(i + np.arange(policy_window))

```

The last valid window start is `len(obs) - policy_window`, and range excludes its upper bound, so that window is never built. As a result:

- The last step of every gather is never part of any window, so it never gets a policy gradient.

- When `len(obs) == policy_window`, no windows are built at all. Then `torch.randint(0, len(window_indices), ...) `at line 454 fails with random\_ expects ‘from’ to be less than ‘to’, an error that doesn’t mention the window size.

The fix is `range(len(obs) - policy_window + 1)`. You might also want to include a guard that raises a clear error when `len(obs) < policy_window`.

Both errors come from the same mistake: the upper bound passed to range is one too small. Here are two generic example in Python that demonstrate the general shape of the error:

```auto
# Issue 1: backward recursion skips the last element
delta = [1.0, 1.0, 1.0, 1.0] # per-step values to accumulate
discount = [0.5, 0.5, 0.5, 0.0] # 0.0 marks the end of an episode

buggy = [0.0] * len(delta)
for i in reversed(range(len(delta) - 1)): # first i is len-2; buggy[-1] is never written
  buggy[i] = delta[i] + discount[i] * buggy[i + 1]

fixed = [0.0] * len(delta)
running = 0.0
for i in reversed(range(len(delta))): # first i is len-1
  running = delta[i] + discount[i] * running
  fixed[i] = running
print("buggy:", buggy)
print("fixed:", fixed)

```

Output:

```auto
buggy: [1.75, 1.5, 1.0, 0.0]
fixed: [1.875, 1.75, 1.5, 1.0]

```

# 

```auto
#Issue 2: sliding windows stop one short
n, w = 8, 3
def covered(starts):
  return sorted({s + k for s in starts for k in range(w)})
print("buggy:", len(range(n - w)), "windows, covers", covered(range(n - w)))
print("fixed:", len(range(n - w + 1)), "windows, covers", covered(range(n - w + 1)))
print("n == w -> buggy:", len(range(w - w)), "windows, fixed:", len(range(w - w + 1)), "window")

```

Output:

```auto
buggy: 5 windows, covers [0, 1, 2, 3, 4, 5, 6]
fixed: 6 windows, covers [0, 1, 2, 3, 4, 5, 6, 7]
n == w -> buggy: 0 windows, fixed: 1 window

```

P.S. Apologies if these have already been found and fixed. I could not figure out where the learning agents python code is in the GitHub repo.

---

<div class="post-metadata">

**Author:** ![Deathcalibur](https://d1ap1mz92jnks1.cloudfront.net/user_avatar/forums.unrealengine.com/deathcalibur/32/1037610_2.png) [@Deathcalibur](https://forums.unrealengine.com/u/Deathcalibur)\
**Post date:** [October 27, 2026, 1:39pm UTC](https://forums.unrealengine.com/t/off-by-one-errors-in-ppo-py-affect-the-last-episode-of-a-gather-during-training/2845303/2 "2026-10-27T13:39:19Z")

</div>

Great find. This has been submitted on UE Main. Thanks!