There’s an issue in LearningAgentsRecorder.cpp:67 (UE 5.7.4 EOS, but I don’t think that file has changed much). The condition:
if (StepNum / ChunkSize <= Observations.Num())
is flipped and always true. Instead of allocating a chunk per 1024 state/action pairs, it allocates a chunk for every state action pair (even if the indexing functions don’t use that). Flipping it to
if (StepNum / ChunkSize >= Observations.Num())
makes this work as intended. We found out because our schema contains a convolution of a few kilobyte, hence every chunk became 13 Megabytes. Combine that with roughly 50.000 recorded observations, and the runtime becomes very unhappy.