Hi there,
there’s some rather unfortunate error handling in the shared memory communicator of Learning Agents on the python side that silently shreds larger datasets:
Line 45:
size = int(np.prod(shape) * np.dtype(dtype).itemsize)
will easily wrap around into the negatives if the dataset is reasonably large, since np.prods will produce an int32 by default. For example: if my dataset for behavior cloning is 110.458 steps with an observation dimension of 5.517 (multiplied by sizeof(float32) = 4), the requested size is 2.437.587.144 but then becomes −1.857.380.152. The real problem is though, that the communicator just silently produces an all zero array instead of failing. The validity of the returned handle is never checked again.
Turning this into an int64 mitigates the problem, because too large arrays will most likely lead to an out-of-memory error and force a hard crash, but the else path should probably lead the system into an error state.
Something like:
def shared_memory_map_array(guid, shape, dtype, create=False):
element_num = int(np.prod(shape, dtype=np.int64)) #main change
size = element_num * int(np.dtype(dtype).itemsize)
if size > 0:
was_error = False
try:
handle = SharedMemory(guid, create, size)
except FileExistsError:
if create:
# Windows has trouble cleaning up so use the existing shared memory - C++ side will reinitialize it
handle = SharedMemory(guid, False, size)
was_error = True
assert handle is not None
array = np.frombuffer(handle.buf, dtype=dtype, count=element_num).reshape(shape)
if was_error:
# if we were trying to create, but ended up mapping, then we are responsible for cleaning the memory and not C++
array[:] = 0
if create:
logger.info('Created shared memory with name: ' + guid)
else:
logger.info('Mapped existing shared memory with name: ' + guid)
return handle, array
else:
return None, np.empty(shape, dtype=dtype)
Edit: The forum editor mangled the indentations quite a bit. Copy & paste with care 