Hey I might have found something:
So based on Aris answer if the bounds of an ISM overlaps 2 cells it gets promoted to a higher level cell. In my case it got promoted to the persistent level and never got unloaded. I checked the matrix demo config and found this line which defaults to false:
wp.Runtime.RuntimeSpatialHashPlaceSmallActorsUsingLocation=1
The full config is like this:
[SystemSettings]
; World Partition
wp.Runtime.RuntimeSpatialHashUseAlignedGridLevels=0
wp.Runtime.RuntimeSpatialHashSnapNonAlignedGridLevelsToLowerLevels=0
wp.Runtime.RuntimeSpatialHashPlaceSmallActorsUsingLocation=1
wp.Runtime.RuntimeSpatialHashPlacePartitionActorsUsingLocation=0
Let me know if it solves your issue as well.