Here is a picture of my wall blueprint, I am just scaling the wall, but the issue also occurs if I move the wall instead.
My meshes are nothing special, just a set of numbers 0-9 with a standard box collision on the mesh, I just spawn them in my scene, and the walls push them closer to the center as they should. I also just tried adding an extra box collision to one of the sides of the walls, but the problem still occurs.