Let objects push the character

It is not because it is being scaled, I tested having the 4 sides of the wall just move individually without any scaling, the problem was the exact same. Also, what do you mean by looking into physics simulation?