why box collision push me out of the box collider?

Hi! I apologize if my question is silly, but I’m working on a push and pull system for an object. I have a box collision on the object that detects when the player presses E on the keyboard to interact with it. The player can then move the object forward, backward, right, or left. The problem is that every time I press E, the box collider pushes me out of it and I don’t know what to do, When I’m not pressing the E key, I can be inside the box collider; it only pushes me out when the interaction to push forward, backward, right, or left is activated. Even outside the box collider, the player can move the object, but not with as much precision. So, the end result is the player moving the object, but with the object far away from them.


Inside box collider, I didn’t press E


Pushing me out after press E


Moving the object