Actor movement works unless I enable physics simulation (gravity)

So I have an object that I’m moving to the right like this: [SEE ATTACHMENT]

This works but since I want the object to fall down when it drops off a ledge I tried to enable physics simulation and gravity on the object but to my surprise this completely breaks the movement, like it doesn’t move at all anymore apart from falling down because of gravity. Why does this happen?