Ladder functionality problem

Hey eveyone I have a small problem. I´m trying to make a climbable ladder functionality.

When my thirdperson character overlaps the collision box in the ladder blueprint, It sets the mode of the character movement to flying, and then I set the characters movement in the direction of the up vector. I do this to be able to move up and down the ladder.

But the problem is, when my player character enters fly mode and recieves input for the up vector, for some reason he always turns around to the positive x axis in the world.

As you can see on the pictures, my character is supposed to be facing the ladder, but has turned to the x axis of the world instead.

Any advise is helpful.