I believe the issue is that you are Setting the control rotation instead of adding to it.
So say for example the Axis result of the right arrow key is 1 and the left arrow key is -1. All you are currently doing in your graph is setting the X rotation as either -1, 0 or 1.
What you need is to either instead use the Add Controller Yaw/Pitch Input or first Get the control Rotation, add the Rotation from X Vector and the current control rotation together, and then set the control rotation.