After a while of testing the thing that made the third person BP work but not the normal Character is Use Controller Rotation Yaw.
If someone has the same problem as me and dont want to remake the character movement in a pawn i would suggest turning the Use Controller rot yaw off when simulating your character, also if you want to be able to move the camera around in Yaw axis while your simulating physics you just need to rotate it in Yaw instead of the Controller Yaw. Keep in mind this wont rotate the Controller which is fine for me. And if the camera is rotating weird you can rotate the Yaw in a parent object instead of the camera object as to avoid changing the Roll. You could also restraint the rotation.
Hope this helps!