Aim Offset Issue while riding a horse

Anybody knows how to fix this aim offset issue. It starts out working fine but when I move around on my horse it seems to snap from left to right or vice versa. Maybe I need to reset the players forward direction everytime I move. IDK.