There are couple of threads with this problem. Search for more detailed description.
Anyway this is simplest version:
- get up vector and forward vector of your character. Normalize both.
- calculate vector from center of screen (or character) to location where mouse pointer is. Zero its “depth” value ie it should be [x,0,z] or [0,y,z] depending which plane is your screen plane. Then normalize that vector.
- now calculate dot product of UP and mouse vector, then forward and mouse vector. Both values are cosinus of angle between vectors. No you get one of them and do arcus cosinus to get degrees, then you use other sign to see it its plus or minus that angle from another. That is because dot product gives same angle for right and left sides.
I helped about 4-5 times to solve this problem there are threads discussing it, just search.
Oh and debug every step, for that use “line trace” set its debug line to display for frame or duration.
Check one by one if all vectors are correct. For eg draw, up vector of character (multiply its length by 1000), then stop game draw forward vector. then draw mouse vector make sure its always from character (or screen center) to your mouse (or pad direction). When all vectors are exactly what you want them to be, calculate angles. Its quite common to make vector calculations correct but to have wrong vectors as input, thus making results all wonkyy. For eg if you rotated character by 90 deg (like character setup tutorial says) and you take capsule forward instead of character mesh forward vector, you will get weird results.