After many hours of work I finally got something working. If anyone else wants to do this, i’ll share the method.
I’m using a utility widget to show some debugging, but mostly to get a ticking function.
First thing you’ll want to do is to get the mouse position in viewport and print the coordinates for the top left (TL) and bottom right (BR) corners of the viewport.
Second, use those coords to hard code the viewport data and do some calculations.
Thirdly, capture the mouse location every tick and calculate its position in the viewport.
Then you can position the widget over the viewport to make sure you get proper lines, indicating everything is set up correctly. Make sure TL of the widget represents TL of the viewport.
Now for projecting the screen mouse location to a location relative to the camera world location.
The reason for the Forward vector *11 is because 10 units is the camera near clip plane.
The reason for multiplying the mouse position by 0.019 is scaling the screen coord to world coords… I guess?
The actual line trace.
Hope you find it useful. It’s not as exact as the built in functions, there’s a slight drift but not really noticeable. It also linetraces from the camera location, not the mouse cursor location. But this really is good enough.