Reverse engineering a transform using rotations may work but it is just going to be way more complicated and probably 100x more expensive on the GPU. The correct operation for what you need is a transform. A transform is only a few multiplies and adds on the GPU whereas rotations are many sin, cosine, square root and other very slow calculations involved. Transforms can also go from any coordinate space to any other so there should be no limitation on rotate type like you mentioned, but perhaps I am not understanding what the issue was that you saw.
You said screen space didn’t work, but why?
You can transform into any custom basis using the material function “Invert Transform3x3Matrix”. You need to supply the 3 orthogonal vector for the new space.
If you want to add some sort of FOV or width to the projection you can divide the two other coordinates by 1+the distance along the axis of the projection. Ie if you are projecting along camera X axis, then you need to divide the YZ by the X after it has been scaled.