Your setup is missing depth (world direction). You may not be able to achieve consistent results without fancier de-projection that accounts for camera frustum.
Consider this approach instead:
Seems very close to what you need. You can skip the trace in the example, and use the Intersection as is.