Yes, line trace 
For some reason ( no doubt a youtube vid ), there’s this tidal wave of people using overlaps for interaction. It is not a good method.
One of the downsides you’ve seen. Another is that you can interact with things with your BUTT!
Line trace is the way, you can only interact with something if you’re looking at it ( or a component of it ).
