Ok I think I understand a bit better. So right off the bat, I think the reason that the ball shoots off behind the player sometimes when it shouldn’t (when the forward vector and direction of the trace to the actor behind the player align) is due to you getting the absolute value of the dot product. The sign of the dot product is crucial because it determines the direction of the vector given two points relative to the player’s forward vector (the player’s location and the target actor’s location)
Here’s an image to show how the signs are important.
Notice that the values fluctuate between -1 and 1 where 1 means same direction and -1 means opposite direction. You’re going to see the -1 when the actor is behind the player. So you’ll want the values to be somewhere between 0 and 1 where 0 is 90 degrees in either direction in front of the player and 1 is directly in front of the player. That will at least filter out your possible actors.
Then I’m not sure if you want to get the closest target to the player’s position or the closest target that the player is looking based on rotation. Like a target may be 10 meters away, but has a dot product result of 1 (you are looking directly at the actor) and another actor is in view that is 5 meters away but the dot product gives you something like 0.4 (you are sort of looking at it but it’s a little bit to the left of the screen). So you’ll want to choose your final target based on those criteria.
Let me know if any of that was helpful or if you need to give or need any other clarification! 