I assume that just giving those objects a collision and then tracing for them is not an option.
So I have two ideas that would work
Idea 1:
Well, basically it is your idea
Pseudocode:
for(object in objectArray)
{
float DotResult = FVector::Dot(CamForwardVector, (object.Location - CamLocation).SafeNormal());
if(DotResult > BestResult)
{
bestObject = object;
}
}
IF both vectors are normalized, DotProduct gives you, well the angle between them. It returns the angle in a range of -1 to 1 where 1 -> 0degree difference and -1 -> 180°difference and 0 -> 90°.
Note: You might want to add a distance from camera to this evaluation, so if two objects are kinda the same angle but one is closer to the cam, you might want to take the one closer, even if it’s angle is bigger.
Idea 2:
Just project the points onto your 2D ViewPlane using Canvas->Project(). Then just check the distance from the point to the center of the screen and pick the smallest.
That does not work if you have points behind you though.