The sight sense uses as a default GetActorLocation as the target point for tracing to check if an actor is visible. So one thing you should check is the pivot point of your mesh. If it’s at the base of the mesh, then it may be coincident with or slightly below the landscape. Even on a perfectly flat landscape, floating point precision errors could mean that sometimes the trace will hit the landscape before hitting the mesh, which will result in a failed sight sense.
An easy way to see if this is the issue would be to replace your tower meshes with the default sphere mesh, and see if that fixes things.