Hey -
You are correct. And it is not weird or dumb as that is exactly how reflection actors work currently.
I threw together a test map to better illustrate the point, but it is important to remember that the reflection actors take a capture within the bounds of the actor along the sphere curvature (or box) and transfer that capture as a cubemap to the materials based on their metallic and roughness settings.
Hopefully this will help explain how it works but if you have any further questions please let me know-
Eric Ketchum