Hi dysplaced -
This is a result of the blending between Static and Dynamic Shadows in the Lightmapping from your directional light. Essentially, the switch from dynamically lit object to Static Lighting is causing that jump you are seeing. It is more pronounced because of the size of your object. I was able to reproduce this exact behavior by just scaling down a provided engine asset. You can mitigate this distance transition by adjusting the Dynamic Shadow Distance Stationary setting in your Directional Light source under Cascaded Shadow Maps. I used a moderate setting of just 500 units from camera and was able to fix my test case. You will have to adjust based on your view of the light switch and how far away you will be seeing the object.
Let me know if I can help you further with this issue -
Eric Ketchum