Replying so that other travellers might see my solution:
I found that if I play around with the Cascaded Shadow Maps settings on the DirectionalLight I could increase the render distance and even fade it better. Now it all looks natural.
It’s the dynamic shadows render distance causing this.