Virtual Shadow Maps in 5.3 - Close to camera shadow bug

I noticed some weird shadow map annoyance im hoping anyone has a solution for. Ive been looking around for someone else who has this issue but everyone seems to have the problem with that the shadow maps dissapear at a distance, but im instead getting that the shadow map disappear close to the player.

It appears if a directional light (the sun) is blocked by an object far away from the player. It first looked really bad with spots everywhere without any shadows:

But that was an easy fix. It was just to check the “Far Shadow” on the object and it looked nice and clean again:

Unfortunately, the thrill of success didnt last long, cause when i moved the object a bit further up, away from the height where the player is, this thing happens now:

It looks like the closest Clipmap/mipmap is bugged out for the shadow map or something.

An easy solution is ofc… just put it closer to the same height as the player… but that is a fairly big restriction to keep in mind for everything in a level, i would rather find a real solution.

I have tested the old shadow maps system, btw. It works fine there, but it is worse in so many other ways. The new Virtual shadow maps looks much better in general so it would be nice to be able to use that.