Indeed. Virtual Shadow Maps use caching to improve performance for static objects. It is divided into a grid and the engine checks for certain things to decide if the page should be invalidated (which is why it sometimes fixes itself). Disabling the cache entirely is extremely expensive because then all shadows need to be recalculated per frame, but there are experimental features described in your linked page that can help so pay special attention to the sections about the cache and foliage.
If those do not help, VSMs may not be a good fit for your project in its current form but it is still in it’s early days.