We have been running into the same issue on our project. We have been able to mitigate the issue somewhat by changing the feedback factor to a much higher number, which seems a bit counter intuitive when reading the tooltip (Lower factor will increase virtual texture feedback resolution which increases CPU/GPU overhead, but may decrease streaming latency, especially if materials use many virtual textures.).
The number we settled on for now is 512, which seems to give us the sharpest results ingame, with the least amount of waiting for the system to load the higher resolution mips. However we still need to do a deeper dive into why this number works better than the default value, and how it actually works under the hood. Hope this helps some of you for now.