Runtime virtual texture only works on fractions of geometry

I have a runtime virtual texture setup for a terrain and it seems to kind of work at least.
It renders parts of the RVT data, but only on fractions of the surface:

My guess is, that it renders a lower res mipmap as from a (very) far distance the size of the rendered tiles gets bigger relative to the geometry:

For the setup, I used the setup as suggested in this Quixel blog entry (last blending method in the post is RVT):

Does anybody know a solution, so the RVT renders properly on the entire geometry?
I think my general setup is fine at least, because it doesn’t render purely black, but some RVT data is rendered.