I don’t think virtual textures will matter, that is more of a memory optimization. You need to reduce the number of materials you use.
When I last tested it, it seemed like only the materials that were within the camera frustum impacted performance, so it may not necessarily be every material in the scene, just the ones in the direction of the camera.
You’d have to try doing a stress test