Convert to Virtual Texture not working right

Whenever I right click → Convert to Virtual Texture ue5 decides that almost all of my textures and materials in the project need to be converted too… But using the reference viewer, the texture only has a few references. The other textures and materials in the project are unrelated and use different master materials. Is anyone else experiencing this / found a workaround? It would be very tedious to have to convert everything manually…