ID map and RGB mask for layered material multiple surfaces

I want it to be all over the mesh, but I want the parts to be different depending on if it’s wear on the metal or leather or wood.

I might have been going too complex with the layered material approach, I tried making a new simple material.

All I want is to be able to have 1 single material with 3 surfaces, based on ID map.
Then I want a detail map for the dirt, dust, wear. But I need it to cover the entire mesh (uv2) but look different depending on the surface. So no shiny metal wear on fabric.

I guess I need to either isolate the detail map based on the ID map somehow or have individual parameters for each layers detail/scratches

Here is my new very basic material, i’m just trying to make this as simple and as optimized as I can.

Surface controlled by ID map

the detail map, red for dust, green for wear, blue for dirt

Here is the albedo node set up, 3 diffuse textures. Each applied based on the ID map

and finally here is the full material setup

Now everything works except I don’t know how I can have the details show up and be different for each layer when it comes to roughness, color etc.