I’m not much a materials guy I’m afraid. The problem with packing things into the vertex colour channels is that the unit size of the terrain is pretty large when you get up close and the smoothing between vertices can be pretty ugly. I use some noise to make the material transitions a little more natural looking.
You can certainly pack whatever you want into the Vertex Colour information, it’s just a single line of code to adjust it.
Maybe decals would work well for what you’re after? Project a decal from the building onto the terrain? Haven’t tried it but in theory…