Incorrect/incomplete diffuse used for lightbake and GI

So my material uses vert colors from the mesh to mask off areas for individual adjustments in color, wear, roughness, etc. But the lightbaker doesnt seem to be able to recognize that masking, it only uses the last color input for the entire meshes diffuse contribution. I can override that with the lightmass replace node, but again, only one big solid color.

Anyway to fix this?