Lightmass bakes out materials in the 0-1 UV space a single time to keep build times low, instead of for every triangle they are applied to. This means that any per-instance properties like vertex color, world position, camera vector etc are not going to be correct and use defaults. You can use a LightmassReplace node to have it use something different for lighting than what you see rendered.