Layer material performance impact if layer's visibility is zero.

I’m aware of the performance impact when using layered materials.

What I would like to know is what is the performance impact of layered material, which let’s say has 3 layers, but two of them are currently not used (e.g. completely masked out by texture or vertex colors).
I guess textures from all the layers are still in the memory, but is there some CPU/GPU performance hit? If so, is it the same as if the layer would be at least partially visible or is it something in-between (more then simple non-layered material but less then if the layer would be visible)?

Thanks