It works. In 5.6 (and I guess on later versions too)
The things to know:
-
UV has to be correct and the channel selected correctly in the physical material mask asset
-
the mesh needs to have the flag Support Physical Material Mask

-
the texture can be a simple PNG texture with the correct colors values. (I disabled compress with alpha but I don’t know if it matters)
-
Right click on the Physical Material Mask asset to import it
-
The mesh needs to use complex collisions instead of the simple ones (probably? haven’t tested with simple collisions)
-
The tracing for the hit has to be set to complex (probably?)
-
Here’s the real tricky part: YOU CAN’T use a material instance. It simply doesn’t seem to work with a material instance. You need to set this up as a material on it’s own.