the only way to do it is to use multiple materials on the mesh, so each material can have its own Physical Material