You have to make sure the collision query parameters had the bReturnPhysicalMaterial set to true, otherwise it will always be the default.
If using complex collision, the material is specified by the material shader. If using simple collision, it’s specified by the collision body.