Instead of a physics material, use a custom line trace channel, set to ignore by default. On the meshes you want to check block this trace channel.
Instead of a physics material, use a custom line trace channel, set to ignore by default. On the meshes you want to check block this trace channel.