Shadows from meshes that are set to "Only Owner See" are still casted in non-owner view

Exactly as the title suggests. When I set a mesh to “Only Owner See” I am still getting shadows casted in view where I would not expect to see them. Is there a way to make this not happen?