Making a physics block that the player cannot interact with directly?

Hi there. I’m working on a project which involves several physics enabled assets of different shapes, which can be controlled with a series of switches elsewhere. Those work- my current issue is trying to stop the player from being able to cheat by pushing them around through physical contact.

I was able to limit this through friction, mass and so on, but the player can still move the objects extremely slowly by pushing. In particular, if the player runs around on top of the object and then falls off it, there seems to be a sudden burst of force on the object as well. Is there a way to remove this force, or better yet, a way to stop the player being able to impart force at all?