There are like (8?) grip types now, only the “interactive with physics” types require simulation, the rest directly move the held object instead and should function fine assuming that you don’t run their own movement on themselves during the grip.
Use PhysicsOnly, it just sets their transform every frame to match the desired controller offset.