How do you make gripped objects collide with static meshes that does not simulate physics?

Hello!

I got a problem in which there is an element to my game were it’s encourage to hit some static meshes (kinematic). My problem is that if they don’t simulate physics my gripped object just goes right through them, but if I make them simulate physics my gripped object will just push them away and they will not stay in one place.
I have tried to put constrain on the x,y,z movements etc, but it seems to not work.

How can I make my gripped object to collide with a static mesh that does not simulate physics? I appreciate any answers!