It’s not clear what values? If you already disabled physics interaction on your character it shouldn’t move things around when walking into them. I don’t know any other setting affecting the physics handle itself on this without diving into the code.
There are related posts:
How to disable physical interaction of attached Actor on player back?
How to remove Character physical interactions - #3 by STRiFE.x
Possibly you could disable physics interaction on the actor you are holding through the handle as well. In my grab system when I hold an item I must also ignore the collision of what I’m holding so I can walk around without being blocked. Ignoring collision can be set up from actor to actor.