Is it possible to have no movement between components via physics handle?

I use Physics Handle to connect one component attached to my hand (PU) and another component (Detonator). The thing is that I need to connect one specific section of both. I tried making the connection with an offset and it works, but when I move the hand attached to the PU, I get an undesired swinging movement, even when I disable the soft angular constraint, soft linear constraints, and interpolate target option in the physics handle details. I can’t use physics constraints, because I need to connect the two things when the PU is closer to the Detonator. Attach component is also not an option, because, when I use it, the physics in the Detonator stop working.
Does anyone know how to fix the swinging movement?
I’m working with VR by the way.