I solved it by creating my own PhysicsHandleComponent in C++.
I basically created a new class based on SceneComponent, copied the code from Epic’s Physics Handle from github, removed the interpolation and instead set the KinActor’s Location to that of MyPhysicsHandleComponent. Then I changed the PxD6Constraint to be locked on every axis instead of being free on every axis and using the Position/Rotation Drive. Then I check if I the grabbed component reached the right location every frame. If it didn’t, I am switching back to the Position/Rotation Drive, because we must have hit a static object, and we don’t want to try to go through that static object. Once the Position/Rotation Drive moved the grabbed object back to the Handle Location, I am switching back to locking the axis.
To use PhysX code, check out Rama’s tutorial for that on the wiki.