@Everynone, I already had it disabled because I did not want my character to affect those cubes as well. Sadly, this setting is part of the character movement, so it only affects the player character, not what it is holding with physics handle.
I also tried tweaking the interpolation speed of the handle, but it only slows down the interpolation of the grabbed object, which reduces the force but doesn’t remove it.