Weird physics handle behavior

I had the same problem, I have the pickup hold location specified by a scene component attached to a spring arm. It was caused by “attach rotation to movement” character movement parameter set to true. With this set to false it works (but your char wont turn of course)