How do I prevent my physics constraint from drifting?

[Lirrec]

Have you tried turning projection on, with a low tolerance?
That should keep the head from moving away from the handle.
I had had a similar issue ( one of the contrained actors moving where it should not ) and projection helped.