I find it also interesting to know why z starts to rotate, but if you can exclude wrong calculations, I guess only a physx guy can explain, why the solvers tend to be inaccurate/unstable with some given input.
Perhaps this is normal behavior as errors accumulate ? Seems to be wrong.
You could perhaps try to find an physx engineer if you have a repro.
Good luck