Ways to remove jitter from continuous forced collision?

Hi All,

I have a setup involving an motion controlled (oculus touch) character that is also working with a physics rig.
My problem is that as soon as a bone/body/section gets in contact with another, it starts to jitter and shake.
For example if I press my hand towards my own chest or leg.

I have a physical animation profile that is tweaked to look and behave roughly the way I want it, so I don’t want to change the damping values too much from what I have.
I just also tried to attach a physics material with zero friction and bounce.
EDIT: Just trying enabling CCD but that does not seem to make any difference.

But I still have too much jitter as two body parts get in continuous contact.

Does anyone know if there’s anything I should try to do to get rid of this, as it doesn’t make the character look very professional.