I can also confirm that the behavior has changed from 4.19 onwards and MaxContactOffset only has a limited impact. Looking at the source code it seems the problem comes from a line in FBodySetupShapeIterator::ForEachShape in BodySetup.cpp. In 4.19 it looked like this:
const float ContactOffset = ComputeContactOffset(Geom);
And in 4.20 it is this:
const float ContactOffset = FMath::Max(ComputeContactOffset(Geom), RestOffset + 1.f);
So the contact offset will never be able to go lower than 1 + whatever RestOffset is. I reverted this particular line of code with theone from 4.19 and that seemed to fix the problem.