Hi! Sorry for digging up an old post, but did you ever get to solve this?
I’m dealing with a very similar issue and my widgets also show the behaviour you describe when zooming in and out, so I figured they might be related: Inconsistent text size with Wrap Text At + Scale Box