I’m also having exactly this issue. The Chaperone bounds lag a centimeter or two behind geometry. Causes an unpleasant swimmy effect when the chaperone is visible.
As you say, deleting the player camera fixes it.
Found this other question which has a code solution…
I’m updating to 4.11.1 now to see if it’s fixed.