Temporary fix I found, but I hope someone has a real fix or knows anything about that:
I set the scale of the player start very low (0.1. I tried 0.01 but then my chaperone bounds were moving around for some reason) and put it as close to the floor as I could.
It’s still a bit off (10cm higher than the ground) but it does the job for now.