Issue between physics objects and character controller

Does anyone have an idea how to fix these weird issues when the character controller interacts with a physics object ?

I have tried putting my capsule collider as Query only and mesh as Query and Physics but it seems that having Query and Physics makes things go wild and objects that the player is standing on start to fly around and stuff. I would love to still be able to push objects around but obviously without making them and myself fly at the same time.