RecastNavMesh-Default → Details → Max Simplification Error
I set this super high and it fixed my issues.