Hey Jikinto!
So the issue is that the Agent’s size comes into play. The navmesh itself is only caring that your capsule’s midpoint, its 0,0,0, is within the navmesh. The solution here is to either yes, give the character something to collide with, or use a Navmesh Modifier to cut out area you don’t want there to be navmesh. Alternatively, you could change the Agent size by going to Edit>Project Settings(or Editor settings? I think it’s Project Settings) then search for navmesh and you can set the agent to be smaller/larger. I’d play around with that and see what interaction you get! Let us know your results!