Weird routing in navmesh

I have an odd issue that I can’t wrap my head around. I’m trying to learn to use the navigation mesh, in a setting where geometry can be added at runtime.

The issue is that sometimes when adding an object, the nav agents take a big detour, just to get around that new object. If I add new objects after that, sometimes the detour will be cancelled.

I tried using the visual logging tool to visualize the issue. The screenshots are from after adding three objects while the agent was navigating.

The first path is straight from the start to finish:

After adding the top-left object that is not blocking according to the green navigation mesh seen below the teal section, the agent takes a detour around that object.

Then when I add a second object (top-right) the path changes again to the straight line.

Can anyone see what might be going on here? Any tips for what I can try? I have tried a lot of different stuff at this point, but I might be missing something.

Thank you!