Hi Zachary,
This could be due to the fact that Fully Async Nav Data Gathering is done with only a single worker thread. It is also limited to a single thread or else it might crash as the engine does not support parallel gathering. Also, using async data gathering limits to building for a single tile at a time which is related to the gathering being on a single thread.
We do make heavy use of invokers for Lego FN with 4+ supported agents and each agent has its own navmesh to be built. Several systems were created to help with invokers such as the seeding and priority of invokers. You can also set an invoker to only build for a subset of the navigation agents instead of building for all agents. You could also look into changing the NavRegenTimeSlicer settings to change the amount of time spent on building tiles per frame. We dynamically change these values throughout gameplay in FNBR based on the current state of the match.
-James