So does it ever work? Like it works on the other level for both or only for default agent on both levels?
Did you check the navigation mesh settings in project settings? Not the agents but the other one, it has things in generation like “minimum agent radius” and “maximum agent height” and I believe that basically makes your agents invalid if they don’t fit into that.
The whole agent thing is a tad overcomplicated for me tbh. to the point that I had to write c++ changes despite never really programming in c++ (did some c# back in the day but hey, copy paste is a wonderful thing) as Navigation Raycast was taking “controller” instead of pathfinding context which could take the recast navmeshes like in FindPath (and thus work) ALWAYS resulting in usage of the default agent obviously not helping when I have more agents. Thinking of making a blueprint callable node of “get agent props” or something and navigation nodes I can plug it into, maybe turn it into a plugin? Currently on UE4.27.2 so not sure if it will be of any use to others but who know… might give up halfway as is 
The navigation raycast node does work though so I’m already happy.
bool UCustomNavigation::CustomNavRaycast(UObject* WorldContextObject, const FVector& RayStart, const FVector& RayEnd, FVector& HitLocation, TSubclassOf<UNavigationQueryFilter> FilterClass, AActor* PathfindingContext)
{
UWorld* World = NULL;
if (WorldContextObject != NULL)
{
World = GEngine->GetWorldFromContextObject(WorldContextObject, EGetWorldErrorMode::LogAndReturnNull);
}
if (World == NULL && PathfindingContext != NULL)
{
World = GEngine->GetWorldFromContextObject(PathfindingContext, EGetWorldErrorMode::LogAndReturnNull);
}
// blocked, i.e. not traversable, by default
bool bRaycastBlocked = true;
HitLocation = RayStart;
const UNavigationSystemV1* NavSys = FNavigationSystem::GetCurrent<UNavigationSystemV1>(World);
if (NavSys)
{
// figure out which navigation data to use
const ANavigationData* NavData = NULL;
INavAgentInterface* MyNavAgent = Cast<INavAgentInterface>(PathfindingContext);
if (MyNavAgent)
{
const FNavAgentProperties& AgentProps = MyNavAgent->GetNavAgentPropertiesRef();
NavData = NavSys->GetNavDataForProps(AgentProps, RayStart);
//UE_LOG(LogTemp, Warning, TEXT("NavRay_MyNavAgent VALID"));
}
if (NavData == NULL)
{
if (Cast<ANavigationData>(PathfindingContext))
{
//UE_LOG(LogTemp, Warning, TEXT("NavRay_NavAgent INVALID but Context Cast SUCCESS"));
NavData = (ANavigationData*)PathfindingContext;
}
else
{
UE_LOG(LogTemp, Warning, TEXT("NavRay_NavData casting FAIL"));
NavData = NavSys->GetDefaultNavDataInstance();
}
}
if (NavData != NULL)
{
bRaycastBlocked = NavData->Raycast(RayStart, RayEnd, HitLocation, UNavigationQueryFilter::GetQueryFilter(*NavData, PathfindingContext, FilterClass));
}
}
return bRaycastBlocked;
}