Hello!
We had run into this previously for Lego FN. We introduced a CVar for MassLODSubsystem to include all player controllers regardless of current camera/pawn. The CL# is 29665207 in UE5 Main, but I believe it should be present in the 5.4 stream as well at CL 30834909. The CVar name is bLODSubsystemIncludeAllPlayerControllers.
Does enabling that CVar address the issue you are having? It is always possible that we did not account for an edge case that you are encountering.
-James