Combination of Server Streaming, World Partition and Iris causes instant player disconnection in PIE.

Development World Creation
, ,
1

This question was created in reference to: [Server closing client connection with MissingLevelPackage in [Content removed]

Hi,

Our project (UE 5.7.2) uses Iris and requires server streaming. Unfortunately, if I start a PIE single process (two clients, one of which is ListenServer), the combination of those two causes automatic player disconnection due to MissingLevelPackage.

It’s worth noting that if we disable Iris or start clients as standalone editors, everything works fine.

Here are a few of our cvars that might be related to the problem:

[SystemSettings]
net.SubObjects.DefaultUseSubObjectReplicationList=1
net.IsPushModelEnabled=1
net.Iris.UseIrisReplication=1
net.PushModelSkipUndirtiedReplication=True
net.PushModelSkipUndirtiedFastArrays=True
Controller.IsPushBased=True
PlayerController.IsPushBased=True
 
wp.Runtime.EnableServerStreaming=1
wp.Runtime.EnableServerStreamingOut=1
LevelStreaming.DefaultAllowClientUseMakingVisibleTransactionRequests=1

Best regards,

Szymon

[Attachment Removed]

2

Steps to Reproduce

  • Set cvars to values listed below:
net.Iris.UseIrisReplication=1
 
wp.Runtime.EnableServerStreaming=1
wp.Runtime.EnableServerStreamingOut=1
LevelStreaming.DefaultAllowClientUseMakingVisibleTransactionRequests=1
  • Load the map with world partition.
  • Enter PIE (single process) with two players, one of which is a listen server
  • Notice that the second player is instantly kicked from the session with the MissingLevelPackage error.
    [Attachment Removed]
3

There is another finding worth noting:

The provided CVar setup only works when we start two standalone processes from the editor GUI. If we run two clients or a client and a dedicated server using the command line (-game and -server flags), clients are also kicked just after joining.

Packaging a game and starting a listen server using an open command works fine.

[Attachment Removed]

4

Hi,

To get a better idea of the problem, I have a couple questions.

First, I just want to confirm that “Run Under One Process” is set to true in your case. We did recently find a bug with how package names are remapped when the listen server is in the editor’s process while the client is in a separate process (UE-363862).

Next, just to make sure there isn’t some issue with how the engine uses these CVars, does the issue still occur if wp.Runtime.UseMakingVisibleTransactionRequests and wp.Runtime.UseMakingInvisibleTransactionRequests are set to true?

Finally, could you provide client and server logs from when the issue occurs, with the verbosities increased for the LogNet, LogIris, LogIrisReferences, LogLevel, LogStreaming, LogLevelStreaming, and LogPlayerController categories?

Thanks,

Alex

[Attachment Removed]

5

Hello Alex,

Thank you for your quick response.

Firstly, I am 99% sure that I am running a single-process client and listen server setup. Unfortunately, as I wrote in a previous message, the issue is also affecting situations where the client and the listen server are separate, standalone editor processes.

Answering your second question, I tested our setup with those two cvars set to true, but sadly without success. In the attachment I added logs from a few runs on different client+server configurations.

However, our team did some more investigation, and [mention removed]​ found an indirect reason for what causes those strange streaming issues. In the scenarios where a player gets kicked, a GameState’s `bReplicatedHasBegunPlay` variable gets replicated (and has its OnRep notify called) before the PlayerController’s Actor channel gets open (and tells the connection to handle Client Player). This causes a Begin Play Notification during which the World’s Net Driver Server connection is still marked as Pending. During the Begin Play process, the World Partition forces the Level Streaming completion, during which the streaming logic updates the levels’ visibility, but the `ULevelStreaming::IsConcernedByNetVisibilityTransactionAck()` function returns false as it requires the Server Connection to be in an Open state, but it’s still Pending in this case, so the client ends up not waiting for that Visibility Transaction ack from the server. If we add a workaround that we delay the BeginPlay Notification until the Player Controller’s Actor channel opens (which changes the connection state to Open), then the client no longer gets kicked in any of our mentioned test cases.

We have no clue why the bunches related to that replicated Begin Play and Player Controller channel are received by the connecting client in the reverse order in the problematic scenarios I described earlier. We tried increasing the Player Controller’s NetPriority to a value higher than the GameState’s NetPriority, but it didn’t help with this issue.

Please feel free to reach out if you would like to learn more about our workaround. We would be happy to provide you with additional details, and I hope that info and logs will help you in the investigation.

Best regards,

Szymon

[Attachment Removed]

6

Hi,

Thank you for the logs and the additional information! I’ve been able to reproduce this in Lyra, and I believe I have a better understanding of what is happening.

As you pointed out, if the client loads these streaming levels before the connection is opened, ULevelStreaming::IsConcernedByNetVisibilityTransactionAck will return false, so the client won’t wait for acknowledgement from the server before making these levels visible and adding them to the world. When UNetConnection::HandleClientPlayer is later called, the client will send a ServerUpdateMultipleLevelsVisibility RPC containing all of its visible levels.

Because the client didn’t send visibility transaction requests for these levels, the server may not have them loaded, causing the MissingLevelPackage error and disconnect when the server receives this RPC.

Like you said, BeginPlay will force async loading of these levels to complete, so it makes sense that if OnRep_ReplicatedHasBegunPlay is called before HandleClientPlayer, this error can occur. We don’t guarantee that actors will be received in any specific order on the client, so depending on packet loss and latency, its possible for the client to receive the PlayerController before the GameState and vice versa, regardless of priority.

Your workaround does make sense, but it’s worth noting that I was able to observe this error occur even when the player controller was received before the GameState. In that case, some streaming levels completed loading on the client and were added to the world before the client had received either received the PlayerController or the GameState.

I’m not sure what the best solution here would be. We may need to make sure the client can’t add levels to the world when the connection is pending, but I’m not sure if that would cause some other undesirable side effects. I’ve opened a new issue for this, UE-364613, which should be visible in the public tracker in a day or so.

In the meanwhile, your workaround does seem reasonable if that is the only situation in which you’re seeing the disconnect occur. In some cases, you can also enable the “net.SkipMissingLevelDisconnect” CVar to ignore these warnings, although we only recommend doing so for testing purposes.

Thanks,

Alex

[Attachment Removed]

7

Hello,

Thank you for your assistance; we are looking forward to the bug fix.

Let us know if we can help you.

Best regards,

Szymon

[Attachment Removed]