replication with nDisplay cluster

Development Programming & Scripting Multiplayer & Networking
, ,
1

I’m trying to replicate in the context of an nDisplay cluster, but the replication does not seems to work. I’ve probably missed somehthing. By the way, in a server/client configuration the replication i works.

In the attached small repro case for property and movement replictions. I’ve made a BP actor « MovingActor » that move itself and increment a property « Counter ». This is done only on the IsMaster condition (i.e. only on server OR primary node). On all instances the tick triggers the display update and some onscreen printing.

1/ client/server: launching from the editor with netmode to listen server and 3 players => debug message are ok on server AND clients

2/ nDisplay cluster: launching with nDisplay launcher with node_front, node_left and node_right checked. Primary set on node_from => debug message are ok only on primary BUT NOT on secondary

BTW:

  • actor has “Replicates” to true
  • actor has “Replicate Movement” to true
  • actor property Counter has “Replication” to"Replicated" and “Replication Condition” to “None”
  • nDisplay cluster has “Render Sync Policy / Type” to “ReplicatePrimary”

Could you please have a look of what’s going wrong and what I’ve missed ?

2

Steps to Reproduce
See description

3

Hello Stephane,

The editor multiplayer is not supported at this moment, so this won’t work, unfortunately.

Let us take a look, and I will be back soon with instructions how to get replication up and running without dedicated server.

4

Let me rephrase because I’m not sure to correcly understand or maybe using the right terms:

  • In editor i’m just using listen server+clients for prototyping and testing. It’s “Just” replication like classic multiplayer gaming without nDisplay stuff. => it works but this is not the target, just tests.
  • The target is to have packages acting as nodes of an nDisplay cluster with replication between primary node and secondary. For prototyping I’m using nDisplay Launcher that launch editor nodes. => it doesn’t work and it’s our goal

1/ One simple dummy additional question => when you have to share a state between all nodes of a nDisplay cluster, let’s say the position of a spaceship or a blinking light. how do you do that? where and how to you set states, share them and update the visual?

2/ with unreal ndisplay can I replicate property between primary node and secondary nodes?

5

Hello Stephane, there is seems a small bug in 5.5.

Enabling replication in project settings should write specific parameters in the ini file which is missing in your project and I confirm that behavior in 5.5

Please add the following lines manually in the Config/DefaultEngine.ini in your project

[/Script/Engine.GameEngine]
+NetDriverDefinitions=(DefName=GameNetDriver,DriverClassName=/Script/DisplayClusterReplication.DisplayClusterNetDriver,DriverClassNameFallback=/Script/OnlineSubsystemUtils.IpNetDriver)
+NetDriverDefinitions=(DefName=DemoNetDriver,DriverClassName=/Script/Engine.DemoNetDriver,DriverClassNameFallback=/Script/Engine.DemoNetDriver)
 
[/Script/DisplayClusterReplication.DisplayClusterNetDriver]
NetConnectionClassName=DisplayClusterReplication.DisplayClusterNetConnection

can I replicate property between primary node and secondary nodes

Yes.

Please let me know how it’s going with the fix above!

6

I was able to get some progress. Changes I’ve made:

  • manually add network drivers in DefaultEngine.ini
  • enable multiplayer in order to have the primary node running a listen server multiplayer (based on your 2023 document Franck send me).
  • launch evertything with Switchboard (I was using “nDisplay launcher” that doesn’t lead to the same result even when multiplayer is enabled in this plugin). Does the bFollowLocalPlayerCamera parameter in the nDisplay config file is available in the nDisplay launcher advanced settings? It could be helpful for iteration.
  • import of a ndisplay configuration 5.5.x file in 5.6.1 => crash. Need to rebuild config from scratch.

Still somes issues / questions to be fully usabled:

  • jerky motion replication. Note that My test case is very simple and launch 3 editor nodes on the same computer.
  • very frequently one of the nodes is not synchronized at startup and never sync
  • over time, the player’s camera becomes out of sync. Even some nodes no more respond.
  • in Switchboard multiplayerServer devices seems to be mandatory. Probably embedded in the primary node when setting “Multiplayer Server Mode” to “Listen Server”
  • In Switchboard, Iin the “Tools” menu, I do not have “Testing” as mentionned in your documentation.

I will post the updatd project.

7

Hello Stephane,

For a matter of test, we also launched both dedicatedandlisten servers for testing, and it seems all is working well.

Please find comments below:

Jerky motion replication. Note that my test case is very simple and launches three editor nodes on the same computer.

This is the nature of networking. Data packets are not sent for replication every single frame but gathered in buckets and has own algorithms for distribution. You might try improving the behavior by tuning up the replication rate parameters but it wont be perfect unless you implement your own smoothing alignment methods that works specifically for your case.

I would recomend to look into the Vehicle template and code that handle movement which is not simple.

  • very frequently one of the nodes is not synchronized at startup and never sync

Over time, the player’s camera becomes out of sync. Even some nodes no more respond.

The default player pawn and controller are not designed for nDisplay and synchronous capabilities. You have to implement custom input that only works on the “driver” machine and control camera too and then passes to the nodes. I dont think that replication is the best methods for camera, cluster events are much more robust in that regard.

in Switchboard multiplayerServer devices does not seems to be mandatory. Probably embedded in the primary node when setting “Multiplayer Server Mode” to “Listen Server”

There is 2 modes of operation, dedicated server - standalone console application which run server side and must be compiled from source, and listen server where primary node do the role of the server, keep in mind that listen server node going to be always one frame forward than cluster because its not only server but node at the same so there is no delay for packet transfers.

In Switchboard, Iin the “Tools” menu, I do not have “Testing” as mentioned in your documentation.

This portion was experimental and not part of the Switchboard.

8

On the video you’ve shooted, there’s visible delay/stuttering/jerky movement on secondary nodes (specifically visible on the top-right node in the video). This point is unacceptable for our production needs. Even the counter lags a lot between the nodes. We are a bit disappointed that this “simple” case is not handled by the default Unreal implementation.

To summarize the current thread :

Useful findings are:

  1. nDisplay replication isn’t automatic - it requires enabling multiplayer functionality
  2. Primary node must act as listen server for replication to work
  3. Manual network driver configuration is needed in DefaultEngine.ini

Outstanding issues needing support are:

  1. bug? Camera Synchronization at Startup: with bFollowLocalPlayerCamera set to true, one or more nodes frequently fail to sync the camera player position at startup and never recover. I never was able to have more than one secondary node synchronized. Note that IF i disable the listen server used for replication all nodes are synched to the camera. Does the bFollowLocalPlayerCamera works when multiplayer is enabled in nDisplay
  2. Jerky Motion Replication: The stuttering movement on secondary nodes remains a critical problem
  3. Node Positioning Problems: On some PCs, the x/y positioning settings for nodes are ignored, causing all nodes to center on the main screen instead of their configured positions (possibly related to admin/authorization issues in Switchboard launch scripts
  4. Enabling the nDisplay plugin, required that the machine that load the project has access to internet in order to download and install its own version of Python. How can we bypass that? We’ve tried to copy Engine\Extras\ThirdPartyNotUE\SwitchboardThirdParty by hand without success: launching an unreal editor with nDisplay enabled block on Python if internet is not available.
  5. bug? Export a ndisplay file from a ndisplay root actor always reset bFollowLocalPlayerCamera to false
  6. bug to be fixed: netdriver are not correcly put in settingds when enabling the cluster replication
9

Hello Stephane, please find comments below:

  • nDisplay replication isn’t automatic - it requires enabling multiplayer functionality

First, let me double-check whether you are aware of nDisplay Cluster Events?

This synchronized communication mechanism was explicitly developed for nDisplay to trigger events and share data per frame, guaranteed.

It is used for external inputs and a variety of other cases (Traffic data coming from a simulation server, etc)

Please, check it out: https://dev.epicgames.com/documentation/en-us/unreal-engine/using-cluster-events-with-ndisplay-in-unreal-engine

  • The primary node must act as listener server for replication to work

If you still consider traditional multiplayer, which is instead used for more complex, multiplayer titles with complex game logic and using a variety of interpolation mechanisms, then the preferred method is to use a dedicated server, listen server on the primary is rather a debug mode for tests and can’t be used in production with primary visible at the same time with other nodes since it is always 1 frame forward due to nature of replication.

  • Does the bFollowLocalPlayerCamera work when multiplayer is enabled in nDisplay

Multiplayer replication support is indeed limited, and this type of behavior for DefaultPawns is not supported. It requires input controls takeover and pawn\camera positioning, etc, etc.

  • causing all nodes to center on the main screen instead of their configured positions

Make sure that you have set Scale to 100% under the Scale & Layout section within Display Settings (Desktop). Values other than 100% could break some windows resize/positioning logic. We will verify that on our end, too. Could you please share log file with us too?

  • Switchboard offline installation. Here is a short guide:

How to deploy Switchboard at airgapped PC’s

1) On a machine that has internet access, run this command to download the Python packages from PyPI and store them locally:

> cd /d “D:\P4\…\Main”

> Engine\Binaries\ThirdParty\Python3\Win64\python.exe -m pip download -r Engine\Extras\ThirdPartyNotUE\SwitchboardThirdParty\requirements.txt -d Engine\Extras\ThirdPartyNotUE\SwitchboardThirdParty\download\

2) Take the resulting Engine\Extras\ThirdPartyNotUE\SwitchboardThirdParty\download\ directory and copy it to the airgapped machine(s) of the same engine version / OS / architecture:

3) On the airgapped machine, tell

pip

to look in the provided directory during SB first-time setup:

> set PIP_NO_INDEX=1

> set PIP_FIND_LINKS=D:\P4\…\Main\Engine\Extras\ThirdPartyNotUE\SwitchboardThirdParty\download

4) Run

switchboard.bat

as you typically would:

> D:\P4\…\Main\Engine\Plugins\VirtualProduction\Switchboard\Source\Switchboard\switchboard.bat

  • Export a ndisplay file from a ndisplay root actor always reset bFollowLocalPlayerCamera to false

Thank you for bringing this up! We will double-check on our end and fix it!

  • netdriver is not correctly put in settings when enabling the cluster replication

Yes, known bug we expect fix to come through in 5.7

10

Stephane,

we could not reproduce following issue:

  • Export a ndisplay file from a ndisplay root actor always reset bFollowLocalPlayerCamera to false

Could you please help us with more specific step by step repro? Thank you!

11

The repro case WAS (but I’m no more able to reproduce it, the only change I’ve made is the airgapped Python fix whom I don’t think is related):

  • edit *.ndisplay file and set bFollowLocalPlayerCamera to true
  • launch a project that contains a nDisplay Root Actor
  • edit the actor loaded with this file and do export and overwrite => bFollowLocalPlayerCamera is now back to false (as there is nowhere in the editor where this flag is viewable/editable)

Note that in 5.6.1, “Import” is crashing sometimes.

BTW I’ll be vigilant and overwatch if it happens again and send you back a repro.

12

The airgapped Python fix is working fine => checked.

13

Regarding the position of the windows, we have a guys in charge of overriding by code these position from other internal configuration. We will come back to you if required. And on my side I’m trying to find a computer that repro the issue => standby

14

Stephane, thank you for reporting the crash on import. The team is working on the fix!