Sep 13, 2021.Knowledge
When creating a multiplayer game or networked project in the Unreal Engine, profiling, testing and debugging your project may be more difficult or confusing than for a single player experience. You’ll need to contend with running multiple instances of your project, the different functionality present on clients vs servers, and the general unreliability and instability that comes with networked communication. This article seeks to provide some guidance and best practices on how to test and identify networking issues in your project.
Testing In the Editor
To help with testing multiplayer games, the editor provides several options for running multiple instances of a project. You can find more information here: Testing Multiplayer | Unreal Engine Documentation
Beyond the options listed above, there are a few other ways to launch a multiplayer game. If “Launch Separate Server” is selected and the “Play Net Mode” is set to “Play Standalone,” a dedicated server instance will still be created, but the other instances won’t automatically connect to it. You can then connect these standalone instances to the server, such as with the command “open 127.0.0.1:” or by using the session interface, which can be useful for testing your project’s connection flow. Additionally, if the “Play Net Mode” is set to “Play as Client,” “Launch Separate Server” does not need to be enabled for a dedicated server instance to be launched.
When using the different multiplayer “Play Net Mode” options in the editor, these instances are automatically connected directly to each other by IP address, the equivalent of running the “open 127.0.0.1:17777” command on a client to connect to the server. This is important to note, as this connection process does not use the Session Interface. The server will not create an online multiplayer session, and clients will not search for and join this session. For most gameplay testing purposes, this won’t make a difference, but certain online features that rely on the session interface, such as voice chat, will not be available.
There are subtle differences between these various modes that may not be immediately obvious. For example, if “Run Under One Process” is selected, all client and server instances will share the same tick rate as the editor. This is different to running these instances separately, such as in standalone mode, where you can control the server’s tick rate using the NetServerMaxTickRate config value. This can affect certain behavior which uses the engine’s tick rate, like calculating the bandwidth limit for a single net update. Certain functionality is also not supported in PIE, such as server/client traveling. Your project will also need to be run in standalone mode as a separate process outside of the editor in order to test these features. Finally, when running under separate processes, one process will be considered a PIE instance and the others will be considered standalone, which may lead to some differences in behavior when compared to running all instances as either PIE or standalone. For example, UEditorEngine::NetworkRemapPath will need to be called in order to remove/add PIE prefixes on static actors’ paths sent across the network.
When debugging networking issues while running multiple client and server instances, it can be difficult to know which instance you’re attached to. When using breakpoints to debug a PIE instance, adding “UE4Editor-Engine!GPlayInEditorContextString” to your watch window can help determine which instance you’re currently stepping through. Another property that can be useful to watch is the NetDriver’s ServerConnection. On clients, this will hold a reference to the NetConnection to the server, and on the server, this value will be Null, allowing you to quickly check which instance you’re in when debugging the replication system.
Another useful trick is to check an actor’s Role property, either by placing a call to GetLocalRole() in a watch or by calling it directly in code. This function will return how much control the instance has over this actor. If you’re debugging issues within a replicated actor, GetLocalRole() will return one of three things:
- ROLE_Authority, meaning this actor is on the server instance
- ROLE_AutonomousProxy, meaning this actor is a character or pawn possessed by a local PlayerController on this client instance.
- ROLE_SimulatedProxy, meaning this actor is on a client instance.
When testing networking functionality in the editor, it is very important to run tests with the Network Emulation Settings enabled. Without these settings enabled, running multiple instances on one machine will create an almost impossibly ideal environment for your replicated data. While the replicated data will be passed through the appropriate systems in the same way as in a real networked environment, there will be no lag or packet loss. The Network Emulation Settings allow you to simulate lag and packet loss on either the server, clients, or both, and can be crucial in identifying issues that may arise in average and less-than ideal networking environments.These settings are configurable in the editor, in configs, and through the console, allowing for multiple ways to use these settings in different contexts, such as testing high latency, packet loss, or jitter. You can find more information on how to use these settings here: Finding Network-based Exploits - Unreal Engine
Troubleshooting Connection Issues
If you are experiencing issues connecting client and server instances, the following are some troubleshooting tips:
- Ensure you don’t have any extra network adapters active (can occur with tools like VirtualBox), and ensure that you are using the IP address for your primary ethernet adapter.
- You can add the -notimeouts argument to both the client and server to ensure you aren’t hitting timeouts. Timeouts are by default disabled in PIE.
- Ensure your server can correctly bind to the default port: 17777
- Ensure the server and client Net CLs, as well as their game versions, match. These can be found in the client and server logs under the LogNetVersion category.
- Ensure the instances are using the same or compatible NetDrivers and Online Subsystems.
- If using Online Subsystem Null, ensure the server and client instances are on the same local area network.
Gauntlet and Functional Testing
The Gauntlet Automation Framework supports launching multiple sessions, such as a server and clients, and it can be an invaluable tool for testing and validating your multiplayer project. To learn more about Gauntlet, a good starting point is our Gauntlet Primer knowledge base article: Gauntlet Primer
There is also an implementation example of using Gauntlet for a multiplayer game in the ShooterGame sample project. [ProjectRoot]/Build/Scripts contains a ShooterGame automation C# project that drives the tests, and the project’s native test controller code is located in [ProjectRoot]/Source/ShooterGame/Private[ and Public]/Tests/.
The engine also includes the ability to set up and run Functional Tests through the level blueprint, although running these tests in a multiplayer project may be more complicated than in a single-player one. If your functional tests just need to run on one instance of your project, running these tests in a multiplayer project should work the same as any other project. You can start the level containing the tests in a dedicated/listen server instance or a client instance, and the tests will run as normal.
However, you may want to set up a functional test that runs on both a server and client(s). For example, the server sets a replicated property to a new value, and then the client checks that this new replicated value was received. Setting up a functional test that runs across multiple instances like this is more complicated, and there currently isn’t any formal support for this kind of testing. However, the EngineTest project (located in the engine’s perforce streams) has several networking test maps that run both blueprint and C++ functional net tests, which are made up of client and server steps that run on their respective instances. However, these networking functional tests are currently only in the EngineTest project due to how the EngineTestNetPlayerState is used in conjunction with these functional tests. If you need this kind of functional testing, it is recommended to copy over the main functionality of EngineTest’s net functional tests and the EngineTestNetPlayerState to your project and use these net functional tests as a reference for implementing your own.
Another important part of testing a networked multiplayer game is profiling. Networking Insights, part of the Unreal Insights profiling tool, provides detailed information in order to help analyze, debug, and optimize a project’s network traffic. Each column in the tool’s Packet Overview Panel corresponds to a single packet, and the Packet Content Panel offers a comprehensive look at each individual element within a selected packet, including data on the content, offset, size, and more. The Net Stats panel provides information on networking trace events, such as the event’s count and total/max inclusive size in bits, and these events are organized into levels based on where the event originated. You can find more information on how to set up and use Networking Insights here: Networking Insights Overview | Unreal Engine Documentation
Also included with the engine is the Network Profiler, an older tool which offers a different view of a project’s network traffic. While the information this tool provides is less detailed than that of Networking Insights, it can still be used to provide a high level overview on your game’s bandwidth usage as well as stats on individual actors, properties, or RPCs.
Checking the logs of your client and server instances is important for identifying and debugging networking issues. While many networking logs fall into the “LogNet” category, there are a number of related categories that can also contain useful information. Depending on the problem, checking the related log category can provide greater insight into the issue. However, some of these categories are not enabled by default, and many have varying amounts of verbosity that you may want to adjust in order to gain as much information as possible on the problem you’re experiencing. The following is not a complete list, but here are some categories to look out for:
- Setting this to VeryVerbose will log all network traffic, which may be helpful but can also quickly bloat your log files
- For information related to how NetGUIDs are sent, received, and acknowledged
- LogRep and LogRepTraffic
- These categories are mainly used by FRepLayout and FObjectReplicator and are related to property replication and repnotify functions
- For more detailed information on the sending and receiving of replicated properties
- For information on the packet handler and its components. Many of the components have their own categories as well, such as LogDTLSHandler, OodleNetworkHandlerComponentLog, and LogHandshake
- For information on recording and playing back replays. Each replay streamer has a related log category as well: LogLocalFileReplay, LogSaveGameReplay, LogNullReplay, and LogMemoryReplay
You can enable these categories and adjust their verbosity either by passing in the command line argument -LogCmds=“ ”; using the console command “Log ”; or by setting them in your project’s DefaultEngine.ini under “[Core.Log]”. For example:
[Core.Log] LogNetPackageMap=Log LogNetTraffic=Verbose LogRep=VeryVerbose
When reading logs, there are a few key lines that can be useful for determining what kind of error occurred. The first is “UEngine::BroadcastNetworkFailure,” which will be printed when a net driver encounters some major error. The log line will include the failure type, the error string, and the description of the net driver that encountered the error. You can see a list of the possible network failures with brief descriptions in the ENetworkFailure enum in EngineBaseTypes.h.
Another useful line to look out for is “UNetConnection::Close,” which will include a description of the connection being closed. If you’re encountering issues with a specific actor’s channel being closed, “UActorChannel::Close” will be logged under the LogNetTraffic category and will include the channel index, the actor for that channel, and the reason it was closed. Checking the logs around these lines can help provide some indication as to why a connection or actor channel was closed.
Finally, the command line argument “-LogTrace=” allows you to do stack traces from partial log message strings. For example, “-LogTrace=”UNetConnection::Close”” will produce a stack trace whenever “UNetConnection::Close” is printed to the logs. The command line argument “-DumpRPCs’’ allows you to dump RPCs and their parameters, which can be very useful for tracking what RPCs are being sent and what their parameters are. LogTrace and DumpRPCs require NetcodeUnitTest to be enabled.
Outside of the logs, the engine does include a number of console variables and console commands that can be helpful for debugging. While this is not a complete list, some that you may find useful include:
- Can be enabled to draw debug information for replicated actors’ dormancy and relevancy
- Prints a list of all actor channels to the log
- Prints a list of all NetGUIDs to the log
- Can be enabled to print a log message every time a reliable bunch is sent
- Debugs the PackageMap serialization of all objects
- Debugs the PackageMap serialization of a specific object
- Debugs replication of a specific property
- Can be enabled to print all RPCs sent
- Can be enabled to have the Obj list print out info on network memory usage
- Can be enabled to draw debug info for replicated movement
- Can be enabled to draw color-coded client position corrections
- Demorec , Demostop, and Demoplay
- Can be used to record and playback a replay from the console
For issues concerning the Replication Graph, you can find the full list of debugging commands in ReplicationGraphDebugging.cpp. Some useful commands include the “Net.RepGraph.PrintGraph” command, which will print the full contents of your Replication Graph. For information on specific actors within the Replication Graph, “Net.RepGraph.PrintAllActorInfo ” can be used to print global and connection specific info about actors whose pathname contains . Additionally, the Replication Graph also provides the ability to set a specific actor as the Debug Actor through the “Net.RepGraph.SetDebugActor ” command, allowing you to breakpoint on a specific actor when debugging the graph.