Multiplayer, replication getting past the limit of AI bots

Development Programming & Scripting Multiplayer & Networking
1

For testing i created very simple snail AI bot, that just randomly wanders.
Then i tried to test how many of those i can spawn on server and replicate.
The bad thing is that i can do only around 20-25 without noticeable lag. This is very bad news.

So any ideas how to put around 400 of those AI bots in level? Why 400, its because that is when my PC starts to choke with non replicated version, so i kind of want to eliminate soft cap from replication.

My first idea is to replicate only AI orders. For eg server manages all bots, gets random destination for new order, tells bot to go there.
And instead of replicating that bot movement, it replicates only event that gives order “go to [X,Y,Z]”, all snail bots on clients will got there on their own.
But before i rewrite half of AI code, did anybody try something like this? I am worried that in big crowd even slight difference to when event fired may give different outcomes, so clients will see wrong state of snail swarm.

Ps.
Are ther some useful console commands to see what exactly eats up bandwith. I also suspect that this lag may be due to collision of snail to snail, or maybe navigation goes nuts with 30 of snails trying to find path. So console commands for seeing what is going on with navigation and how much collision costs would be great.

2 Likes
2

Unfortunately I haven’t played around with multiplayer yet, but I hope my input can still help you.

Have you tried a deterministic approach?
Try synchronizing a seed, assuming the numbers are generated in the same order on all clients, the destinations (and of course other things) should be the same. Of course this only works with simple AI, as soon as it reacts to stuff (e.g. player) it gets more complicated.

Many games calculate on the client and cross check/update only rarely. You could try updating only parts of your swarms (e.g. in bunches of 25).

3 Likes
3

Use Network Profiler to see what is actualy going to. And did this 400 snail thing work when you were not replicating movement, like in singplayer?

2 Likes
4

Yeah it seems strange that 20 to 25 mobs are causing you to lag. You must have something wrong there. I have seen 200+ mobs replicating basic movement without any noticeable lag.

Are you just letting the actor itself replicate its movement or are you replicating all the ai logic? Cause just the move actor should not be hitting a limit at / around 20 - 25 or so. Should be able to do way more then that.

2 Likes
5

I did some more tests on it (also redesigned my whole multiplayer setup). It is a bit better, but nowhere around 400 or so ai enemies.

I am animating all of them on server side, then using built in replication. This is what i try to change next. Ie doing them on server side then replicating only locations, so clients can place their own enemies that are not replicated. If this does not help i will place (above certain density) pack of enemies instead of single actors. But that is probably too much coding for small game feature (ie huge crowds/swarms). There was some instanced crowd support back in UDK times, this would help me for my problem. However i seen nothing like that in ue4.

I think big part of problem was collision, they were simple AI, after changing them to detour, i could go to 70 or so.
Also I am running 3 game instances on quite powerful rig (980ti skylake, i7, 32gb ram), Single instance could probably go easy to 200 or so.
But those 3 instances help me estimate how game would run on some older hardware.
There is also network saturation that plays huge role in this lag.

2 Likes
6

Have you looked at the debug stats to see how much is being sent / num of packets /ect. ?

3 Likes
7

What kind of lags are you experiencing?

3 Likes
8

With about 100 of actors (and 3 game instances running on single PC) ai actors start warping.
I know it is a lot for single PC to handle, but mine is quite good, and running those 3 instances should simulate load on something around minimal config i am aiming at.

If i run this same game 3 times without lan simulation it has low fps (20-30) but no warping.

I haven’t look into lan stats, but I think it is saturated lan connection.

9

I know this is an old thread, but I’m having the exact same problem. Once I have about 20+ AI characters wandering around, then their movement on the client becomes noticeably jerky. This is running separate instances of the game with just two players over a local network. I can’t believe that the underlying networking is so inefficient that it chokes on a LAN with just 20 characters doing nothing other than wandering around. There must be something wrong.

For anyone interested I created a bare minimum project to reproduce the problem. You can download it over at my AnswerHub post if anyone wants to try it out to see what results they get.

10

I noticed the same. Not for the AI movement replication itself, but when doing other things in parallel. When having 25 AI Actor moving around and starting some actions which relocated from server to client, they have some higher lag. I also have some minor meshes in the world with physics simulations on, when I hit around 20 AI characters moving, the physical simulation of the the actors (which are set to replicate movement also) begins to get really, really laggy and rubber-banding.
Welcome for any advises.

1 Like
11

I also need help with this. Does anyone know of a guide on how to reduce rubberbanding on AI movement with blueprints? I find it annoying that my players’ movements are fine while AI rubberbands. I think this is the last challenge I will be facing for my project.

Do I have to pay someone to teach me? Where do I find a ue4 teacher?

1 Like
12

I haven’t tried it yet, but there is a new tool for analyzing the network performance.

https://docs.unrealengine.com/en-US/…hts/index.html

1 Like
13

Despite this thread being active from 2016-2020, I think this is a timeless issue, and it might be helpful to have a reply here for people struggling with it.

There are two typical issues that one runs into trying to increase the number of replicated actors (whether AI or player controlled).

  1. The first is that, by default, Actors are replicated in really general and simple ways that are not network optimized. Just beacuse you made an AI “just wander around” does not mean it is taking less replication bandwidth. There are ways to optimize networking to control the priority of different Actors, making actors far away or less important receive less frequent updates. This article is a pretty clear overview of the concepts:

How Epic Games optimized Unreal Engine for Fortnite Battle Royale ? | by Sagar | Medium

This type of optimization uses plugins like the Significance Manager and Replication Graph, which you must customize for your project with C++ code.

The Significance Manager | Unreal Engine 4.27 Documentation

Networking in 4.20: The Replication Graph | Feature Highlight | Unreal Engine Livestream - YouTube

Replication Graph | Unreal Engine 4.27 Documentation

  1. The second is related to custom movement and movement prediction. The built-in hardcoded Unreal movement component handles network replication and movement prediction for you, but only for the built-in actions. However, as soon as you start writing custom movement actions in blueprints and animations, your blueprint code does not have movement prediction logic in it. You won’t notice this problem when you test locally, because there is near zero latency between game instances running on your local machine. However, in the “real world” games will always have 10-70ms minimum latency, and without movement prediction logic, server authorative movement will have rubberbanding. Here is a video overview of the issue:

You NEED this if you want to make a multiplayer game in unreal - YouTube

There are a few strategies to fix this, depending on the game requirements. One method is to use client-authoratitive movement, which doesn’t create the rubberbanding of client->server->client prediction conflicts. However, doing this makes it possible for clients to “cheat” movement.

Some users have had success with an (expensive) marketplace compoenent called GMC which claims to make it easy (or easier) to handle movement prediction for blueprint authored custom actions. (disclaimer: I have no experience with this component, and no ties to it’s author)

GMCv2 - Advanced Locomotion & Network Prediction Framework in Code Plugins - UE Marketplace

Whenever this topic of large enemy AI count is discussed, the idea of deterministic simulation tends to come up. This is a strategy where one sends initial positions for actors, with a psudorandom seed, and all clients, because they are running the same code, can simulate all actors deterministically without network traffic. Only action commands are sent. Starcraft 2 is an example of a game using this strategy, which is more formally called Deterministic State Simulation, or Lockstep Detemrinistic State Simulation.

Unreal Engine does not support deterministic simulation. For most of it’s life, Unreal Engine did not support fixed-timestep, a necessary element to providing determinism. (There is a forum discussion about why UE4 didn’t support fixed physics timesteps that may be helpful.) We’ve been told that one of the goals of UE5 Chaos physics is to support networked physics and rollback, and there have been some dribbles of progress on this, but I’m not sure the current state.

The game Stormgate uses UE for rendering, and deterministic networking, but they did it by implementing their own completely custom physics and networking engine that replaces UE physics and networking.

Using the default UE physics and networking, for larger networked actor count, one is shooting for a hybrid of Significance weighted server->client update rates, and some client-side prediction/simulation to fill in the gaps.

For anyone who got this far, these videos may also be helpful and/or entertaining, and are related to the topic of larger actor counts in Unreal rendering and networking:

Simulating Large Crowds In Niagara | Unreal Engine - YouTube

Optimizing UE4 for Fortnite: Battle Royale - Part 1 | GDC 2018 | Unreal Engine - YouTube

Large Numbers of Entities with Mass in UE5 | Feature Highlight | State of Unreal 2022 - YouTube

Also interesting is this Epic Mega Grants backed Masters Thesis investigating what “next generation multiplayer games” might look like. It covers the networking topics here and more.

Next Gen Networked Games - Games as a medium (Episode 1) - YouTube

Next Gen Networked Games - Networking Physics / Lockstep VS State sync (Episode 2) - YouTube