We are currently in an extremely frustrating situation: we’ve been developing a multiplayer FPS game using Unreal Engine 4 and plan to release it within the coming months, but a serious issue has prevented us from moving foreward.
The problem is that the game server will ALWAYS crash when attacked by DDoS/invalid packets.
We get stuff like this:
This is caused by too many invalid packets being sent to the game server, causing the process to crash and has nothing to do with the actual machine staying online.
Please note that this issue also affects Unreal Tournament 4 servers, so it’s a “blanket” issue. Anyone can keep a server down by sending bad packets. In spite of this, nobody seems to care. Even official Epic Games UT servers go down in a couple of seconds.
Packets would need to be specifically filtered on 7777 and 7778 UDP in order to block garbage. The problem is that hosting companies don’t seem to offer such support and only cover major titles such as Counter Strike and Minecraft.
Do you know of any hosting service which supports UDP packet filtering on request? What can we do?
This is truly a horrible situation for us.
Hell, its even possible for UE4 clients to accidentally DDOS a game server if the game server crashes and restarts. Once the server comes back online, the clients send a whole lot of garbage data to the server but never time out because technically they are no longer timing out.
If you get the ip and port of a server for …other… UE4 driven games server you are connected to with something like netstat, you can also interfere with them a bit.
Usually the company that runs your game server will handle DDoS and other cybersecurity threats (quite possibly at an additional charge). If you’re running your own server, then it’s up to you to find a method (hardware based firewall) to negate these types of attacks.
Yes, we’re talking UE4 layer. The game server process cannot handle invalid packets. In fact no hosting company was able to help us.
This is a dead end for us. We’re scrapping 3 years of work because of this.
The issues does indeed affect UT and Epic in general, but only theoretically. People don’t tend to DDoS free games.
This is a whole different story if you plan to actually SELL a multiplayer game and don’t have thousands of dollars to fund 100+ servers.
Why would a hosting company be able to help you fix UE4? I understand you’re frustrated, but you need to focus on what’s wrong and fix it or wait for Epic to fix it.
UE4 clearly catches the overflow so it shouldn’t be a huge problem to find the culprit. You confusing this with a DDoS attack (and speculating on free games vs paid ones) doesn’t help. Realize what it is and try to fix it. Or rest assured someone else will before you release your game; there are a lot of talented programmers in this community and someone will eventually fix it if Epic doesn’t. I know I will if it’s not fixed by the time I release anything to the public.
Please note that this issue also affects Unreal Tournament 4 servers, so it’s a “blanket” issue. Anyone can keep a server down by sending bad packets. In spite of this, nobody seems to care. Even official Epic Games UT servers go down in a couple of seconds.
[/]
Zeb89, Have you got a thread / post reference on UT forums of this happening?
Bad packets crash/block the server process. Both UT4 and our game server seem unable to handle/discard these bad packets. Anyone sending bad packets can thus crash/block a server.
If we publish our game in spite of these issue our servers will go down on day one and every single person who spent good money on the product would issue a refund request on Steam. This would inevitably boild down to disaster.
There seems to be no way of filtering bad packets from the “good” ones, or at least nobody has come up with a solution as of yet. Various hosting services cannot do anything since the issue is software related (correct me if I’m wrong).
Once a kid purchases a 3€ botnet, he can then proceed to attack port 7777 UDP with garbage packets and send any server he wishes to hell in a handbasket.
There is squat we can do about it as of now.
If you’d like to try this out for yourselves, just host an UT4 server and attempt a stress test with the cheapest of IP stressers.
Tests have shown us that even Epic Games UT4 hubs & servers go down with so much as a fart.
I am not a networking expert by any stretch of the imagination, but the situation certainly looks dreadful.
[=richardboegli;478403]
Zeb89, Have you got a thread / post reference on UT forums of this happening?
Thanks.
[/]
We haven’t reported this on the UT4 forums because it concerns the whole networking of the engine and not just UT4.
Bad packets crash/block the server process. Both UT4 and our game server seem unable to handle/discard these bad packets. Anyone sending bad packets can thus crash/block a server.
If we publish our game in spite of these issue our servers will go down on day one and every single person who spent good money on the product would issue a refund request on Steam. This would inevitably boild down to disaster.
There seems to be no way of filtering bad packets from the “good” ones, or at least nobody has come up with a solution as of yet. Various hosting services cannot do anything since the issue is software related (correct me if I’m wrong).
Once a kid purchases a 3€ botnet, he can then proceed to attack port 7777 UDP with garbage packets and send any server he wishes to hell in a handbasket.
There is squat we can do about it as of now.
If you’d like to try this out for yourselves, just host an UT4 server and attempt a stress test with the cheapest of IP stressers.
Tests have shown us that even Epic Games UT4 hubs & servers go down with so much as a fart.
I am not a networking expert by any stretch of the imagination, but the situation certainly looks dreadful.
We haven’t reported this on the UT4 forums because it concerns the whole networking of the engine and not just UT4.
[/]
Zeb89 thanks for the clarification.
Something that just came to mind is that I am wondering now if Street Fighter V is also suffering for the same issues? Being an extremely high profile game it wouldn’t surprise me if people were DDOSing it for fun.
Why don’t you go into that c++ function FbitReader::SetOverflow() call - at the point where to throws the LogMessage and add the IP if the incoming packet to the firewall…
[=;478795]
Why don’t you go into that c++ function FbitReader::SetOverflow() call - at the point where to throws the LogMessage and add the IP if the incoming packet to the firewall…
That would be as useful as an armelss drummer since botnets attack from hundreds of different IP addresses. We’d need something a little broader which is able to discard foul packets regardless of the IP.
[=richardboegli;478741]
I am wondering now if Street Fighter V is also suffering for the same issues? Being an extremely high profile game it wouldn’t surprise me if people were DDOSing it for fun.
[/]
Wouldn’t surprise me in the least. This needs to be addressed ASAP, for all developers concerned.
If your business plan is not to release dedicated servers to the public, people can keep your service down for as long as they please.
Thank you for your posts. We are aware of the current situation and are investigating solutions to resolve this. If you have any callstacks or reproduction steps that we can follow to cause these crashes to occur, please post them here.
Thank you for your posts. We are aware of the current situation and are investigating solutions to resolve this. If you have any callstacks or reproduction steps that we can follow to cause these crashes to occur, please post them here.
[/]
Thank you for looking into this. In order to reproduce the issue you need to:
Host an unreal Tournament 4 dedicated server or any other UE4 game.
Perform a stress test using a botnet/ip stresser service.
The server will lock up/crash.
You probably need to check the headers of incoming packets. Just a guess.
[= ;479657]
Thank you Zeb69, what specific fuzzer are you currently using during your repro steps?
[/]
I don’t know specifically because we have a guy who does it for us. There are various IP stresser services available which cost close to nothing,
but if you prefer not to get involved in DDoS and IP stressing, add me on Skype and we can arrange a test. Alternatively, you can provide me with the IP of the UT4 server you’d like us to send the stress test to.