Inexplicable crash and bad performance over network

Development Programming & Scripting Multiplayer & Networking
, , ,
1

Hi! I recently tested my project over the network (I hadn’t tested it over network in a little while, but not done anything that would interfere with clientside stuff) and suddenly found some serious problems.

First, there’s extreme jittery-ness with physics objects as they are moved about that wasn’t there before. It may be a problem in 4.9, since I didn’t do a lot of network testing in 4.9 yet, this problem certainly wasn’t happening in 4.8 but I’m not sure if it’s the version or something else.

On top of that, my game crashes instantly when I try to perform certain actions that didn’t have problems before (namely, “chopping in half” my log object and having it spawn a new one) and gives this error message:

Unknown exception - code 00000001 (first/second chance not available)

"Assertion failed: (Index >= 0) & (Index < ArrayNum) [File:D:\BuildFarm\buildmachine_++depot+UE4-Releases+4.9\Engine\Source\Runtime\Core\Public\Containers\Array.h] [Line: 678]
Array index out

UE4Editor_Core!FDebug::AssertFailed() [d:\buildfarm\buildmachine_++depot+ue4-releases+4.9\engine\source\runtime\core\private\misc\outputdevice.cpp:354]
UE4Editor_Engine!USceneComponent::OnComponentDestroyed() [d:\buildfarm\buildmachine_++depot+ue4-releases+4.9\engine\source\runtime\engine\private\components\scenecomponent.cpp:664]
UE4Editor_Engine!UActorComponent::DestroyComponent() [d:\buildfarm\buildmachine_++depot+ue4-releases+4.9\engine\source\runtime\engine\private\components\actorcomponent.cpp:944]
UE4Editor_Engine!UActorChannel::ReadContentBlockHeader() [d:\buildfarm\buildmachine_++depot+ue4-releases+4.9\engine\source\runtime\engine\private\datachannel.cpp:2551]
UE4Editor_Engine!UActorChannel::ProcessBunch() [d:\buildfarm\buildmachine_++depot+ue4-releases+4.9\engine\source\runtime\engine\private\datachannel.cpp:1967]
UE4Editor_Engine!UActorChannel::ReceivedBunch() [d:\buildfarm\buildmachine_++depot+ue4-releases+4.9\engine\source\runtime\engine\private\datachannel.cpp:1897]
UE4Editor_Engine!UChannel::ReceivedSequencedBunch() [d:\buildfarm\buildmachine_++depot+ue4-releases+4.9\engine\source\runtime\engine\private\datachannel.cpp:270]
UE4Editor_Engine!UChannel::ReceivedNextBunch() [d:\buildfarm\buildmachine_++depot+ue4-releases+4.9\engine\source\runtime\engine\private\datachannel.cpp:614]
UE4Editor_Engine!UChannel::ReceivedRawBunch() [d:\buildfarm\buildmachine_++depot+ue4-releases+4.9\engine\source\runtime\engine\private\datachannel.cpp:354]
UE4Editor_Engine!UNetConnection::ReceivedPacket() [d:\buildfarm\buildmachine_++depot+ue4-releases+4.9\engine\source\runtime\engine\private\netconnection.cpp:1044]
UE4Editor_Engine!UNetConnection::ReceivedRawPacket() [d:\buildfarm\buildmachine_++depot+ue4-releases+4.9\engine\source\runtime\engine\private\netconnection.cpp:495]
UE4Editor_OnlineSubsystemUtils!UIpNetDriver::TickDispatch() [d:\buildfarm\buildmachine_++depot+ue4-releases+4.9\engine\source\runtime\online\onlinesubsystemutils\private\ipnetdriver.cpp:174]
UE4Editor_Engine!TBaseUObjectMethodDelegateInstance<0,UNetDriver,void cdecl(float)>::ExecuteIfSafe() [d:\buildfarm\buildmachine++depot+ue4-releases+4.9\engine\source\runtime\core\public\delegates\delegateinstancesimpl_variadics.inl:772]
UE4Editor_Engine!TBaseMulticastDelegate::Broadcast() [d:\buildfarm\buildmachine++depot+ue4-releases+4.9\engine\source\runtime\core\public\delegates\delegatesignatureimpl_variadics.inl:809]
UE4Editor_Engine!UWorld::Tick() [d:\buildfarm\buildmachine_++depot+ue4-releases+4.9\engine\source\runtime\engine\private\leveltick.cpp:1062]
UE4Editor_UnrealEd!UEditorEngine::Tick() [d:\buildfarm\buildmachine_++depot+ue4-releases+4.9\engine\source\editor\unrealed\private\editorengine.cpp:1347]
UE4Editor_UnrealEd!UUnrealEdEngine::Tick() [d:\buildfarm\buildmachine_++depot+ue4-releases+4.9\engine\source\editor\unrealed\private\unrealedengine.cpp:366]
UE4Editor!FEngineLoop::Tick() [d:\buildfarm\buildmachine_++depot+ue4-releases+4.9\engine\source\runtime\launch\private\launchengineloop.cpp:2428]
UE4Editor!GuardedMain() [d:\buildfarm\buildmachine_++depot+ue4-releases+4.9\engine\source\runtime\launch\private\launch.cpp:142]
UE4Editor!GuardedMainWrapper() [d:\buildfarm\buildmachine_++depot+ue4-releases+4.9\engine\source\runtime\launch\private\windows\launchwindows.cpp:126]
UE4Editor!WinMain() [d:\buildfarm\buildmachine_++depot+ue4-releases+4.9\engine\source\runtime\launch\private\windows\launchwindows.cpp:200]

There’s a lot of weirdness to this. First off, the folder that it references (D:\BuildFarm) I can’t find at all on my drive, in fact my D drive’s UE4 folder is totally empty (I have my stuff on my F drive, though at one point something was on D)

The other strange thing is I looked up the crash and only found two other posts with this crash.

Both of which were all about lighting issues causing crashes. However, the top one for example has the exact same assertion failed statement, same folder, same line (678), as my crash. But my crash has nothing to do with lighting, as far as I know - my world has a day/night cycle so I don’t use static lighting and I never build lighting. And it’s really weird because it only occurs in multiplayer or when I use the dedicated server option, otherwise it works fine. The proposed solutions that seemed to have worked for the other posters involved clearing the SwarmAgent cache, but I don’t know how to do that - I followed the same path as what they showed on the A new, community-hosted Unreal Engine Wiki - Announcements - Unreal Engine Forums page, but the cache file just isn’t there. I’m still doubtful that it’s even the issue at hand, but I would like to be able to clear the cache just in case.

I’m going to sleep now so I’ll do some further investigation of the issue tomorrow, but this is all a little bizarre, considering it’s not just the crash that’s the problem but also the jittery laggy physics objects and possibly more issues I haven’t tested yet. If anyone knows how to clear the cache or has advice as to what might be able to stop this from happening, that would be very helpful.

2

Hi DylanBrady,

I don’t believe you’re facing the same crash as the two links you posted, nor anything with the swarm agent or lighting build. It looks like this may be happening when a component is destroyed. I don’t remember: is your log cutting BP destroying any components? Can you show me an image of your component destruction setup in your BP?

It may be helpful to get your Machine ID from the Crash Reporter log next time you reproduce the crash, so we can look it up in our system if you hit Send.

I haven’t seen any jittery physics problems in 4.9 between networked packaged games so far. Can you reproduce that in a new test project? If so, can you upload it somewhere for me? Thanks!

3

Hi Ben! Thanks for helping me out again!

First off, here’s the Machine ID from my most recent crash:
MachineId:343DEFF54E9DEA2DD8FA8DA83AFCA19D

As far as component destruction, it looks like you’re correct on that one. I tried destroying a log part with a different function, and it crashed. I tried taking out the part where it destroys a log part after the chop, and it didn’t crash.

There’s a bit more to it than that, though. My logs are made up of two sets of components - a set of primitives, for the static meshes, and a set of scene components corresponding to those primitives to control blueprints and variables relating to them. When I do a chop, I destroy both the primitive and it’s corresponding scene component.

So for the sake of testing, I tried seeing what happens if I destroy one but not the other. If I destroy the primitive component but not the scene component, the game does not crash. It seems to have a big lag spike on the first chop, that didn’t happen in the past, but it doesn’t crash immediately and I am able to continue testing and chopping more parts without any more lag spikes or crashing during that test. However, if I chopped even just one component, as soon as I finish the test and exit Play-In-Editor it crashes (same crash message as before - it even references !USceneComponent::OnComponentDestroyed() still, despite the fact that I destroyed a primitive, not a scene component)

Meanwhile, if I destroy the scene component but not the primitive component, it doesn’t seem to have any issues as far as crashing or lag spikes, and I’m able to finish the test without a crash popping up.

The chopping function is large and convoluted, but the destroy part is fairly simple.

http://puu.sh/k5VEd/dfd7e9043a.jpg

http://puu.sh/k5VGF/ec33f74105.png

‘Temp Log Actor’ is a reference to the original actor that owns the components, so that it can tell that actor to destroy the components.

4

So I tried recreating the jitter problem in a test project but no luck. I may try again later but as of now I couldn’t reproduce it.

It seems to occur on my logs, which are made up of many components, but not on other objects that are made up of just one component.

I checked out the 4.8 version of my project, and it doesn’t have the problem. To be sure, I created a 4.9 copy of my old 4.8 version, so that it wouldn’t have any of the changes I’ve made since 4.9 came out, and the problem existed. So from what I can tell it is most definitely a 4.9-specific problem.

I downloaded a recording program and made some videos to show the comparison between local/network physics in 4.8 and 4.9 for my logs.

I just uploaded the video to YouTube, this should open it up - YouTube

I hope that can help. You can see at the end (for a little bit) that the tree that is falling has regular physics, but once it turns into a log it jitters again.

5

Hm, I’m not clear on what you mean by two sets of components. So you have a Scene component and a StaticMesh component childed to it? And destroying both at the same time causes the crash, but not one or the other?

I’m not getting any kind of crash by destroying components. Are you able to reproduce this in a new project as well?

6

I’m not able to right now, but later tonight I’ll try again to see if I can reproduce either or both of the issues I’m experiencing.

To clarify, I have a set of StaticMesh components that are children of my root StaticMesh component, and each of the static mesh components (including the root) also have a Scene component child. I use the scene components to record variables and information about that individual log ‘part’, like health and so on. Whenever I create or destroy the log parts, I add the StaticMesh and then the Scene component and attach the Scene to the static.

I get the crash everytime when I call Destroy on both the staticmesh and it’s child scene component. During my tests I tried destroying one but not the other, and got more inconsistent behavior. After I tested several times it seemed that the most consistent thing that was occuring was that there was no crash if I destroyed just the Scene component, but if I destroyed the StaticMesh component it would have some odd behavior and then crash at the end of the test, and if I destroyed both it would crash immediately.

Once again, I’m sorry it’s all convoluted and I’m having a hard time nailing the problem down. I’ve had to jump through a lot of hoops to get the logs to work the exact way I want them, which makes it hard to control for these issues because there are so many different things going on and trying to reproduce the whole process is a little complex. Especially because the problem is occurring over the network this time, unlike the previous problem, so I can’t click through breakpoints and so on to see the exact point the crash occurs and such. I’ll try to reproduce it from the ground up, but if that doesn’t work I’ll maybe try just copying over how it currently works to a new project (minus some extra irrelevant stuff) and seeing if I can get it from there.

7

I managed to reproduce the jittery physics issue, but not the crash, in a test project. Once I got it working I started stripping down the pieces and found that it’s really easy - just have an object, create it with a bunch of components attached, and tell it to simulate physics. It’s that simple.

To make sure it’s a 4.9 specific problem, I recreated it in 4.8. I have two downloads here, one for the 4.8 version and one for the 4.9 version. To make sure they were totally the same, the 4.9 version is the same project as the 4.8 one, just a 4.9 copy.

There’s a variable called LogLength which controls how many components are created for the log. At a higher value, there is a larger amount of jitter.

Here’s the 4.9 download: TestProject3 4.9

And the 4.8 download for comparison: TestProject3 4.8

I hope that helps! I have another test project where I’ve been attempting to reproduce the crash, but so far no luck on that one.

8

That’s fine! Let me know if you’re able to reproduce in a new project. If you can upload it someplace like Google Drive or Dropbox again, I can take a look. If you’re only able to reproduce it by copying over what you already have, that’s less ideal but I can work with it.

9

I’ve got a test project now that I’ve managed to recreate the crash-on-chop issue in. It also shows the other error, with the jittery physics. Both problems only exist when playing over network.

To reproduce the issue, play with 2 players (Dedicated Server 1-player will also experience the issue, but it doesn’t seem to work right with the test function I set up for sending out a trace). Press F while facing a portion of the log. In 1-player, it will chop. In 2-player, it will crash immediately. You will also see jittery physics as the log falls to the ground as long as you’re playing over dedicated server or with multiple players (unless you’re watching through the server viewport).

This test project is a copied version of my project, which I then cleaned out of everything that wasn’t directly involved here. I hope the project works for you, and I hope this helps!

10

Hi Ben, sorry to be adding extra bother, but I wanted to make sure you got a notification or something to see that I’ve put up test projects on these issues?

I just wanted to make sure downloading them worked and all that and that the projects were able to be used to produce the issues I’ve been describing. If not, I can try again

11

No bother. I did get the project, and I am seeing the same crash you’ve described. I am in the process of narrowing down the cause so we can get a bug report entered properly, and I’ll let you know once I’ve done so. Thanks for your patience!

12

Hey DylanBrady,

Sorry for the delay. I wasn’t able to narrow down the cause of the crash very much beyond the Component destruction, so I’ve included your project with a bug report for the developers to look at (UE-21855). I’ll let you know if I see any update on it, but finding an alternate solution to destroying those components is your best bet.

The reason you see so much lag on the client is because the actor is fairly network heavy, due probably to the amount and complexity of components created with your setup. If you take a look at our Network Profiler tools and use it in your project, you’ll see a drain coming from this actor.