Inexplicable crash and bad performance over network

Development Programming & Scripting Multiplayer & Networking
, , ,
1

Hi! I recently tested my project over network (I hadn’t tested it over network in a little while, but not done anything that would interfere with clientside stuff) and suddenly found some serious problems.

First, there’s extreme jittery-ness with physics objects as they are moved about that wasn’t there before. It may be a problem in 4.9, since I didn’t do a lot of network testing in 4.9 yet, this problem certainly wasn’t happening in 4.8 but I’m not sure if it’s version or something else.

On top of that, my game crashes instantly when I try to perform certain actions that didn’t have problems before (namely, “chopping in half” my log object and having it spawn a new one) and gives this error message:

Unknown exception - code 00000001 (first/second chance not available)

"Assertion failed: (Index >= 0) & (Index < ArrayNum) [File:D:\BuildFarm\buildmachine_++depot+UE4-Releases+4.9\Engine\Source\Runtime\Core\Public\Containers\Array.h] [Line: 678]
Array index out

UE4Editor_Core!FDebug::AssertFailed() [d:\buildfarm\buildmachine_++depot+ue4-releases+4.9\engine\source\runtime\core\private\misc\outputdevice.cpp:354]
UE4Editor_Engine!USceneComponent::OnComponentDestroyed() [d:\buildfarm\buildmachine_++depot+ue4-releases+4.9\engine\source\runtime\engine\private\components\scenecomponent.cpp:664]
UE4Editor_Engine!UActorComponent::DestroyComponent() [d:\buildfarm\buildmachine_++depot+ue4-releases+4.9\engine\source\runtime\engine\private\components\actorcomponent.cpp:944]
UE4Editor_Engine!UActorChannel::ReadContentBlockHeader() [d:\buildfarm\buildmachine_++depot+ue4-releases+4.9\engine\source\runtime\engine\private\datachannel.cpp:2551]
UE4Editor_Engine!UActorChannel::ProcessBunch() [d:\buildfarm\buildmachine_++depot+ue4-releases+4.9\engine\source\runtime\engine\private\datachannel.cpp:1967]
UE4Editor_Engine!UActorChannel::ReceivedBunch() [d:\buildfarm\buildmachine_++depot+ue4-releases+4.9\engine\source\runtime\engine\private\datachannel.cpp:1897]
UE4Editor_Engine!UChannel::ReceivedSequencedBunch() [d:\buildfarm\buildmachine_++depot+ue4-releases+4.9\engine\source\runtime\engine\private\datachannel.cpp:270]
UE4Editor_Engine!UChannel::ReceivedNextBunch() [d:\buildfarm\buildmachine_++depot+ue4-releases+4.9\engine\source\runtime\engine\private\datachannel.cpp:614]
UE4Editor_Engine!UChannel::ReceivedRawBunch() [d:\buildfarm\buildmachine_++depot+ue4-releases+4.9\engine\source\runtime\engine\private\datachannel.cpp:354]
UE4Editor_Engine!UNetConnection::ReceivedPacket() [d:\buildfarm\buildmachine_++depot+ue4-releases+4.9\engine\source\runtime\engine\private\netconnection.cpp:1044]
UE4Editor_Engine!UNetConnection::ReceivedRawPacket() [d:\buildfarm\buildmachine_++depot+ue4-releases+4.9\engine\source\runtime\engine\private\netconnection.cpp:495]
UE4Editor_OnlineSubsystemUtils!UIpNetDriver::TickDispatch() [d:\buildfarm\buildmachine_++depot+ue4-releases+4.9\engine\source\runtime\online\onlinesubsystemutils\private\ipnetdriver.cpp:174]
UE4Editor_Engine!TBaseUObjectMethodDelegateInstance<0,UNetDriver,void cdecl(float)>::ExecuteIfSafe() [d:\buildfarm\buildmachine++depot+ue4-releases+4.9\engine\source\runtime\core\public\delegates\delegateinstancesimpl_variadics.inl:772]
UE4Editor_Engine!TBaseMulticastDelegate::Broadcast() [d:\buildfarm\buildmachine++depot+ue4-releases+4.9\engine\source\runtime\core\public\delegates\delegatesignatureimpl_variadics.inl:809]
UE4Editor_Engine!UWorld::Tick() [d:\buildfarm\buildmachine_++depot+ue4-releases+4.9\engine\source\runtime\engine\private\leveltick.cpp:1062]
UE4Editor_UnrealEd!UEditorEngine::Tick() [d:\buildfarm\buildmachine_++depot+ue4-releases+4.9\engine\source\editor\unrealed\private\editorengine.cpp:1347]
UE4Editor_UnrealEd!UUnrealEdEngine::Tick() [d:\buildfarm\buildmachine_++depot+ue4-releases+4.9\engine\source\editor\unrealed\private\unrealedengine.cpp:366]
UE4Editor!FEngineLoop::Tick() [d:\buildfarm\buildmachine_++depot+ue4-releases+4.9\engine\source\runtime\launch\private\launchengineloop.cpp:2428]
UE4Editor!GuardedMain() [d:\buildfarm\buildmachine_++depot+ue4-releases+4.9\engine\source\runtime\launch\private\launch.cpp:142]
UE4Editor!GuardedMainWrapper() [d:\buildfarm\buildmachine_++depot+ue4-releases+4.9\engine\source\runtime\launch\private\windows\launchwindows.cpp:126]
UE4Editor!WinMain() [d:\buildfarm\buildmachine_++depot+ue4-releases+4.9\engine\source\runtime\launch\private\windows\launchwindows.cpp:200]

There’s a lot of weirdness to this. First off, folder that it references (D:\BuildFarm) I can’t find at all on my drive, in fact my D drive’s UE4 folder is totally empty (I have my stuff on my F drive, though at one point something was on D)

other strange thing is I looked up crash and only found two other posts with this crash.

Both of which were all about lighting issues causing crashes. However, top one for example has exact same assertion failed statement, same folder, same line (678), as my crash. But my crash has nothing to do with lighting, as far as I know - my world has a day/night cycle so I don’t use static lighting and I never build lighting. And it’s really weird because it only occurs in multiplayer or when I use dedicated server option, otherwise it works fine. proposed solutions that seemed to have worked for other posters involved clearing SwarmAgent cache, but I don’t know how to do that - I followed same path as what they showed on A new, community-hosted Unreal Engine Wiki - Announcements - Epic Developer Community Forums page, but cache file just isn’t there. I’m still doubtful that it’s even issue at hand, but I would like to be able to clear cache just in case.

I’m going to sleep now so I’ll do some further investigation of issue tomorrow, but this is all a little bizarre, considering it’s not just crash that’s problem but also jittery laggy physics objects and possibly more issues I haven’t tested yet. If anyone knows how to clear cache or has advice as to what might be able to stop this from happening, that would be very helpful.

2

Hi DylanBrady,

I don’t believe you’re facing same crash as two links you posted, nor anything with swarm agent or lighting build. It looks like this may be happening when a component is destroyed. I don’t remember: is your log cutting BP destroying any components? Can you show me an image of your component destruction setup in your BP?

It may be helpful to get your Machine ID from Crash Reporter log next time you reproduce crash, so we can look it up in our system if you hit Send.

I haven’t seen any jittery physics problems in 4.9 between networked packaged games so far. Can you reproduce that in a new test project? If so, can you upload it somewhere for me? Thanks!

3

Hi ! Thanks for helping me out again!

First off, here’s Machine ID from my most recent crash:
MachineId:343DEFF54E9DEA2DD8FA8DA83AFCA19D

As far as component destruction, it looks like you’re correct on that one. I tried destroying a log part with a different function, and it crashed. I tried taking out part where it destroys a log part after chop, and it didn’t crash.

There’s a bit more to it than that, though. My logs are made up of two sets of components - a set of primitives, for static meshes, and a set of scene components corresponding to those primitives to control blueprints and variables relating to them. When I do a chop, I destroy both primitive and it’s corresponding scene component.

So for sake of testing, I tried seeing what happens if I destroy one but not other. If I destroy primitive component but not scene component, game does not crash. It seems to have a big lag spike on first chop, that didn’t happen in past, but it doesn’t crash immediately and I am able to continue testing and chopping more parts without any more lag spikes or crashing during that test. However, if I chopped even just one component, as soon as I finish test and exit Play-In-Editor it crashes (same crash message as before - it even references !USceneComponent::OnComponentDestroyed() still, despite fact that I destroyed a primitive, not a scene component)

Meanwhile, if I destroy scene component but not primitive component, it doesn’t seem to have any issues as far as crashing or lag spikes, and I’m able to finish test without a crash popping up.

chopping function is large and convoluted, but destroy part is fairly simple.

http://puu.sh/k5VEd/dfd7e9043a.jpg

http://puu.sh/k5VGF/ec33f74105.png

‘Temp Log Actor’ is a reference to original actor that owns components, so that it can tell that actor to destroy components.

4

So I tried recreating jitter problem in a test project but no luck. I may try again later but as of now I couldn’t reproduce it.

It seems to occur on my logs, which are made up of many components, but not on other objects that are made up of just one component.

I checked out 4.8 version of my project, and it doesn’t have problem. To be sure, I created a 4.9 copy of my old 4.8 version, so that it wouldn’t have any of changes I’ve made since 4.9 came out, and problem existed. So from what I can tell it is most definitely a 4.9-specific problem.

I downloaded a recording program and made some videos to show comparison between local/network physics in 4.8 and 4.9 for my logs.

I just uploaded video to , this should open it up https://www…com/watch?v=5PQqMF0XhKc

I hope that can help. You can see at end (for a little bit) that tree that is falling has regular physics, but once it turns into a log it jitters again.

5

Hm, I’m not clear on what you mean by two sets of components. So you have a Scene component and a StaticMesh component childed to it? And destroying both at same time causes crash, but not one or other?

I’m not getting any kind of crash by destroying components. Are you able to reproduce this in a new project as well?

6

I’m not able to right now, but later tonight I’ll try again to see if I can reproduce either or both of issues I’m experiencing.

To clarify, I have a set of StaticMesh components that are children of my root StaticMesh component, and each of static mesh components (including root) also have a Scene component child. I use scene components to record variables and information about that individual log ‘part’, like health and so on. Whenever I create or destroy log parts, I add StaticMesh and then Scene component and attach Scene to static.

I get crash everytime when I call Destroy on both staticmesh and it’s child scene component. During my tests I tried destroying one but not other, and got more inconsistent behavior. After I tested several times it seemed that most consistent thing that was occuring was that there was no crash if I destroyed just Scene component, but if I destroyed StaticMesh component it would have some odd behavior and then crash at end of test, and if I destroyed both it would crash immediately.

Once again, I’m sorry it’s all convoluted and I’m having a hard time nailing problem down. I’ve had to jump through a lot of hoops to get logs to work exact way I want them, which makes it hard to control for these issues because there are so many different things going on and trying to reproduce whole process is a little complex. Especially because problem is occurring over network this time, unlike previous problem, so I can’t click through breakpoints and so on to see exact point crash occurs and such. I’ll try to reproduce it from ground up, but if that doesn’t work I’ll maybe try just copying over how it currently works to a new project (minus some extra irrelevant stuff) and seeing if I can get it from there.

7

I managed to reproduce jittery physics issue, but not crash, in a test project. Once I got it working I started stripping down pieces and found that it’s really easy - just have an object, create it with a bunch of components attached, and tell it to simulate physics. It’s that simple.

To make sure it’s a 4.9 specific problem, I recreated it in 4.8. I have two downloads here, one for 4.8 version and one for 4.9 version. To make sure they were totally same, 4.9 version is same project as 4.8 one, just a 4.9 copy.

There’s a variable called LogLength which controls how many components are created for log. At a higher value, there is a larger amount of jitter.

Here’s 4.9 download: TestProject3 4.9

And 4.8 download for comparison: TestProject3 4.8

I hope that helps! I have another test project where I’ve been attempting to reproduce crash, but so far no luck on that one.

8

That’s fine! Let me know if you’re able to reproduce in a new project. If you can upload it someplace like Google Drive or Dropbox again, I can take a look. If you’re only able to reproduce it by copying over what you already have, that’s less ideal but I can work with it.

9

I’ve got a test project now that I’ve managed to recreate crash-on-chop issue in. It also shows other error, with jittery physics. Both problems only exist when playing over network.

To reproduce issue, play with 2 players (Dedicated Server 1-player will also experience issue, but it doesn’t seem to work right with test function I set up for sending out a trace). Press F while facing a portion of log. In 1-player, it will chop. In 2-player, it will crash immediately. You will also see jittery physics as log falls to ground as long as you’re playing over dedicated server or with multiple players (unless you’re watching through server viewport).

This test project is a copied version of my project, which I then cleaned out of everything that wasn’t directly involved here. I hope project works for you, and I hope this helps!

10

Hi , sorry to be adding extra bother, but I wanted to make sure you got a notification or something to see that I’ve put up test projects on these issues?

I just wanted to make sure downloading them worked and all that and that projects were able to be used to produce issues I’ve been describing. If not, I can try again

11

No bother. I did get project, and I am seeing same crash you’ve described. I am in process of narrowing down cause so we can get a bug report entered properly, and I’ll let you know once I’ve done so. Thanks for your patience!

12

Hey DylanBrady,

Sorry for delay. I wasn’t able to narrow down cause of crash very much beyond Component destruction, so I’ve included your project with a bug report for developers to look at (UE-21855). I’ll let you know if I see any update on it, but finding an alternate solution to destroying those components is your best bet.

reason you see so much lag on client is because actor is fairly network heavy, due probably to amount and complexity of components created with your setup. If you take a look at our Network Profiler tools and use it in your project, you’ll see a drain coming from this actor.