We’re getting a GC crash where there’s an index out of range happening when the client exits the game and the world tears down. Has anyone seen anything like this before or have any experience on how this is meant to be debugged? Looking at the code, I don’t think it’s a programming error. The indices are set with what appear to be correct lengths so perhaps there’s a concurrency issue or something.
Can you attach the call stack? It might help narrow this down, especially knowing which array it’s failing with and what variable contains that index, so we know an expected range for it to be in.
The index is really high, I have a suspicion that whatever variable holds that index might’ve gotten memory stomped.
Until you send the call stack, I’m giving you some GC memory stomp crash tips, which might help until we know more:
The most common reason for crashes like this is when the game has a pointer to a UObject-based class but doesn’t decorate it with the UPROPERTY macro. That means that when the GC runs, the pointer won’t be nulled, and it will be pointing at trash data. Then UE fills that memory with something else, and if you use that stale pointer, then you’re basically writing invalid data into another entirely different object, which can span many of its properties, resulting in a “memory stomp.” For example, overwriting the index of an FName with unrelated data so that it points outside the FName global table.
The Garbage Collector is usually what makes data invalid. It’s the only system that’s allowed to destroy UObjects, after which the memory is reused for new objects, which are then the objects whose data gets overwritten by the stale pointers.
The crash typically occurs later, after the actual memory stomp has occurred, when code dereferences pointers for classes for which their memory is no longer valid. So, trying to catch the crash in the debugger to understand what happened is often futile.
Instead, these should be attempted to fix or find the source of the memory stomp:
Go through the game’s code, and make sure that any class member that references a UObject-based class (that includes AActors) has a UPROPERTY decorator. Make sure that FStruct member variables that contain UObject class members are also marked as UPROPERTY. And yes, also for TArray<*>, and TObjectPtr. All need to have a UPROPERTY, up the entire chain. If even one of them is missed, you’ll likely get hard-to-debug crashes.
If you do that and it’s still crashing, you can make each run a bit less random. The crash likely occurs only after the garbage collector has run at least once, as that’s what triggers this behavior. You can make the run more deterministic by making the garbage collector run every frame with the “gc.CollectGarbageEveryFrame 1” console command. You can also start the process with -ExecCmds=“gc.CollectGarbageEveryFrame=1”. This will ensure that the offending code is usually running in the same or previous frame from when the crash occurs, making it much easier to determine the cause. In the example above, they had an asynchronous save game system running concurrently with the garbage collector.
It might be other threads doing something at the time of GC collection. If you can’t find the source using the first two steps, try running with the GC every frame command while a CPU sampling profiler is also connected. This allows you to see which threads were running at the time the crash happens. Then you can audit what those threads are doing.