We are experiencing constant crashes when trying to run our subclass of WorldPartitionBuilder as a commandlet on a large open world. The commandlet is just loading cells (via the existing WorldPartitionBuilder framework) and gathering data. It is not making or saving any changes to the world.
It crashes on every execution while serializing, and always in StructuredArchive.cpp or StructuredArchiveSlots.cpp.
It appears to crash on a different actor/component each time.
It always crashes on some form of mesh component, e.g HISM, FISM, landscape deformer mesh, static mesh.
Sometimes the crash is near immediate, sometimes it is hours in.
The only time it has successfully run was when 95% of the data layers in our world were set to unloaded.
Notably, this commandlet runs fine on considerably smaller/emptier maps.
We have tried running with -NoAsyncPostLoad.
Have you seen anything like this in serialization code before? Any potential fixes or workarounds / tips for debugging this further?
We haven’t seen such issues internally. Is the commandlet running the GC regularly or it’s relying on the framework to run it? You could try to run the GC after each cell to help determine if you have stale asset references. Inspecting the parallel stacks could also be useful to validate this is not a concurrence problem.
When we run an empty WorldPartitionBuilder subclass which just prints a string each time it loads a cell + does garbage collection we are getting the same kind of crashes.
We are also seeing these crashes trying to run the default WorldPartitionMiniMapBuilder, which as far as I can see doesn’t do any of its own GC.
This is the way we are calling garbage collection in RunInternal()
The problem doesn’t seem linked to the builder itself but to memory corruption. Do you have custom code that has “raw” C++ pointers on UObject based types? This type of weird corruption can often be linked to missing UPROPERTY tags in class definitions. If you have non-UObject code that is referencing UObjects, you either have to use Strong\Weak reference pointers (for reference: https://dev.epicgames.com/documentation/en\-us/unreal\-engine/object\-pointers\-in\-unreal\-engine?application\_version\=5\.6\)
We looked for raw pointers and couldn’t find anything. However we swapped out the hardware RAM and it solved the issue. It looks like it was memory corruption of a different kind!