Pak files are created (after cooking) using UnrealPak.exe command from within the editor, No Chunking. They are then distributed and loaded at runtime as mods for our game.
[Attachment Removed]
Steps to Reproduce
Have a C++ class that a blueprint inherits from called CharacterData. We are able to cook, pak, and load this class fine within our game.
If we were to make a change to CharacterData like adding a new UPROPERTY then any previous paks no longer load, and get this error during serialization. We are unsure how to not break previously created pak files when updating the game.
LogWindows: Error: appError called: Assertion failed: Index.IsExport() && ExportMap.IsValidIndex(Index.ToExport()) [File:D:\Rivals2Snapnet\Engine\Source\Runtime\CoreUObject\Public\UObject\Linker.h] [Line: 165]
[Attachment Removed]
Hey Joshau,
Unfortunately, this is per design.
Cooked data is serialized as unversioned data that does not support upgrading from older versions, in constrast to the way that migrations in the Editor are handled. Cooked data is assumed to be only compatible with the same Engine version it was built with and won’t be able to deal with migrations or layout changes.
For patch releases (e.g. 5.4.X) we guarantee binary compatibility, but for any major Engine upgrades or changes to your game classes it is quite possible that older data built in a previous version will not load.
If any base classes or other dependencies change their serialized layout it will be necessary to re-cook the mods to stay compatible.
A good practice is to store the engine/game version that a mod was built for in it’s .uplugin file or some separate metadata and then prevent loading it if it is not compatible.
Kind Regards,
Sebastian
[Attachment Removed]
The problem isn’t that we are performing an engine change, the version of the engine will remain the same. The issue is a new build of the game where there is a change to the underlying C++ of a UClass object. As stated in the above response. If I have something like
UCLASS()
class MYGAME_API MyCharacter : public UObject
{
UPROPERTY()
FString Name;
}
Then I were to modify it to something like
UCLASS()
class MYGAME_API MyCharacter : public UObject
{
UPROPERTY()
FString Name;
UPROPERTY()
Int32 Level;
}
Then any .pak files that were made with the first iteration of the class will fail to deserialize at runtime and cause a crash. Though my understanding is that it should be able to convert in place and Level = NULL.
[Attachment Removed]
hmmm so if I’m understanding correctly, the best way to handle previously cooked content remains compatible is to make sure any underlying C++ headers don’t have a layout change (no adding or removing fields/methods)? But if we do need to make changes then there isn’t anything we can do for runtime conversion, we just need to notify the modding community for them to re-cook their mods with an updated version of the project?
[Attachment Removed]
I guess then the only other solution is to roll out our own serialization for these classes to maintain compatibility, but we were really hoping the default serialization would cover the changes.
[Attachment Removed]
> if I’m understanding correctly, the best way to handle previously cooked content remains compatible is to make sure any underlying C++ headers don’t have a layout change (no adding or removing fields/methods)? But if we do need to make changes then there isn’t anything we can do for runtime conversion, we just need to notify the modding community for them to re-cook their mods with an updated version of the project?
In general, with the default cook settings, you’re correct. Cooked projects don’t retain the information needed for graceful handling of layout or type changes by default. As such, a packaged game can’t upgrade old data the way the Editor does.
We’re aware this makes modding and UGC in general more complicated. These choices where in the early days of the Engine with a focus on the best loading performance at runtime.
You can, however, try to cook with versioned content, to use the same serialization as in Editor.
This will affect performance and this path is not exercised much by us or other licensees. I can’t promise that it will actually work, but it might be worth a try to increase your mod compatibility at the risk of using a fairly niche feature that might have other lingering issues.
To enable versioned cooks you’ll need to add the -VersionCookedContent flag to UAT’s BuildCookRun (instead of -UnversionedCookedContent which is the default). Alternatively you can remove the -unversioned flag from the CookCommandlet invocation directly (UAT just hands things through to the commandlet).
You’ll need to compile both the game and the mods with that option.
Kind Regards,
Sebastian
[Attachment Removed]
hmmmm we were cooking the mods with the following command FString Command = TEXT( “../../../Project/Game.uproject -run=cook -targetplatform=Windows -iterate -stdout -unattended -UTF8Output” ); So unfortunately that didn’t really seem to solve our problem. I am hoping that I can just override the FArchive* operator<<() function to allow us to handle the serialization. But in my early tests it doesn’t look like we are going into the function call. Do you know if perhaps SerializeDefaultObject uses a different serializer call? So far I have tried the following functions, but have not hit them at runtime.
virtual void Serialize(FArchive& Ar) override;
friend FArchive& operator <<(FArchive& Ar, UCharacterData& CharData);
[Attachment Removed]
Also for more context into the problem we are bumping up against when loading older pak files, here is our specific call stack
[Inline Frame] Rivals2.exe!FLinkerTables::Exp(FPackageIndex) Line 165
at D:\Rivals2Snapnet\Engine\Source\Runtime\CoreUObject\Public\UObject\Linker.h(165)
[Inline Frame] Rivals2.exe!FLinkerLoad::ResolveResource(FPackageIndex) Line 4286
at D:\Rivals2Snapnet\Engine\Source\Runtime\CoreUObject\Private\UObject\LinkerLoad.cpp(4286)
Rivals2.exe!FLinkerLoad::operator<<(UObject * & Object) Line 6166
at D:\Rivals2Snapnet\Engine\Source\Runtime\CoreUObject\Private\UObject\LinkerLoad.cpp(6166)
Rivals2.exe!FArchiveProxy::operator<<(UObject * & Value) Line 46
at D:\Rivals2Snapnet\Engine\Source\Runtime\Core\Public\Serialization\ArchiveProxy.h(46)
[Inline Frame] Rivals2.exe!FBinaryArchiveFormatter::Serialize(UObject * &) Line 273
at D:\Rivals2Snapnet\Engine\Source\Runtime\Core\Public\Serialization\Formatters\BinaryArchiveFormatter.h(273)
[Inline Frame] Rivals2.exe!FStructuredArchiveSlot::operator<<(UObject * &) Line 349
at D:\Rivals2Snapnet\Engine\Source\Runtime\Core\Public\Serialization\StructuredArchiveSlots.h(349)
Rivals2.exe!FObjectProperty::SerializeItem(FStructuredArchiveSlot Slot, void * Value, const void * Defaults) Line 227
at D:\Rivals2Snapnet\Engine\Source\Runtime\CoreUObject\Private\UObject\PropertyObject.cpp(227)
[Inline Frame] Rivals2.exe!FUnversionedPropertySerializer::Serialize(FStructuredArchiveSlot) Line 114
at D:\Rivals2Snapnet\Engine\Source\Runtime\CoreUObject\Private\Serialization\UnversionedPropertySerialization.cpp(114)
Rivals2.exe!SerializeUnversionedProperties(const UStruct * Struct, FStructuredArchiveSlot Slot, unsigned char * Data, UStruct * DefaultsStruct, unsigned char * DefaultsData) Line 909
at D:\Rivals2Snapnet\Engine\Source\Runtime\CoreUObject\Private\Serialization\UnversionedPropertySerialization.cpp(909)
Rivals2.exe!UStruct::SerializeTaggedProperties(FStructuredArchiveSlot Slot, unsigned char * Data, UStruct * DefaultsStruct, unsigned char * Defaults, const UObject * BreakRecursionIfFullyLoad) Line 1328
at D:\Rivals2Snapnet\Engine\Source\Runtime\CoreUObject\Private\UObject\Class.cpp(1328)
Rivals2.exe!UClass::SerializeDefaultObject(UObject * Object, FStructuredArchiveSlot Slot) Line 5583
at D:\Rivals2Snapnet\Engine\Source\Runtime\CoreUObject\Private\UObject\Class.cpp(5583)
Rivals2.exe!UBlueprintGeneratedClass::SerializeDefaultObject(UObject * Object, FStructuredArchiveSlot Slot) Line 735
at D:\Rivals2Snapnet\Engine\Source\Runtime\Engine\Private\BlueprintGeneratedClass.cpp(735)
[Inline Frame] Rivals2.exe!UClass::SerializeDefaultObject(UObject *) Line 3410
at D:\Rivals2Snapnet\Engine\Source\Runtime\CoreUObject\Public\UObject\Class.h(3410)
Rivals2.exe!FAsyncPackage::EventDrivenSerializeExport(int LocalExportIndex) Line 3534
at D:\Rivals2Snapnet\Engine\Source\Runtime\CoreUObject\Private\Serialization\AsyncLoading.cpp(3534)
Rivals2.exe!FAsyncPackage::ProcessImportsAndExports_Event() Line 3853
at D:\Rivals2Snapnet\Engine\Source\Runtime\CoreUObject\Private\Serialization\AsyncLoading.cpp(3853)
Rivals2.exe!FAsyncPackage::Event_ProcessImportsAndExports() Line 2912
at D:\Rivals2Snapnet\Engine\Source\Runtime\CoreUObject\Private\Serialization\AsyncLoading.cpp(2912)
Rivals2.exe!FAsyncLoadingThread::QueueEvent_ProcessImportsAndExports::__l2::<lambda>(FAsyncLoadEventArgs & Args) Line 2689
at D:\Rivals2Snapnet\Engine\Source\Runtime\CoreUObject\Private\Serialization\AsyncLoading.cpp(2689)
[Inline Frame] Rivals2.exe!UE::Core::Private::Function::TFunctionRefBase<UE::Core::Private::Function::TFunctionStorage<0>,void __cdecl(FAsyncLoadEventArgs &)>::operator()(FAsyncLoadEventArgs &) Line 555
at D:\Rivals2Snapnet\Engine\Source\Runtime\Core\Public\Templates\Function.h(555)
[Inline Frame] Rivals2.exe!FAsyncLoadEventQueue::PopAndExecute(FAsyncLoadEventArgs &) Line 108
at D:\Rivals2Snapnet\Engine\Source\Runtime\CoreUObject\Private\Serialization\AsyncLoadingThread.h(108)
Rivals2.exe!FAsyncLoadingThread::ProcessAsyncLoading(int & OutPackagesProcessed, bool bUseTimeLimit, bool bUseFullTimeLimit, float TimeLimit, FFlushRequest & FlushRequest) Line 4393
at D:\Rivals2Snapnet\Engine\Source\Runtime\CoreUObject\Private\Serialization\AsyncLoading.cpp(4393)
Rivals2.exe!FAsyncLoadingThread::TickAsyncThread(bool bUseTimeLimit, bool bUseFullTimeLimit, double TimeLimit, bool & bDidSomething, FFlushRequest & FlushRequest) Line 5300
at D:\Rivals2Snapnet\Engine\Source\Runtime\CoreUObject\Private\Serialization\AsyncLoading.cpp(5300)
Rivals2.exe!FAsyncLoadingThread::Run() Line 5225
at D:\Rivals2Snapnet\Engine\Source\Runtime\CoreUObject\Private\Serialization\AsyncLoading.cpp(5225)
Rivals2.exe!FRunnableThreadWin::Run() Line 149
at D:\Rivals2Snapnet\Engine\Source\Runtime\Core\Private\Windows\WindowsRunnableThread.cpp(149)
Rivals2.exe!FRunnableThreadWin::GuardedRun() Line 71
at D:\Rivals2Snapnet\Engine\Source\Runtime\Core\Private\Windows\WindowsRunnableThread.cpp(71)
[Attachment Removed]
Hey Joshua,
apologies for the wait. I’ll loop in one of the developers on the loading team, they might be able to provide you a bit more background info.
Please be aware that making the unversioned serialization support versioning after the fact is a bit too much of an engine modification for us to be able to provide you support with it, if you really want to go this route you’d be mostly on your own.
[Attachment Removed]
Thanks for the update. It looks like an answer has been selected as best, and you’re on another project at this point. Is there anything else you need support with on this topic, or are we good to close up this ticket?
[Attachment Removed]
Unfortunately the -VersionCookedContent flag did not resolve this issue we are seeing.
.\RunUAT.bat BuildCookRun -project="$ENV:ROA2Workspace\Game\Rivals2.uproject" -noP4 -platform="Win64" -target=Rivals2 -clientconfig="Development" -serverconfig="Development" -cook -VersionCookedContent -allmaps -build -stage -pak -archive -archivedirectory="$ENV:ROA2Workspace\Tools\Upload\Platforms"We also have always had the -Unversioned flag removed from our cooking of mods. I am a bit skeptical that the build did build with version content, as we are still seeing a call stack that is calling FUnversionedPropertySerializer::Serialize().
[Attachment Removed]
The above issue is more pressing, as this is what is actually causing our crash. The ResolveResource function tries to access an invalid index (usually 3 or 4) in the linker ExportMap (which never has more than 2 entries, CharacterData_C and Default_CharacterData_C).
[Attachment Removed]
I am actually no longer on the project that had this issue. However, the last direction we were going was to instead use JSON to repopulate a new version of the classes that way we don’t have to worry about versioning. A bit more work than we were hoping for. If I may make a suggestion perhaps add more clarifying info about the versioning to existing documentation. The way that I had interpreted it was that pak files would be able to handle deserialization as long as the Engine version remained the same, but that seems to only be an Editor specific feature.
[Attachment Removed]