Download

DDC Versioning?

Hi all,

I’m interested in adding some functionality to Unreal Engine as part of an existing project (the project in question is currently on 4.25, however this seems to exist in 5 main as well from my understanding)

I’m noticing some issues when serializing from an old DDC generated before my changes.

For instance, If I wanted to add some additional serialized property for FPositionMeshVertexBuffer (the changes are genericized, but in essence, I want to add support for a compressed vertex buffer format, and to be able to serialize to/from that format wherever we decide to use it.):

void FPositionVertexBuffer::SerializeMetaData(FArchive& Ar)
{
	Ar << Stride << NumVertices;
	// Add additional field
	if (Ar.LicenseeUE4Ver() >= EUnrealEngineObjectLicenseeUE4Version::VER_USE_NEW_POSITION_VERTEX_BUFFER_OPTION)
	{
		Ar << bUseNewOption;
	}
	else 
	{
        // Ideally, we default to using the older behavior if we are loading from an older archive.
		bUseNewOption = false;
	}
}

According to the Unreal Engine Documentation regarding use of versioned assets

This seems like an appropriate use of using engine versioning, however in practice I can see that when I deserialize from a DDC, the versioning info is not set:

See StaticMesh.cpp#L2720 (in ue5-main, the code seems unchanged from UE4 as well…)
https://github.com/EpicGames/UnrealEngine/blob/515378597ff5923607e478efe5a0a4f1ca7b0fb8/Engine/Source/Runtime/Engine/Private/StaticMesh.cpp#L2720

When we initialize the base FArchiveState object we default to the current engine version:
https://github.com/EpicGames/UnrealEngine/blob/515378597ff5923607e478efe5a0a4f1ca7b0fb8/Engine/Source/Runtime/Core/Private/Serialization/Archive.cpp#L72
https://github.com/EpicGames/UnrealEngine/blob/515378597ff5923607e478efe5a0a4f1ca7b0fb8/Engine/Source/Runtime/Core/Private/Serialization/Archive.cpp#L156
https://github.com/EpicGames/UnrealEngine/blob/515378597ff5923607e478efe5a0a4f1ca7b0fb8/Engine/Source/Runtime/Core/Private/UObject/ObjectVersion.cpp#L20

Which causes us to take the top branch and deserialize the boolean that doesn’t exist, which eventually causes a crash as we are no longer aligned with the old data.

I guess my questions are:

  1. is this expected behavior? Are DDCs not intended to be versioned like if this data was loaded from a PAK?

  2. How can one achieve the expected behavior of the new serialization code “just working”? Its not quite feasible to invalidate the cache if we don’t know the status of its contents.

Thank you!

I’m definitely not well versed enough in this area to offer some direct advice, but I can say that a project I’m working on just upgraded from 4.26 to 4.27, and for a brief time, I had both versions of the project installed and I was working in both – the DDC was completely rebuilt for the new installation, and the previous version DDC remained. Switching back and forth between them, it was able to load the DDC for the correct engine version.

So there is at least a mechanism by which it understands that it shouldn’t use the existing cache.

Thanks for the insight! looking into how we version the DDC cache for StaticMeshes led me to this bit of code that retrieves the relevant package from cache:

https://github.com/EpicGames/UnrealEngine/blob/d94b38ae3446da52224bedd2568c078f828b4039/Engine/Source/Runtime/Engine/Private/StaticMesh.cpp#L2591-L2600

And this blob that actually generates the “key” to a cached asset package:

https://github.com/EpicGames/UnrealEngine/blob/d94b38ae3446da52224bedd2568c078f828b4039/Engine/Source/Runtime/Engine/Private/StaticMesh.cpp#L2305-L2318

Which uses this GUID defined here for static meshes:
https://github.com/EpicGames/UnrealEngine/blob/d94b38ae3446da52224bedd2568c078f828b4039/Engine/Source/Runtime/Engine/Private/StaticMesh.cpp#L2303

And similarily, I needed to regenerate the equivalent GUID for SkeletalMesh derived data cache versioning as well, as they both deserialize PositionVertexBufffer: (another “fun” discovery through this endeavor)
https://github.com/EpicGames/UnrealEngine/blob/d94b38ae3446da52224bedd2568c078f828b4039/Engine/Source/Runtime/Engine/Private/SkeletalMeshRenderData.cpp#L188

After making these modifications, I appear to now be backwards-compatible with a “wild” instance of this project, saving me and my build alot of headache.

*Phew* finally fixed.:sweat_smile:

1 Like