DDC related crash in Mutable GetMesh code in 5.6

Development Character & Animation
, ,
1

Had this error occur, just trying to open one of our CustomizableObjects. It also happened in game when it was trying to use the same CustomizableObject.

I did some debugging around mu::FModelReader::FOperationID FUnrealMutableModelBulkReader::BeginReadBlock

Block->FileId is 3, but PinnedModelStreamableBulkData->DDCValues is an empty array.

It occurs in this code:

#if WITH_EDITOR
		if (PinnedModelStreamableBulkData->bIsStoredInDDC)
		{
			using namespace UE::DerivedData;

			UE::FSharedString SharedName = UE::FSharedString(TEXT("UnrealMutableModelBulkReader"));

			FCacheGetValueRequest Request;
			Request.Name = SharedName;
			Request.Key = PinnedModelStreamableBulkData->DDCKey;
			Request.Key.Hash = PinnedModelStreamableBulkData->DDCValues[Block->FileId];
2

Hi Kurt,

There seems to be invalid data stored in the DDC, which shouldn’t happen. Please add a breakpoint here and check if the DDCValues array is empty after loading it from the DDC.

// CustomizableObjectCompiler.cpp line 1429
PlatformData->ModelStreamableBulkData = LoadModelStreamableBulk_Internal(ModelStreamablesReader);
PlatformData->ModelStreamableBulkData->bIsStoredInDDC = true;

If it’s empty on load, please try check if the data uploaded to the DDC is valid. To do so, you’ll need to follow these steps.

  1. Turn off Mutable via CVar with ‘Mutable.enabled false’ and then open the conflictive CO.
  2. Move a node to mark the CO dirty and save it.
  3. Add a breakpoint where the DDCValues is populated (CustomizableObjectCompileRunnable.cpp line 489 and 495), and re-enable Mutable with ‘Mutable.enabled true’.
const auto WriteBulkDataDDC = [&PutValueRequests, &ModelStreamablesDDC, SharedName, DefaultDDCPolicy = DefaultDDCPolicy, DDCKey = DDCKey]
			(MutablePrivate::FFile& File, TArray64<uint8>& FileBulkData, uint32 FileIndex)
			{
				FCachePutValueRequest& ValueRequest = PutValueRequests.AddDefaulted_GetRef();
				ValueRequest.Name = SharedName;
				ValueRequest.Policy = DefaultDDCPolicy;
				ValueRequest.Value = FValue::Compress(FSharedBuffer::MakeView(FileBulkData.GetData(), FileBulkData.Num()));
				ValueRequest.Key = DDCKey;
				ValueRequest.Key.Hash = ValueRequest.Value.GetRawHash();
				ModelStreamablesDDC.DDCValues.Add(ValueRequest.Value.GetRawHash()); // Here
			};
 
		constexpr bool bDropData = false;
		MutablePrivate::SerializeBulkDataFiles(*PlatformData.Get(), BulkDataFilesDDC, WriteBulkDataDDC, bDropData); // Here

This should invalidate the DDC and auto-compile the CO. SerializeBulkDataFiles will populate the DDCValues array.

Let me know what you find.

3

The DDCValues array is empty after loading it from the DDC.

When I reenable “Mutable.enabled true”, I do hit those breakpoints, but it immediately is followed by the same error I got before with DDCValues being empty and Block->FileId being 3.

Why might recompiling the CO result in DDCValues being empty?

4

I will also note that this isn’t just affecting one CO, but seems to affect every one I’ve tried so far.

5

That might happen if the CO doesn’t have resources to stream because they’re small and can be kept in memory, but in that case the code that crashes would not be executed.

Can you retry those steps again, and at the second breakpoint, check the values of these variables?

  • PlatformData->ModelStreamableData->Data. It must have entries.
  • BulkDataFilesDDC. It must have more than three entries.

If that is correct, this code should be called BulkDataFilesDDC.Num() times.

ModelStreamablesDDC.DDCValues.Add(ValueRequest.Value.GetRawHash());

6

oh, just realized the code you shared is different than what we have:

// Store streamable resources as FValues
	{
		MUTABLE_CPUPROFILER_SCOPE(SerializeBulkDataForDDC);
	
		const auto WriteBulkDataDDC = [&RecordBuilder](MutablePrivate::FFile& File, TArray64<uint8>& FileBulkData, uint32 FileIndex)
		{
				const FValueId ValueId = GetDerivedDataValueIdForResource(File.DataType, FileIndex, File.ResourceType, File.Flags);
				const FValue Value = FValue::Compress(FSharedBuffer::MakeView(FileBulkData.GetData(), FileBulkData.Num()));
 
				RecordBuilder.AddValue(ValueId, Value);
			};
 
		constexpr bool bDropData = false;
		MutablePrivate::SerializeBulkDataFiles(*PlatformData.Get(), BulkDataFilesDDC, WriteBulkDataDDC, bDropData);
	}

We recently moved to 5.6.1, which I think is when this problem started. A search for DDCValues.Add in the solution returns nothing currently.

7

Yep, that would explain it.

This is the commit that added the DDCValues. You may want to double-check the port to ensure that no more changes are missing.

https://github.com/EpicGames/UnrealEngine/commit/b4e69e4c23d44d14a2fa3b90c28be17f6ad1c6fe

Regards,

Pere