Mutables crash Editor when using Cloud DDC since 5.6

We’re having an issue where Mutable/CustomizableObject assets are being read incorrectly for PIE from Cloud (S3) DDC after merging 5.6 UE to our project. At least this is my understanding of it.

Firstly it’s hitting an ensure that it does not hit when launching without Cloud DDC.

Ensure condition failed: bSuccess [File:D:\PROJECT\UnrealEngine\Engine\Plugins\Mutable\Source\CustomizableObject\Private\MuCO\UnrealMutableModelDiskStreamer.cpp] [Line: 384]

Summary

UnrealEditor_CustomizableObject!FUnrealMutableModelBulkReader::BeginReadBlock'::92’::<lambda_3>::operator()() [D:\PROJECT\UnrealEngine\Engine\Plugins\Mutable\Source\CustomizableObject\Private\MuCO\UnrealMutableModelDiskStreamer.cpp:384]
UnrealEditor_DerivedDataCache!``UE::DerivedData::FCacheStoreAsync::Execute<UE::DerivedData::FCacheGetRequest,TUniqueFunction<void __cdecl(UE::DerivedData::FCacheGetResponse && __ptr64)>,void (__cdecl UE::DerivedData::ICacheStore::)(TArrayView<UE::DerivedDat()
UnrealEditor_DerivedDataCache!UE::DerivedData::FCacheStoreHierarchy::TGetBatch<UE::DerivedData::FCacheStoreHierarchy::TBatchParamsUE::DerivedData::FCachePutRequest,UE::DerivedData::FCacheGetRequest >::DispatchRequests()
UnrealEditor_DerivedDataCache!UE::DerivedData::FCacheStoreHierarchy::TGetBatch<UE::DerivedData::FCacheStoreHierarchy::TBatchParamsUE::DerivedData::FCachePutRequest,UE::DerivedData::FCacheGetRequest >::Begin()
UnrealEditor_DerivedDataCache!UE::DerivedData::FCacheStoreHierarchy::Get()
UnrealEditor_DerivedDataCache!UE::DerivedData::FCacheStoreAsync::Execute<UE::DerivedData::FCacheGetRequest,TUniqueFunction<void __cdecl(UE::DerivedData::FCacheGetResponse && __ptr64)>,void (__cdecl UE::DerivedData::ICacheStore::*)(TArrayView<UE::DerivedData() UnrealEditor_DerivedDataCache!UE::DerivedData::FCacheStoreAsync::Execute<UE::DerivedData::FCacheGetRequest,TUniqueFunction<void __cdecl(UE::DerivedData::FCacheGetResponse && __ptr64)>,void (__cdecl UE::DerivedData::ICacheStore::
)(TArrayView<UE::DerivedData()
UnrealEditor_DerivedDataCache!UE::DerivedData::FThreadPoolTaskRequest::TryEnd'::2’::<lambda_1>::operator()()
UnrealEditor_DerivedDataCache!UE::DerivedData::IRequestOwner::End<UE::DerivedData::FThreadPoolTaskRequest::TryEnd'::2’::<lambda_1> >()
UnrealEditor_DerivedDataCache!UE::DerivedData::FThreadPoolTaskRequest::TryEnd()
UnrealEditor_DerivedDataCache!UE::DerivedData::FThreadPoolTaskRequest::DoThreadedWork()
UnrealEditor_Core!FQueuedThread::Run()
UnrealEditor_Core!FRunnableThreadWin::Run()

Secondly it crashes (not immediately after, ensure may be unrelated). This crash also does not happen without Cloud DDC.

Assertion failed: Pos + InSize <= Size [File:C:\PROJECT\UnrealEngine\Engine\Plugins\Mutable\Source\MutableRuntime\Private\MuR\Serialisation.cpp] [Line: 110]

Summary

UnrealEditor_Core!FDebug::CheckVerifyFailedImpl2()
UnrealEditor_MutableRuntime!mu::FInputMemoryStream::Read()
UnrealEditor_MutableRuntime!mu::FMeshBufferSet::Unserialise()
UnrealEditor_MutableRuntime!mu::FMesh::Unserialise()
UnrealEditor_MutableRuntime!mu::FMesh::StaticUnserialise()
UnrealEditor_MutableRuntime!mu::CodeRunner::FLoadMeshRomTask::Complete()
UnrealEditor_MutableRuntime!mu::CodeRunner::Run()
UnrealEditor_MutableRuntime!UE::Tasks::Private::TExecutableTaskBase<mu::CodeRunner::Run'::122’::<lambda_1>,void,void>::ExecuteTask()
UnrealEditor_MutableRuntime!UE::Tasks::Private::FTaskBase::TryExecuteTask()
UnrealEditor_MutableRuntime!LowLevelTasks::TTaskDelegate<LowLevelTasks::FTask * __ptr64 __cdecl(bool),48>::TTaskDelegateImpl<LowLevelTasks::FTask::Init<UE::Tasks::Private::FTaskBase::Init’::2'::<lambda_1> >'::13’::<lambda_1>,0>::CallAndMove()
UnrealEditor_Core!LowLevelTasks::FTask::ExecuteTask()
UnrealEditor_Core!LowLevelTasks::FScheduler::ExecuteTask()
UnrealEditor_Core!LowLevelTasks::FScheduler::WorkerLoop()
UnrealEditor_Core!LowLevelTasks::FScheduler::CreateWorker'::2’::<lambda_1>::operator()()
UnrealEditor_Core!FThreadImpl::Run()
UnrealEditor_Core!FRunnableThreadWin::Run()

Our engine is modified, although I don’t think these modifications affect Cloud DDC and Mutable parts of the code.

VERY IMPORTANT: when user’s cvar “mutable.DerivedDataCacheUsage” is false, this crash doesn’t happen even when using Cloud DDC.

I would really love some pointers on how to fix that/if it’s a known problem/etc., this affects our remote team in a big way.