Mutables crash Editor when using Cloud DDC since 5.6

Development Platform & Builds
, , ,
1

We’re having an issue where Mutable/CustomizableObject assets are being read incorrectly for PIE from Cloud (S3) DDC after merging 5.6 UE to our project. At least this is my understanding of it.

Firstly it’s hitting an ensure that it does not hit when launching without Cloud DDC.

Ensure condition failed: bSuccess [File:D:\PROJECT\UnrealEngine\Engine\Plugins\Mutable\Source\CustomizableObject\Private\MuCO\UnrealMutableModelDiskStreamer.cpp] [Line: 384]

Summary

UnrealEditor_CustomizableObject!FUnrealMutableModelBulkReader::BeginReadBlock'::92’::<lambda_3>::operator()() [D:\PROJECT\UnrealEngine\Engine\Plugins\Mutable\Source\CustomizableObject\Private\MuCO\UnrealMutableModelDiskStreamer.cpp:384]
UnrealEditor_DerivedDataCache!``UE::DerivedData::FCacheStoreAsync::Execute<UE::DerivedData::FCacheGetRequest,TUniqueFunction<void __cdecl(UE::DerivedData::FCacheGetResponse && __ptr64)>,void (__cdecl UE::DerivedData::ICacheStore::)(TArrayView<UE::DerivedDat()
UnrealEditor_DerivedDataCache!UE::DerivedData::FCacheStoreHierarchy::TGetBatch<UE::DerivedData::FCacheStoreHierarchy::TBatchParamsUE::DerivedData::FCachePutRequest,UE::DerivedData::FCacheGetRequest >::DispatchRequests()
UnrealEditor_DerivedDataCache!UE::DerivedData::FCacheStoreHierarchy::TGetBatch<UE::DerivedData::FCacheStoreHierarchy::TBatchParamsUE::DerivedData::FCachePutRequest,UE::DerivedData::FCacheGetRequest >::Begin()
UnrealEditor_DerivedDataCache!UE::DerivedData::FCacheStoreHierarchy::Get()
UnrealEditor_DerivedDataCache!UE::DerivedData::FCacheStoreAsync::Execute<UE::DerivedData::FCacheGetRequest,TUniqueFunction<void __cdecl(UE::DerivedData::FCacheGetResponse && __ptr64)>,void (__cdecl UE::DerivedData::ICacheStore::*)(TArrayView<UE::DerivedData() UnrealEditor_DerivedDataCache!UE::DerivedData::FCacheStoreAsync::Execute<UE::DerivedData::FCacheGetRequest,TUniqueFunction<void __cdecl(UE::DerivedData::FCacheGetResponse && __ptr64)>,void (__cdecl UE::DerivedData::ICacheStore::)(TArrayView<UE::DerivedData()
UnrealEditor_DerivedDataCache!UE::DerivedData::FThreadPoolTaskRequest::TryEnd'::2’::<lambda_1>::operator()()
UnrealEditor_DerivedDataCache!UE::DerivedData::IRequestOwner::End<UE::DerivedData::FThreadPoolTaskRequest::TryEnd'::2’::<lambda_1> >()
UnrealEditor_DerivedDataCache!UE::DerivedData::FThreadPoolTaskRequest::TryEnd()
UnrealEditor_DerivedDataCache!UE::DerivedData::FThreadPoolTaskRequest::DoThreadedWork()
UnrealEditor_Core!FQueuedThread::Run()
UnrealEditor_Core!FRunnableThreadWin::Run()

Secondly it crashes (not immediately after, ensure may be unrelated). This crash also does not happen without Cloud DDC.

Assertion failed: Pos + InSize <= Size [File:C:\PROJECT\UnrealEngine\Engine\Plugins\Mutable\Source\MutableRuntime\Private\MuR\Serialisation.cpp] [Line: 110]

Summary

UnrealEditor_Core!FDebug::CheckVerifyFailedImpl2()
UnrealEditor_MutableRuntime!mu::FInputMemoryStream::Read()
UnrealEditor_MutableRuntime!mu::FMeshBufferSet::Unserialise()
UnrealEditor_MutableRuntime!mu::FMesh::Unserialise()
UnrealEditor_MutableRuntime!mu::FMesh::StaticUnserialise()
UnrealEditor_MutableRuntime!mu::CodeRunner::FLoadMeshRomTask::Complete()
UnrealEditor_MutableRuntime!mu::CodeRunner::Run()
UnrealEditor_MutableRuntime!UE::Tasks::Private::TExecutableTaskBase<mu::CodeRunner::Run'::122’::<lambda_1>,void,void>::ExecuteTask()
UnrealEditor_MutableRuntime!UE::Tasks::Private::FTaskBase::TryExecuteTask()
UnrealEditor_MutableRuntime!LowLevelTasks::TTaskDelegate<LowLevelTasks::FTask * __ptr64 __cdecl(bool),48>::TTaskDelegateImpl<LowLevelTasks::FTask::Init<UE::Tasks::Private::FTaskBase::Init’::2'::<lambda_1> >'::13’::<lambda_1>,0>::CallAndMove()
UnrealEditor_Core!LowLevelTasks::FTask::ExecuteTask()
UnrealEditor_Core!LowLevelTasks::FScheduler::ExecuteTask()
UnrealEditor_Core!LowLevelTasks::FScheduler::WorkerLoop()
UnrealEditor_Core!LowLevelTasks::FScheduler::CreateWorker'::2’::<lambda_1>::operator()()
UnrealEditor_Core!FThreadImpl::Run()
UnrealEditor_Core!FRunnableThreadWin::Run()

Our engine is modified, although I don’t think these modifications affect Cloud DDC and Mutable parts of the code.

VERY IMPORTANT: when user’s cvar “mutable.DerivedDataCacheUsage” is false, this crash doesn’t happen even when using Cloud DDC.

I would really love some pointers on how to fix that/if it’s a known problem/etc., this affects our remote team in a big way.

2

Hi alomatkina,
This and other issues related to the use of DDC have been fixed in the latest version (UE5 Main). The change you’re looking for is the one below.

[Mutable] Fix crash when finalizing a DDC read request due to the request owner being destroyed within the completion callback.
Fix crash after failed DDC read request.
Fix missing DDC FValues.
Fix performance regression when streaming data from DDC due to the size of the files to download and the decompression method used.

https://github.com/EpicGames/UnrealEngine/commit/f9956e59cc692cb0d34046b6bf712a4bf4bc895e

Regards,
Pere

3

Hello. Thanks a lot! As of right now, after some testing, this seems to work. (or at least it’s not not-working :slight_smile: )