Possible race condition in UMaterialInterface::SortTextureStreamingData with FastGeoStreaming

Hey folks!

We are occasionally running into this ensure in UMaterialInterface::GetTextureDensity during PIE:

(MatchingData is valid, but the TextureNames are not equal)

Unfortunately, we haven’t found a reliable way to reproduce this, but my theory is that this is caused by FastGeoStreaming registering multiple primitives using the same material on different threads at the same time.

FindTextureStreamingDataIndexRange will internally call SortTextureStreamingData, which modifies the TextureStreamingData array of the material and (as far as I can tell) is not threadsafe.

Cheers,

Dave

Callstack for the ensure (I added this when I created this ticket, but apparently it got lost somehow?):

 	UnrealEditor-Engine.dll!UMaterialInterface::GetTextureDensity(FName TextureName, const FMeshUVChannelInfo & UVChannelData) Line 2051	C++
	UnrealEditor-Engine.dll!UMaterialInstance::GetTextureDensity(FName TextureName, const FMeshUVChannelInfo & UVChannelData) Line 5401	C++
 	[Inline Frame] UnrealEditor-Engine.dll!FStreamingTextureLevelContext::ProcessMaterial::__l2::<lambda_1>::operator()(UTexture *) Line 666	C++
 	UnrealEditor-Engine.dll!FStreamingTextureLevelContext::ProcessMaterial(const UE::Math::TBoxSphereBounds<double,double> & ComponentBounds, const FPrimitiveMaterialInfo & MaterialData, float ComponentScaling, TArray<FStreamingRenderAssetPrimitiveInfo,TSizedDefaultAllocator<32>> & OutStreamingTextures, bool bIsComponentBuildDataValid, const UPrimitiveComponent * DebugComponent) Line 752	C++
 	UnrealEditor-Engine.dll!FStaticMeshSceneProxy::GetStreamableRenderAssetInfo(const UE::Math::TBoxSphereBounds<double,double> & InPrimitiveBounds, TArray<FStreamingRenderAssetPrimitiveInfo,TSizedDefaultAllocator<32>> & OutStreamableRenderAssets) Line 552	C++
 	[Inline Frame] UnrealEditor-Renderer.dll!FSimpleStreamableAssetManager::FRegister::{ctor}(const FPrimitiveSceneProxy *) Line 493	C++
 	UnrealEditor-Renderer.dll!FScene::BatchAddPrimitivesInternal<FPrimitiveSceneDesc>(TArrayView<FPrimitiveSceneDesc *,int> InPrimitives) Line 1452	C++
 	UnrealEditor-Renderer.dll!FScene::AddPrimitive(FPrimitiveSceneDesc * Primitive) Line 1321	C++
 	UnrealEditor-FastGeoStreaming.dll!FFastGeoPrimitiveComponent::CreateRenderState(FRegisterComponentContext * Context) Line 368	C++
 	UnrealEditor-FastGeoStreaming.dll!UFastGeoContainer::OnCreateRenderState_Concurrent::__l5::<lambda_1>::operator()(int Index) Line 454	C++
 	[Inline Frame] UnrealEditor-FastGeoStreaming.dll!UE::Core::Private::Function::TFunctionRefBase<UE::Core::Private::Function::FFunctionRefStoragePolicy,void __cdecl(int)>::operator()(int <Params_0>) Line 414	C++
 	[Inline Frame] UnrealEditor-FastGeoStreaming.dll!ParallelForImpl::CallBody(const TFunctionRef<void __cdecl(int)> &) Line 81	C++
 	UnrealEditor-FastGeoStreaming.dll!`ParallelForImpl::ParallelForInternal<TFunctionRef<void __cdecl(int)>,`ParallelFor'::`2'::<lambda_1>,std::nullptr_t>'::`2'::FParallelExecutor::operator()(const bool bIsMaster) Line 358	C++
 	UnrealEditor-FastGeoStreaming.dll!ParallelForImpl::ParallelForInternal<TFunctionRef<void __cdecl(int)>,`ParallelFor'::`2'::<lambda_1>,std::nullptr_t>(const wchar_t * DebugName, int Num, int MinBatchSize, TFunctionRef<void __cdecl(int)> Body, ParallelFor::__l2::<lambda_1> CurrentThreadWorkToDoBeforeHelping, EParallelForFlags Flags, const TArrayView<std::nullptr_t,int> & Contexts) Line 440	C++
 	[Inline Frame] UnrealEditor-FastGeoStreaming.dll!ParallelFor(int) Line 528	C++
 	UnrealEditor-FastGeoStreaming.dll!UFastGeoContainer::OnCreateRenderState_Concurrent() Line 460	C++
 	[Inline Frame] UnrealEditor-FastGeoStreaming.dll!FFastGeoAsyncRenderStateJobQueue::FJob::Execute() Line 184	C++
 	UnrealEditor-FastGeoStreaming.dll!FFastGeoAsyncRenderStateJobQueue::FJobSet::Execute() Line 162	C++
 	UnrealEditor-FastGeoStreaming.dll!UE::Tasks::Private::FTaskBase::TryExecuteTask() Line 518	C++
 	[Inline Frame] UnrealEditor-FastGeoStreaming.dll!UE::Tasks::Private::FTaskBase::Init::__l2::<lambda_1>::operator()() Line 180	C++
 	[Inline Frame] UnrealEditor-FastGeoStreaming.dll!LowLevelTasks::FTask::Init::__l13::<lambda_1>::operator()(const bool) Line 499	C++
 	[Inline Frame] UnrealEditor-FastGeoStreaming.dll!Invoke(LowLevelTasks::FTask::Init::__l13::<lambda_1> &) Line 47	C++
 	[Inline Frame] UnrealEditor-FastGeoStreaming.dll!LowLevelTasks::TTaskDelegate<LowLevelTasks::FTask * __cdecl(bool),48>::TTaskDelegateImpl<`LowLevelTasks::FTask::Init<`UE::Tasks::Private::FTaskBase::Init'::`2'::<lambda_1>>'::`13'::<lambda_1>,0>::Call(void *) Line 162	C++
 	UnrealEditor-FastGeoStreaming.dll!LowLevelTasks::TTaskDelegate<LowLevelTasks::FTask * __cdecl(bool),48>::TTaskDelegateImpl<`LowLevelTasks::FTask::Init<`UE::Tasks::Private::FTaskBase::Init'::`2'::<lambda_1>>'::`13'::<lambda_1>,0>::CallAndMove(LowLevelTasks::TTaskDelegate<LowLevelTasks::FTask * __cdecl(bool),48> & Destination, void * InlineData, unsigned int DestInlineSize, bool <Params_0>) Line 171	C++
 	[Inline Frame] UnrealEditor-Core.dll!LowLevelTasks::TTaskDelegate<LowLevelTasks::FTask * __cdecl(bool),48>::CallAndMove(LowLevelTasks::TTaskDelegate<LowLevelTasks::FTask * __cdecl(bool),48> &) Line 309	C++
 	UnrealEditor-Core.dll!LowLevelTasks::FTask::ExecuteTask() Line 627	C++
 	UnrealEditor-Core.dll!LowLevelTasks::FScheduler::ExecuteTask(LowLevelTasks::FTask * InTask) Line 419	C++
 	[Inline Frame] UnrealEditor-Core.dll!LowLevelTasks::FScheduler::TryExecuteTaskFrom(LowLevelTasks::Private::FWaitEvent *) Line 698	C++
 	UnrealEditor-Core.dll!LowLevelTasks::FScheduler::WorkerLoop(LowLevelTasks::Private::FWaitEvent * WorkerEvent, LowLevelTasks::Private::TLocalQueueRegistry<1024,1024>::TLocalQueue * WorkerLocalQueue, unsigned int WaitCycles, bool bPermitBackgroundWork) Line 757	C++
 	[Inline Frame] UnrealEditor-Core.dll!LowLevelTasks::FScheduler::WorkerMain(LowLevelTasks::Private::FWaitEvent * WorkerEvent, LowLevelTasks::Private::TLocalQueueRegistry<1024,1024>::TLocalQueue * WorkerLocalQueue, unsigned int WaitCycles, bool) Line 816	C++
 	UnrealEditor-Core.dll!LowLevelTasks::FScheduler::CreateWorker::__l2::<lambda_1>::operator()() Line 220	C++
 	[Inline Frame] UnrealEditor-Core.dll!UE::Core::Private::Function::TFunctionRefBase<UE::Core::Private::Function::TFunctionStorage<1>,void __cdecl(void)>::operator()() Line 414	C++
 	UnrealEditor-Core.dll!FThreadImpl::Run() Line 69	C++
 	UnrealEditor-Core.dll!FRunnableThreadWin::Run() Line 159	C++
 	UnrealEditor-Core.dll!FRunnableThreadWin::GuardedRun() Line 71	C++
 	kernel32.dll!BaseThreadInitThunk()	Unknown
 	ntdll.dll!RtlUserThreadStart()	Unknown

Hello!

We are currently making a pass at making that code thread safe for version 5.8. Some of the problems are coming from the Simple Streamable Asset Manager. It can be turned off using this cvar: s.StreamableAssets.UseSimpleStreamableAssetManager 0.

Regarding the ensure, it appears that the impact will be at the mip streaming level where you might end up with the wrong mips on startup. Our guess is that the problem will happen when streaming a cell that contains multiple FastGeo components that use the same material that was not already loaded. A quick workaround could be to add a critical section in UMaterialInterface::SortTextureStreamingData so only one instance can sort the array at one time. This will slow things down at the editor level but it might be better than the ensure.

Regards,

Martin

Hey Martin, thanks for the reply!

Unfortunately, we cannot turn off the Simple Streamable Asset Manager, as trying to use FastGeo without it will kind of break texture streaming (see [Content removed] ).

So we’ll go with adding a critical section. Since this is editor-only code, that seems to be a good enough solution.