In a large WorldPartition world in UE5.4, we are seeing prolific crashes during asynchronous rendering of foliage: both dynamically generated landscape grass (UGrassInstancedStaticMeshComponent) and painted foliage (UFoliageInstancedStaticMeshComponent).
We have added engine changes introducing an ensure, to enable more descriptive logging of the problematic StaticMesh just prior to the crash. We noticed that this ensure is triggered before the crashing callstack listed below. However, in our investigation, it did not appear that any particular StaticMesh was the culprit, as the crashes were reproduced for the majority of foliage StaticMesh types. Additionally, the crashes are sporadic, but consistent, occuring for a wide range of hardware. Note, barring the crashes, both dynamic landscape grass and painted foliage render as expected.
FPrimitiveSceneProxy::BuildUniformShaderParameters, PrimitiveSceneProxy.cpp line 885
if (GetInstanceDataHeader().NumInstances > 0)
{
#if DO_ENSURE
ensureAlwaysMsgf(InstanceSceneDataBuffers->InstanceLocalBounds.Num() > 0, TEXT("StaticMesh (%s) on Landscape (%s) with bugged InstanceLocalBounds"), *GetResourceName().ToString(), *GetOwnerName().ToString());
#endif
Builder.InstanceLocalBounds(InstanceSceneDataBuffers->GetInstanceLocalBounds(0));
}
Here is the callstack for the crash:
FInstanceSceneDataBuffers::GetInstanceToPrimitiveRelative (InstanceDataSceneProxy.h:169)
``FGPUScene::UploadGeneral<T>'::`2'::<T>::operator()'::`13'::<T>::operator() (GPUScene.cpp:1209)
ParallelForImpl::CallBody (ParallelFor.h:80)
ParallelForImpl::ParallelForInternal<T> (ParallelFor.h:145)
ParallelForTemplate (ParallelFor.h:496)
`FGPUScene::UploadGeneral<T>'::`2'::<T>::operator() (GPUScene.cpp:1181)
FRDGBuilder::AddCommandListSetupTask::__l2::<T>::operator() (RenderGraphBuilder.inl:644)
FRDGBuilder::AddCommandListSetupTask<T> (RenderGraphBuilder.inl:656)
FRDGBuilder::AddCommandListSetupTask (RenderGraphBuilder.inl:605)
FGPUScene::UploadGeneral<T> (GPUScene.cpp:1128)
FGPUScene::UpdateInternal (GPUScene.cpp:865)
FGPUScene::Update (GPUScene.cpp:1774)
FScene::Update (RendererScene.cpp:6743)
FSceneRenderer::OnRenderBegin (SceneRendering.cpp:3658)
FDeferredShadingSceneRenderer::Render (DeferredShadingRenderer.cpp:1474)
FRHICommandSetBreadcrumbStackTop::{ctor} (RHICommandList.h:2237)
FRHIComputeCommandList::PushBreadcrumb (RHICommandList.h:3007)
FBreadcrumbEvent::{ctor} (RealtimeGPUProfiler.h:52)
UpdateSceneCaptureContentDeferred_RenderThread (SceneCaptureRendering.cpp:385)
UpdateSceneCaptureContent_RenderThread (SceneCaptureRendering.cpp:524)
`FScene::UpdateSceneCaptureContents'::`6'::<T>::operator() (SceneCaptureRendering.cpp:1130)
UE::Core::Private::Function::TFunctionRefBase<T>::operator() (Function.h:555)
`FRenderThreadCommandPipe::EnqueueAndLaunch'::`5'::<T>::operator() (RenderingThread.cpp:1748)
UE::Core::Private::Function::TFunctionRefBase<T>::operator() (Function.h:555)
TFunctionGraphTaskImpl<T>::DoTaskImpl (TaskGraphInterfaces.h:1733)
TFunctionGraphTaskImpl<T>::DoTask (TaskGraphInterfaces.h:1726)
TGraphTask<T>::ExecuteTask (TaskGraphInterfaces.h:1235)
FBaseGraphTask::Execute (TaskGraphInterfaces.h:840)
FNamedTaskThread::ProcessTasksNamedThread (TaskGraph.cpp:760)
FNamedTaskThread::ProcessTasksUntilQuit (TaskGraph.cpp:650)
FRHICommandListImmediate::AcquireThreadOwnership (RHICommandList.h:4523)
RHIAcquireThreadOwnership (RHICommandList.h:5253)
RenderingThreadMain::__l7::FScopedRHIThreadOwnership::{ctor} (RenderingThread.cpp:399)
RenderingThreadMain (RenderingThread.cpp:411)
FRenderingThread::Run (RenderingThread.cpp:538)
FRunnableThreadWin::Run (WindowsRunnableThread.cpp:146)
FRunnableThreadWin::GuardedRun (WindowsRunnableThread.cpp:79)
BaseThreadInitThunk
RtlUserThreadStart
Based on our investigation we currently believe these crashes occur when foliage state changes (primarily on removal for UGrassInstancedStaticMeshComponent). And, specifically, have something to do with the foliages’ associated Landscape components not being visible client-side during these state changes. For instance, in the case of crashes induced by UGrassInstancedStaticMeshComponent, we found that crashes were mitigated by manually removing grass instances via LandscapeSubsystem and then preventing dynamic grass generation for a time, in certain instances (i.e. removing prior to teleport and re-enabling and regenerating grass after teleport completes).
Any help would be greatly appreciated!