Performance Hitch in SetFeatureLevelAndInitialize due to UpdateUniformBuffers

Development Rendering
, ,
1

Hello,

We are seeing seeing large performance hitches on the GameThread on Android mobile devices while waiting for SceneEnd / SetFeatureLevelAndInitialize (up to 20ms on some frames). This hitch occurs when things are getting added to the world during level streaming throughout playing the game.

[Image Removed]

  • Whenever this hitch occurs, there may be up to ~1.3k+ calls to UpdateUniformBuffer (the big purple bar underneath SetFeatureLevelAndInitialize) with an accompanying SceneEnd which has the same number (~1.3k+) calls to TRHILambda
  • Looking at this in code we believe this is coming from this function from file MapBuildData.cpp since in the debugger we see ~1.3k+ elements in the LightmapResourceClusters array.

`void UMapBuildDataRegistry::InitializeClusterRenderingResources(ERHIFeatureLevel::Type InFeatureLevel)
{
// Resource clusters should have been setup during PostLoad, however the cooker makes a dummy level for InitializePhysicsSceneForSaveIfNecessary which is not PostLoaded and contains no build data, ignore it.
check(bSetupResourceClusters || MeshBuildData.Num() == 0);
// If we have any mesh build data, we must have at least one resource cluster, otherwise clusters have not been setup properly.
check(LightmapResourceClusters.Num() > 0 || MeshBuildData.Num() == 0);

ENQUEUE_RENDER_COMMAND(SetFeatureLevelAndInitialize)(
[&LightmapResourceClusters = LightmapResourceClusters, InFeatureLevel](FRHICommandList& RHICmdList)
{
// At this point all lightmap cluster resources are initialized and we can update cluster uniform buffers.
for (FLightmapResourceCluster& Cluster : LightmapResourceClusters)
{
Cluster.SetFeatureLevelAndInitialize(InFeatureLevel);
}
});
}`* Within our game project we have:
+ Baked lighting using a stationary lighting setup where the lightmaps are baked only for the landscape actors
+ Use a lighting scenario which is preloaded in a separate lighting map build data at the start

  • Also we are using world composition for our landscape

It’ll be great to know why there are so many LightmapResourceClusters present, why are they not present every frame and why do they get all update their uniform buffers at such irregular intervals?

One final thing we noticed that this commit from UE5.5 in Level.cpp in ULevel::InitializeRenderingResources may possibly be the fix for this issue? Added StaticLighting support for LevelInstances · EpicGames/UnrealEngine@d01358e.

Specifically this early return in that function

if (!IsMapBuildDataOwner()) { //If we're using the MapBuildData of our owning Level don't touch the rendering resources, owning level will take care of it return; }

Will back porting these changes be a viable fix for us?

Regards,

Arnav

2

Hi Arnav,

The number of clusters seems quite high but not impossible and would depend entirely on the level data. As I imagine this is in a large product, providing a repro case may prove to be difficult?

Can you confirm if the device on which this was profiled is running OpenGL ES or Vulkan? If it is on OpenGL ES, can you confirm if the hitch is equally as significant on Vulkan?

One possible workaround would be to time slice the initialization across a few frames in UWorld::AddToWorld but this would be approximate but perhaps sufficient.

Regarding the commit to Level.cpp mentioned, I consulted with the implementor and it doesn’t appear it would have any impact on the issue at hand.

Best regards.

3

Hey Stéphanem

Yes, unfortunately providing a repro case may not be possible due to the scale and confidentiality of this project.

This issue happens for both OpenGL ES and Vulkan. And the hitch values are more or less the same.

We’ve added time slicing to be considered as a work around. But in the mean time do you think this issue could be due to merging all the world composition lighting information into one lighting build map which we load at initialization. Whereas these hitches happen during gameplay. Adding a log message I get this information (may be relevant?)

<LevelName>:PersistentLevel has 2201 meshes and 1367 lightmap resource clustersCheers.

4

Hi Arnav,

Can you confirm that you mean that your app is merging all the world composition lighting info into a single build map which is loaded at initialization but are experiencing the hitches during gameplay?

Best regards.

5

Hey Stéphane

Yep, that’s exactly it. Here’ some additional information regarding the level setup:

  1. Multiple levels in the world composition with various layer streaming distance
  2. Lighting setup
    1. Direction Light - Stationary
    2. Sky Light - Movable
    3. No other light sources
  3. All static mesh actors lightmap type is set to “force volumetric” so that they can cast shadow but not self shadow.
  4. All static mesh actors have their cast dynamic shadow disabled.
  5. Build data is saved only in the lighting level. So single light build data.
6

Hi Arnav,

In terms of where the call is occurring, it would lend itself well to time slicing. However, can you confirm “Build data is saved only in the lighting level. So single light build data.”? Under those conditions there shouldn’t be any calls to InitializeCluserRenderResources.

Best regards.

7

Hey Stéphane,

Yes we can confirm that the build data is only saved in the lighting level (i.e. single light build data). As you mentioned we shouldn’t be seeing these hitches/calls to InitializeClusterRenderResources yet we do.

Regards,

Arnav

8

Hi Arnav.

The team has revisited the code do not have clear visibility how this scenario could happen with the current light build data setup. The suggest adding logging in

ULevel::InitializeRenderingResources()/ReleaseRenderingResources() to indicate which levels are behind these calls and if there is MapBuildData its stats (these can be obtained via

UMapBuildDataRegistry::GetLightmapResourceClusterStats) would help better determine the origin of the issue.

Best regards.

9

Hi Arnav,

Can you confirm the call stack when the UMapBuildDataRegistry::InitializeClusterRenderingResources is call is made during gameplay that triggers the hitch?

Thanks.

10

Hey Stéphane,

Here is the call stack when we have 1367 lightmap resource clusters. The incoming feature level is ES3_1 and `MeshBuildData.Num()` is 2201. This call stack was captured on the Android mobile device

[Image Removed]Cheers