Stale GPU Scene Lighting Update Issue

Hello!

We’re tripping the stale light update ensure (“Data for Id = %d in GPU Scene Lights is stale.”) inside of FGPUScene::UpdateGPULights. It appears to be an update race condition, or a SceneLightInfoUpdates command omission when updating the Scene’s GPU lights.

What’s happening is that a point light has changed its attenuation radius and reflected this change in Scene.Lights, but none of the AddedUpdated commands in the change set acknowledged it. Therefore, the check-up loop further down finds it as a discrepancy against LightDataArray.

If we log the frame update changes, we see that the very next change set will, indeed, contain an AddedUpdated command for it. In other words:

// Where n is the light index

Frame 0:

- Scene.Lights[n] has an updated InvRadius, but ChangeSet.SceneLightInfoUpdates has no update command for n.

- Further down, the DO_CHECK loop over all the active GPU lights will catch this discrepancy.

Frame 1:

- ChangeSet.SceneLightInfoUpdates has the expected entry for n, updating it as expected (one frame late)

I couldn’t figure out why this was happening, but looking at the sequencer (screenshot attached), I noted that the radius update was keyed exactly at the same time as the light’s transformation key. If I move the radius update a couple of frames to the right, the problem goes away. It seems very unlikely for this to be the cause, but the timing between these two keys definitely brings or removes the assertion.

Please let me know if you have some thoughts or suggestions on how to dive deeper into this issue.

Thanks!

Hi Dan,

Thanks for reaching out about this. To clarify, since you have two updates to the point light component (attenuation radius and transformation), would you expect both to happen within the same frame and not have the data go stale? I found an interesting piece of related information from the comments of the TSceneUpdateCommandQueue class, which I pasted and highlighted below. I wonder if the behavior you are seeing is by design, in that it might not be possible at this moment to enqueue two ESceneUpdateCommandFilter::AddedUpdated commands to the SceneLightInfoUpdates queue.

[Image Removed]For starters, verifying the commands inside the queue before the assert is triggered in the for loop would be good. Maybe only one of the two commands made it into the queue because the first was overwritten or ignored when we iterate over the queued commands in FGPUScene::UpdateGPULights.

Next, I would check how we enqueue light info updates by setting some breakpoints where TSceneUpdateCommandQueue::Enqueue is called for our light sources. The following functions inside FScene should have those:

  1. FScene::UpdateLightInternal_RenderThread(FLightSceneProxy* LightSceneProxy, UpdatePayloadType&& InUpdatePayload)
  2. FScene::UpdateLightInternal(FLightSceneProxy* LightSceneProxy, UpdatePayloadType&& InUpdatePayload)

Based on some of the code present there, I suspect we handle the update to the light’s transform differently, so you are not hitting the assert once you move the update to another frame.

I hope that helps you get started. Do let me know if you have any follow-up questions.

Okay, that sounds reasonable to me. If you have any more questions, please feel free to ask again. Otherwise, I will consider the case solved then.

Hello, this may be a slightly different issue but I’m finding that any change to the attenuation radius of a non-shadow-casting light hits this ensure unless there is a change to color or transform in the same frame, because

FLocalLightSceneProxy::UpdateRadius_GameThread doesn’t trigger a GPU scene update. This can easily be reproduced by making a BP with a point light component, disabling cast shadows, and changing its attenuation radius on tick.

Hi Daniel,

Your issue might still be related. Can you try the proposed workaround and file a new ticket if it does not fix your issue? That way, we can keep unrelated problems in one case.

Hi, Tim!

Thanks for the reply and for the context on this. I had missed the comment on the Scene Update Command Queue. By the looks of it, I concur this is indeed expected. I suspect transform, color, and brightness are handled as direct update commands (along with modifying the SceneInfo), while other light parameter updates are caught with an Add command. So we don’t miss or overwrite parameter changes, we just get them in a slightly different order. The check on UpdateGPULights (which will process the AddUpdate) is just a bit naive for this particular case.

This is easy to sidestep by simply moving the radius change to the next frame, and I believe this particular cutscene is the only instance where I’ve seen this, so I’m going to go with that mitigation for now.

Thank you again for the help!