Hi, since we upgraded to 5.5.4 (from 5.4.2) we’ve started seeing issues where objects will sometimes not render in the lumen surface cache (often when the scalability settings are changed). This is particularly noticeable with the lighting in heavily shadowed areas where suddenly it’s no longer dark due to GI leaking in when the scalability settings change. There doesn’t seem to be any pattern to which scalability settings change since the issue may occur and may work fine with identical settings. We commonly see it when RT is toggled on/off, but simply adjusting Low/Medium/High/Epic can trigger it as well. And it does occur with all combinations of lumen raytracing/no raytracing/raytraced shadows/no raytraced shadows. Sometimes it’s even incorrect on editor startup. The attached screenshot shows an example where a mesh should be casting a shadow but does not.
Of note: we did already cherry pick to fix a crash we had which may or may not be related.
Unfortunately I was unable to figure out how to repro in an empty 5.5 project. I’m not sure if it’s something with the number of instances we have or perhaps how many overlapping instances we have (there are quite a lot in both cases). I don’t know if there’s any more information I can provide that would be useful in diagnosing the problem here, but let me know if there’s something and I can try to provide it.
Hi, since we upgraded to 5.5.4 (from 5.4.2) we’ve started seeing issues where objects will sometimes not render in the lumen surface cache (often when the scalability settings are changed). This is particularly noticeable with the lighting in heavily shadowed areas where suddenly it’s no longer dark due to GI leaking in when the scalability settings change. There doesn’t seem to be any pattern to which scalability settings change since the issue may occur and may work fine with identical settings. We commonly see it when RT is toggled on/off, but simply adjusting Low/Medium/High/Epic can trigger it as well. And it does occur with all combinations of lumen raytracing/no raytracing/raytraced shadows/no raytraced shadows. Sometimes it’s even incorrect on editor startup. The attached screenshot shows an example where a mesh should be casting a shadow but does not.
Of note: we did already cherry pick this change to fix a crash we had which may or may not be related.
Unfortunately I was unable to figure out how to repro in an empty 5.5 project. I’m not sure if it’s something with the number of instances we have or perhaps how many overlapping instances we have (there are quite a lot in both cases). I don’t know if there’s any more information I can provide that would be useful in diagnosing the problem here, but let me know if there’s something and I can try to provide it.
Hi Andrew,
That’s undoubtedly a strange issue, which would be easiest to solve if we could have a repro project, but let’s see what we can do. For starters, have you checked the card placement visualization to see if cards are present when the shadows are not rendered by chance? In-editor, you can use the r.Lumen.Visualize.CardPlacement cvar for this. Alternatively, you can increase the number of cards generated for the meshes with these problematic spots. If none of these help, you should see if you can break up your meshes into smaller, more manageable instances. That way, the card generation algorithm will find proper surface coverage easier. Please don’t hesitate to let me know if this does not help you out, and I can dig a little further into this.
Cheers,
Tim
Looks like the cards are there when the issue is present (the same as without, as far as I can tell), so I don’t think that’s the source of the issue. Also it doesn’t seem like there’s one specific spot that’s more likely to trigger the issue… I’ve seen it it quite a few places so it’s possible it has no relation to the art content (I just can’t rule it out). I just used that particular area in the screenshot since that one should be safe to keep this non-confidential.
With the issue (wrong lumen, correct cards):
[Image Removed]Normal (for reference):
[Image Removed]
One other observation is that it seems that rapidly switch scalability settings that involve lumen seems to be more likely to trigger it than switching them slowly (though we have seen it simply by starting PIE and loading the player’s saved settings). I’m wondering if switching off a setting requests amortizing “unloading” at the same time as another setting is requesting amortizing “loading”, thus getting an unpredictable mishmash of some meshes that contribute/don’t. I’m not familiar enough with how lumen behaves in this regard to ask about anything more concrete here, unfortunately. Would you know if based on that description there may be a suspicious area of the code to investigate?
Yeah, that’s certainly strange, although are you sure the normal case image you sent is correct? To me, it looks like they’re the Lumen is not picking up any coverage and should be considered the “non-normal” case. Do you see any differences when switching between HWRT and SWRT? Also between what scalability settings are switching back and forth to get the surface cache fill appropriately? Based off of this info, I cannot give you too much to narrow down the issue, but I can offer to make the case private so that we can go into more detail. Let me know what you think.
Oh, does that one still look abnormal from the screenshot? That just seemed to look the most like it did in 5.4 (that section is supposed to be heavily shadowed resulting in the issue being more noticeable, hence why I chose that for the screenshots). The only difference I notice between HWRT and SWRT is the typical quality difference, nothing that seems to be very unusual (though an apples-to-apples comparison is difficult because simply changing that “refreshes” lumen which temporarily fixes the issue, so I have to flick it back and forth several times to reproduce it again).
We have definitely observed the issue by toggling
- r.RayTracing.Shadows
- r.Lumen.HardwareRayTracing
- r.Lumen.HardwareRayTracing.LightingMode
- sg.ShadowQuality (regardless of the value of the others, I’ll add)
I’m not sure if toggling sg.ReflectionQuality or sg.GlobalIlluminationQuality have had it happen explicitly, or only because the user had one or both RT settings on and the lowest quality levels also happen to turn off raytracing (so it’s just coincidence). I have not been able to repro with those two cvars when I don’t also have RT enabled. We also do NOT have VSM/nanite/megalights enabled, I might add. Just CSM shadows from a directional light.
That said, we can certainly take this confidential if there’s more specific detail you’d like to know.
Hi Andrew,
I have gone ahead and made the thread private now to get some more information from you. Would it be possible for you to actually upload of you toggling the various raytracing settings that cause the surface cache to not store any data? I think screenshots make it a bit too difficult to tell what could be going wrong. I have a feeling we might have changed some settings between 5.4.2 and 5.5.4 that might need to be adjusted.
Thanks Tim.
That said, upon trying to reproduce this issue today (which was fairly trivial before), I found I could no longer reproduce it. I talked with one of our other developers that had seen it before and he confirmed it was still happening on his machine on the same editor binary. I realized I had updated my Nvidia drivers to the latest version (576.02) and he had not, so we tried updating his drivers to that version and now he too can no longer reproduce the issue. It seems very likely that something must have either been implemented incorrectly on the Nvidia driver side or was getting corrupted in video memory.
Since it seems like the issue is resolved by the driver update, I don’t think we actually need this ticket to be confidential and can consider this closed.
Appreciate the assistance.
Ok, I am glad that this sounds like merely updating the drivers fixes the issue. I am going to go ahead and close out the ticket then and make it public as well, in case someone else is running into the same problem. Thanks for sharing this!