Lumen GI and Reflections feedback thread

jblackwell · August 6, 2023, 1:01am

Considering how hard Nvidia (and now Intel) are pushing real-time path-tracing, I don’t think it’ll be too long at all, for PCs at least. Consoles are a mystery, but a handful of breakthroughs can change the game (EG Nanite).

In regards to your frustration with reflections, I agree that lumen scene reflections can look messy and somewhat illusion-breaking, but I’m not sure standalone RT is actually better in this case, and here’s what makes me think that:

UE Content Examples, 5.3 preview. Lumen GI and reflections with hit lighting (two bounces), everything maxed out. Note the reflection of the white wall on the black floor, and the white wall is indirectly lit.

Same scene, identical settings, minus standalone RT reflections enabled, 2SPP, two bounces, but with a roughness cutoff of 1 (so almost GI at this point).

Note how strangely blue the standalone RT reflections are. They’re almost acting like the wall doesn’t exist from a lighting perspective, and the metal bar is very blue even though it has no direct view of the sky.

If you go into the reflections only viewmode, it’s pretty easy to see why (sidenote: lumen only traces multiple bounces of reflections for surfaces that are smooth enough; it otherwise uses the lumen scene). Even though the lighting is blotchy, the lumen scene is accurately capturing the warm lighting bouncing around the scene, as well as the area opposite the wall. Even though it’s approximate, the lighting is roughly matching the GI.

Now, this is the reflections-only view with standalone RT reflections. Everything is really, really blue, because standalone RT actually has no idea the GI even exists- it just has direct lighting information, and the skylight. So any area in shadow just has flat ambient lighting, like the roof above the blue display that’s just flat grey.

The point of me saying all of this isn’t to criticize anyone who prefers standalone RT over lumen reflections- there are totally legitimate reasons to like the ‘cleaner’ RT reflections over the messier surface cache. But standalone RT doesn’t possess features that lumen does not- the cleaner look comes from a lack of features. And I think it’s good to have this knowledge about the differences in the open so devs and artists can make informed decisions.

jblackwell · August 6, 2023, 1:03am

All that said, I can’t wait for real-time path-tracing. Even with lumen on maximum settings, there are compromises inherit to the architecture that mean is simply can’t be quite as good. It’s amazing that developers can work with dynamic GI and reflections now, but I think the beautiful thing about this industry is that it is always evolving. And with SIGGRAPH in a week, and Intel presenting a paper on how to scale real-time path-tracing down to iGPUs, there’s a chance that real-time PT can be a reality for many people sooner than expected.

TheKJ · August 6, 2023, 6:38am

Yeah, I’ve flipped that on and messes around with the settings and didn’t really see any difference in performance(Not sure if they are even replacing VSMs, doesn’t look like it).

jblackwell · August 6, 2023, 4:30pm

I am really, really excited for what their research could do to the industry. For all the complaints about Moore’s law dying, there is still an astonishing ability to innovate beyond just throwing more processing power at something. And given intel’s shockingly good RT performance on relatively inexpensive GPUs, I feel like they could actually deliver on some form of real-time path-tracing for a wider hardware variety.

Of course, Intel’s GPUs have been architected from the ground-up for next-gen features like RT. If you remove focus on legacy games and optimize specifically for a given task, I’m not too surprised they’re seeing this performance. But even if Intel and Nvidia can do Real-time PT, AMD’s ultimately going to be the real question for widespread adoption. If it can’t scale to a series S, there’ll be hard limits on how readily the technology gets adopted.

The awesome thing that technology like RTXDI and nanite present is that you don’t have to render and shade everything-just exactly what the scene needs/what’s stochastically relevent. If Intel can get their Nanite for RT down to game-ready speeds, if we get improved denoisers, and if it can be implemented on AMD hardware, we could really see an industry shift.

Rawalanche · August 6, 2023, 6:12pm

I think realtime PT is still some years away, but I can’t wait. Not only because of better quality but mainly because of feature deprecations. Unreal’s Actors, mainly lights, but also meshes are completely overloaded with many dozens of checkboxes to tweak individual parameters of many different “fake” components of rasterization, the configuration of which is highly circumstantial. All sorts of culling distances, biases, offsets, etc…

Offline path tracers don’t only have an edge in terms of quality, but also a huge advantage of usability, because scene objects such as light sources have only few necessary physically based parameters.

While Lumen and VSMs are great and scale across wide range of hardware, I hope they are just intermediate step before realtime path tracing with denoising is feasible at high framerates.

Then, we can start shoveling all the wonky checkboxes and value sliders into the deprecation trash bin by dozens

jblackwell · August 7, 2023, 12:23am

I can’t argue with that, there are an absolutely dizzying number of rendering-relevant checkboxes and configuration details and performance levers, and that’s leaving out the physics, networking, blueprint, and source-control parameters. The sheer number of permutations is absolutely staggering. And compared to offline renderers (Arnold, Cycles, etc), you do have massive numbers of settings.

Unfortunately, I do think some (but certainly not all) of the wonky checkboxes may be here to stay, especially with real-time path-tracing. If you open up Portal RTX’s developer options menu, you see hundreds of different variables, controlling everything from the SPP of specific effects, to the denoiser kernel, to culling distances for geo and more. To make real-time PT a reality for people is going to require scalability, no matter how much better the algorithms get. That could be resolution, # of bounces, whether to support complex features like caustics, etc. At the very least though, I believe you’re right: we’ll have way fewer settings to tune than we did in the raster path.

Rawalanche · August 7, 2023, 1:40pm

I think realtime PT, if done right, will not have many performance knobs, but will still be very scalable.

This is a huge post so I've compacted it to not bloat the thread. Click if you want to read 🥹

It was that way with offline renderers as well. There were oldschool ones, like Vray, Mental Ray, and then Corona Renderer rolled over them by having fraction of settings, yet delivering better quality as well as performance. Turns out average users are extremely bad at balancing sampling parameters compared to heuristics.

I think that realtime game path tracer, if done with good care for UX, will have almost no parameters, and scalability will be matter of how heavily you trace, denoise and upscale. Let me explain:

Performance of non-branched path tracers (Cycles being great example) is usually not affected much by the amount of bounces, maybe with the exception of deep refractions, but for game purposes, I’d expect them to default to 2-3 bounces at most. With non-branched path tracers, individual sampling settings per individual light, material or volume are thing of the past for the most part.

I imagine next gen realtime path tracer to simply have a frame time budget you can define as developer (and expose as in-game option to player) which defines how much time can GPU spend casting rays each frame. So if you set it to let’s say 16ms, the GPU will start progressively path tracing the scene until this time limit, and then it denoises and upscales.

The denoising and upscaling will always manifest as some loss of accuracy. You will see smudginess of small sharp shadows, little detail loss on textures, glossy reflections not being as crisp, thin shapes in distance getting lost or flickering a little, etc… and as both developer, and also a player, you will be able to fine tune the trade off between how high framerate you get in your game (responsiveness) vs how smudgy and blurry your game looks.

To me, this sounds far better than the current option jungles most rasterized 3D games have, where you can configure them to look really hideous and at the same very far off of how developers intended them to look like.

This will be also main means of scalability:
I am thinking several years into the future, so I expect by then all mainstream GPUs having some ray tracing capability already. The very low end PCs would for example be able to squeeze out only 8-12 rays per pixel at 720p, which would be aggressively denoised to make up for the very noisy input and then upscaled to 1080p.

So the people on these low end PCs would play a game which looks as good as the Max/Ultra settings, meaning all the shadows, light bouncing, reflections and so on is there, except it would look a bit like smudgy youtube compressed 1080p video (as opposed to source quality) and running at about 30-50 FPS.

On the other hand people with cutting edge hardware would run the game at the exact same detail level, but at 4k, everything super crisp as if you had source footage from a RED camera, and at 144+FPS framerates. As both denoising and upscaling would need to be way less aggressive.

This is how I imagine the future. There’s really no reason to make it complicated. I really think so because for example Cycles in Blender is quite clean and elegant, without many technical knobs, yet there’s nearly no artistic flexibility missing compared to Unreal’s rasterizer.

ZacD · August 7, 2023, 1:51pm

I’m hoping we’ll get AI to auto tune realtime PT per scene/project. Have it compare to ground truth and tune some nobs to increase performance and minimize noticeable artifacts.

Rawalanche · August 7, 2023, 1:56pm

Yes, that’s also possible. The part of the reason of my optimism are rapid advances in AI. I expect the denoiser and upscalers to improve rapidly. Then maybe we get even some neural light transport, as opposed to path tracing, which will simply infer the scene lighting and reflection based on geometry and light sources location, rather than brute forcing towards it. And of course as you said, the possibility of AI fine tuning the samples spending based on scene analysis is also likely

jblackwell · August 8, 2023, 3:10pm

I didn’t even know you could compact posts like that. Did you do that with ‘hide details?’

Thoughts

I’m inclined to agree with you. What’s exciting about path-tracing is that scalability can exist in a much more targeted way- you don’t have to cull broad swaths of geometry or entire categories of effects, but you could reduce the resolution of certain effects, trade off between noise and ghosting. One of the SIGGRAPH papers digs into a developer’s implementation of VRS for ray-tracing, and that could be a very interesting idea- culling rays where it isn’t noticed, and supersampling the variance.

Of course, rendering isn’t the be-all-end-all of performance. Gameplay logic, AI, animation, VFX, physics, and network synchronization all need to be running well to get a game performing, but the future could look very bright for rendering.

jblackwell · August 8, 2023, 4:18pm

Also, while I assume that the lumen devs have both A. Probably already seen/are thinking about this, and B. are likely at SIGGRAPH at the moment, I did want to share this resource on Ray-Aligned Occupancy Map Arrays (ROMA) (beware technical jargon):

(https://onlinelibrary.wiley.com/doi/full/10.1111/cgf.14882)

This is a link to a paper on Ray-aligned Occupancy Map Arrays (ROMA). Without getting too deep into it, they have created a novel tracing method that can resolve visibility (and extended to GI) in constant time. The unique bit of information they offer is that ROMA can support dynamic objects far better than distance field generation (under their schema at least) which makes me wonder if ROMA could be patched into software lumen as a way of supporting dynamic and deforming objects such as characters.

The single biggest limitation is quite similar to the limitations other voxel scene representations have- low resolution means that specular lighting/hard shadows aren’t really an option if you want good quality. Nonetheless, I feel like an option to support high-speed dynamic objects would be an asset to lumen SWRT, as in its’ current state there is only screen-space tracing, which has many large failure cases.

Yaeko · August 8, 2023, 6:25pm

Soo much negativity around the new features, just because they arent meant for current gen.

Nanite is a godsend and can save weeks of tedious optimization work, while allowing you to use hundreds of thousands of meshes while paying 0 fps for them. (I tried with 100k, in addition to what was already there, zero difference in performance. (That was really a mind-blowing moment… seeing a 0 fps impact.))

Nanite scales on a level that was almost unachievable previously, while saving a lot of time… and now it works with Plants too… cant wait for the day it will work with translucent materials. (That would make me very happy )

Lumen is awesome for what it is, even with its limitations. Just imagine for a moment what we got: (relatively) Dynamic GI that runs just fine on last gen cards, while still achieving 30 fps on older midrange cards. And it doesnt even care what brand is written on your Graphics Card… AMD, Intel, nVidia, etc. - it just works.

No baking, you can just turn it on/off at runtime, adjust it at runtime etc… for decades we wanted something like Lumen, now we got it.

==========

Yes, yes, there are bugs, issues and missing parts - but think back a moment: just a year ago, I was poking the Lumen Team and Epic about the Foliage in Lumen, and it (fixes) barely made it into 5.0, just a day or so before the deadline was. (Its either in this thread, or the previous one)

Both of these features (Nanite and Lumen) are still very “young”, and they will need a lot more time and effort to support everything properly. (Afaik, Custom Stencils didnt work with Nanite until 5.2 for example…)

The “performance” side of things is to be expected, since Nanite trades GPU time for CPU time (in the end, it comes down to this), and Lumen does have a base-cost that you wont get rid of in terms of things that need to be calculated, thats just how it works - for everything else we have baked light.

The perceived performance issues on the stagnating “low end” (since 4050 is now 4060…) cards will disappear over time, as new hardware arrives - which is probably why Epic isnt investing too much effort into “scraping the barrel for fractions of a milisecond”.

The performance Delta between the Low and High-End cards is just bonkers nowadays… somewhere between 3x and 4x, depending on what you run, and that isnt even the fastest card nvidia could have made, there still is a larger chip that could have become the 4090 Ti if AMD had decided to also push its cards beyond the current power draw. (go and look, the coolers exist… ridiculous thing, uses actual copper plates to reroute the power input . https://cdn.videocardz.com/1/2023/07/TITAN-ADA-RTX-4090TI-COOLER-4-1.jpg )

What we currently see with “bad performance” of Lumen is completely normal and always happened when technology was pushed. Remember Crysis?

The sole reason we could run “modern” games on something like a 3060 at 4K was the fact that we didnt have had “next gen” games at that point. Whenever a console generation changes, a few years later we see system requirements go up a lot - which is what we see right now. (Remember how everyone was like: 8GB VRAM is enough, just a year ago?.. yeah, and suddenly it isnt anymore, especially not if you try to run RT - and it bites nvidias butt right now. (Meanwhile I am sitting here laughing, with my 16 GB 6900XT having no issues whatsoever.))

==========

Wall of Text ends here

I also recommend to not make games (right now) that require Lumen being enabled for gameplay reasons, it just wont work out. (Been there, considered that, calculated it, shelved it.)

TheKJ · August 8, 2023, 8:58pm

Deleted.

Yaeko · August 8, 2023, 10:18pm

I wasnt specifically aiming at you - I just think different about the new features, since they made my life a loooooooooot easier, after they initially made my life hell. (EA1/2 trauma…)

Which actually is fine, since a 2080 Ti is faster than a 3060 and equal to a 3070 or faster, depending on the game.

They could have just used a 1080 Ti too, its still faster than a 3060, despite having the “DX12 issues”.

In the end, all of this is just numbers.

Fortnite runs at less than 900p at times on PS5 and Series X… then they slap TSR onto it and call it “4K” - as eurogamer cites Epic Games.

Idk… I wouldnt call “< 900p upscaled to 4k, at 60 fps”… “working fine on current gen”, especially given how simple Fortnite is in terms of visuals - which actually gives it a big advantage over more complex games.

I can only imagine what 900p upscaled to 4k looks like in motion… (I wouldnt try to use TSR at less than 80% of 1440p, to get to 1440p - it just doesnt look good in motion.)

Like, literally, Epic has somehow “made it work” but barely… which is unfeasable for anything more complex - and if a feature isnt really usable very well on current hardware, then we cant use it much → wait for next gen.

But I mean, if “it runs at X framerate, regardless of how it looks” is the baseline, then I can make it run on a smartfridge at 240p with TSR upscaled to 8K. (And say: see, its current gen viable. xD)

ZacD · August 9, 2023, 12:39pm

Native resolution is already dead, consoles wanting to output 4k and DLSS2 proving upscaling can look good enough killed native. The industry is moving on, sorry.

Yaeko · August 9, 2023, 3:15pm

TAA etc.

I mean, what are ya going to do?

No AA = everything is flickering
TAA = everything is blurry
TSR (at 80% res scale) = best visual result and you get some performance on top, but still not perfect.

Idk… TSR 80% runs circles around TAA and gives extra Performance (I just recently tested that) - and my straight lines are actually very much a nightmare for AA:
Unbenannt

Lumen itself can also introduce “noise” to the scene, especially on not fully smooth materials, and that isnt even the worst “noise” it creates, since you will see how it updates in the distance.

We will see where Lumen goes in the next few engine versions, I am just glad we have it.

Meh, did anyone see someone from the Lumen Team in this thread since 5.1? (or have they been pushed away?)

I really liked how we could give Feedback and Bugs to them directly…

Upscaling

Its that weird race for “better graphics” while the hardware isnt ready for it yet. Remember previous console generation, where things ran below 720p at times, and every game practically was blurry?

Remember nvidias push for Raytracing? Guess what cards now have issues with it? exactly, the 2XXX cards, and any 3/4XXX card with 8GB vram, since it needs lots of that - it hurts them so much so, that even the (worse at raytracing) AMD cards can overtake them in some games with RT just because they are equipped better. (But hey, at least we now have RTX overdrive in Cyberpunk, that their own cards cant run without DLSS3 fake-frames etc…)

Its just the time we live in -

jblackwell · August 9, 2023, 3:26pm

I just want to say that I appreciate you all keeping things civil on this channel, genuinely. It’s good to have an environment where people can air their thoughts and frustrations about the technology and still be respected for it.

That is a bit of a complex question, and I’d need to pull up ProfileGPU to be sure, but I feel like I should mention that the new forms of nanite don’t always have a flat render time. I got to speak about this with a connection who knows Brian Karis, and he described that Nanite’s programmable raster feature is not guaranteed to cost a flat amount of time. Whenever you have WPO-enabled materials like foliage for example, there is a cost to evaluating that logic that (as I understand) does scale with total triangles rendered. Add to that that foliage is a bad occluder, and you also have overdraw to contend with that’s bottlenecking your performance. The same is true for masked materials, and a handful of other cases. The Electric Dreams demo has absurd nanite costs because it’s filled with extremely high-poly geometry that’s constantly deforming.
Never mind the overdraw that comes as a result of stacks of very thin geometry that can’t use preserve area- again, massive overdraw costs. What I’m essentially saying is that Nanite should be a relatively flat cost in cases of opaque geometry that occludes well-otherwise, it will vary, and the same sort of performance optimization we’re all used to will be called for.

jblackwell · August 9, 2023, 3:31pm

I saw Krzysztof.N on either here or an ancillary channel talking about lightmass and lumen reflections, but that’s about it.

To be honest, I’m not too surprised. I could definitely see them being on here more in the early days, but this channel is labeled ‘Lumen GI and Reflections feedback thread.’ It started off with the more narrow scope of identifying needed features and fixing bugs, but now it’s become a channel that has everything from UE newbies looking for help, to people complaining/identifying technical frustrations, to occasionally completely non-lumen related discussions about game and light transport/what the next-gen looks like.

I think it’s partly because even though the feedback thread is what they created, people needed a lumen general thread for all the things that weren’t directly dev feedback. I actually wish they’d create a new standing channel dedicated specifically to discrete bug identification and feature requests, so they wouldn’t have to sort through this very long channel history to find what matters to them.

Also, we generally don’t @ them too often, which I partly assume is a courtesy because they are very busy, and there aren’t really any killer feature requests I can think of that they don’t know about (reflection denoising has been on the request list for what…a year?)

Yaeko · August 9, 2023, 5:24pm

Hmm, then I will just drop my wishlist and what I know that is still not working:

per Instance Custom Data (just black in Lumen scene, the last time I checked.)
an option to not recalculate parts of the level that didnt change (In my game, like 98% of the area lumen deals with is static… so technically it wouldnt need to recalculate that all the time.)
Distance GI (it was faked in the PS5 trailer with the valley, but we still dont have it in 5.3 afaik.)
if possible, integrade HWRT reflections into Software Lumen (if there are benefits to performance, which I assume due to dedicated RT hw that otherwise is unused. And of course, the obvious visual improvements for reflections that would come from this.)
a function to make “holes” into the Distance fields that Lumen uses, since we still have issues with overocclusion and dense plants/bushes. (I posted somewhat of a workaround in this thread, but its annoying to do.)
move SSGI Data into Lumens permanent Data, so that it isnt regenerated just because I look away, and causing very visible artifacts for large lights. (in general, prevent lumen from “losing” information all the time, even if it costs 100MB VRAM… make it an option. It really doesnt matter if my game needs 3.5 or 3.6 gb vram at 1440p.)

Thats just the things I have in mind right now, that are still unfinished, missing, or “buggy” in 5.2/3.

I really try to avoid @ ing them, since I dont want them to “get annoyed” and never come back. (which is also why I was concerned about the negativity here over the last weeks )

jblackwell · August 9, 2023, 11:53pm

Could you elaborate a bit on what you mean, please?

In many ways, this is what lumen does already, the core of the surface cache is to stably accumulate lighting without needing to trace buckets of new rays every frame. Would you like an option to have it do so more aggressively, and maintain a bigger surface cache over a longer range?

That I can directly speak to actually. In Land Of Nanite, the distant GI was achieved via a reflective shadow map derived from the directional light. It wasn’t exactly faked, but it was only a single bounce of indirect light and only from the directional light. I did a fair amount of testing on it, and it…wasn’t that great for what it cost. Are you just talking about SWRT, or do you also have issues with HWRT’s far-field?

I was under the impression that this was already a working feature. I’ve seen it work very well in levels that I’ve tested (although I don’t remember how exactly it was enabled). Foliage and thin geometry seemed to let light through to an extent, which avoided the incredible over-darkening otherwise experienced with SWRT.

My question for that might be how to move SSGI data into the surface cache (I believe that’s what you’re referencing)?
Screen traces from lumen exist to cover the mismatch between the Gbuffer and the lumen scene, because SWRT’s caching is inheritly inaccurate compared to the Gbuffer. Caching screen traces would require some way of representing their lighting in world-space, and as Krzysztof.N mentioned in a post a while ago, one of their big limitations to improving lumen ATM is they can’t figure out how to make the surface cache better represent the Gbuffer scene.
In addition, the 2022 SIGGRAPH presentation on lumen mentions that the team tried not to over-cache screen traces because they liked having small-scale lighting be much quicker to respond to change than large scale lighting could be.

These were just my thoughts on the requests you’re looking to have fulfilled for lumen. I’d love to hear your thoughts, and just get to better understand what people are looking to get out of the system.