Lumen GI and Reflections feedback thread

[jblackwell]

You have to actually tell the lights to use DF shadows on a per-light basis unfortunately:

The move that Epic suggested for UE4 was to use shadowmaps in the near-field because they can handle skinned meshes and what have you, and use DF shadows in the far-field because they are cheap and can have incredible range. Because VSMs continue to be inordinately expensive, I’m suprised they don’t have an option to use near-field VSMs and DF shadows in the far-field. Then again, with all the caching and connections to nanite, I can’t actually fathom the performance levers VSMs are based around.

[jblackwell]

Considering how hard Nvidia (and now Intel) are pushing real-time path-tracing, I don’t think it’ll be too long at all, for PCs at least. Consoles are a mystery, but a handful of breakthroughs can change the game (EG Nanite).

In regards to your frustration with reflections, I agree that lumen scene reflections can look messy and somewhat illusion-breaking, but I’m not sure standalone RT is actually better in this case, and here’s what makes me think that:

UE Content Examples, 5.3 preview. Lumen GI and reflections with hit lighting (two bounces), everything maxed out. Note the reflection of the white wall on the black floor, and the white wall is indirectly lit.

Same scene, identical settings, minus standalone RT reflections enabled, 2SPP, two bounces, but with a roughness cutoff of 1 (so almost GI at this point).

Note how strangely blue the standalone RT reflections are. They’re almost acting like the wall doesn’t exist from a lighting perspective, and the metal bar is very blue even though it has no direct view of the sky.

If you go into the reflections only viewmode, it’s pretty easy to see why (sidenote: lumen only traces multiple bounces of reflections for surfaces that are smooth enough; it otherwise uses the lumen scene). Even though the lighting is blotchy, the lumen scene is accurately capturing the warm lighting bouncing around the scene, as well as the area opposite the wall. Even though it’s approximate, the lighting is roughly matching the GI.

Now, this is the reflections-only view with standalone RT reflections. Everything is really, really blue, because standalone RT actually has no idea the GI even exists- it just has direct lighting information, and the skylight. So any area in shadow just has flat ambient lighting, like the roof above the blue display that’s just flat grey.

The point of me saying all of this isn’t to criticize anyone who prefers standalone RT over lumen reflections- there are totally legitimate reasons to like the ‘cleaner’ RT reflections over the messier surface cache. But standalone RT doesn’t possess features that lumen does not- the cleaner look comes from a lack of features. And I think it’s good to have this knowledge about the differences in the open so devs and artists can make informed decisions.

[jblackwell]

All that said, I can’t wait for real-time path-tracing. Even with lumen on maximum settings, there are compromises inherit to the architecture that mean is simply can’t be quite as good. It’s amazing that developers can work with dynamic GI and reflections now, but I think the beautiful thing about this industry is that it is always evolving. And with SIGGRAPH in a week, and Intel presenting a paper on how to scale real-time path-tracing down to iGPUs, there’s a chance that real-time PT can be a reality for many people sooner than expected.

[TheKJ]

Holy SMOKES BRO! I just read up on those papers. igpus!? Imagine the cheap ms render time on actual graphic cards! I am shaking with excitement for this. Whatever it is, we NEED it in UE5. Now I’m having new performance hopes about my game.

I won’t get my hopes too high but Intel consistently blows me away with their innovations.

Thanks for letting me know about that. I cannot wait for the presentations.

[TheKJ]

Yeah, I’ve flipped that on and messes around with the settings and didn’t really see any difference in performance(Not sure if they are even replacing VSMs, doesn’t look like it).

[jblackwell]

I am really, really excited for what their research could do to the industry. For all the complaints about Moore’s law dying, there is still an astonishing ability to innovate beyond just throwing more processing power at something. And given intel’s shockingly good RT performance on relatively inexpensive GPUs, I feel like they could actually deliver on some form of real-time path-tracing for a wider hardware variety.

Of course, Intel’s GPUs have been architected from the ground-up for next-gen features like RT. If you remove focus on legacy games and optimize specifically for a given task, I’m not too surprised they’re seeing this performance. But even if Intel and Nvidia can do Real-time PT, AMD’s ultimately going to be the real question for widespread adoption. If it can’t scale to a series S, there’ll be hard limits on how readily the technology gets adopted.

The awesome thing that technology like RTXDI and nanite present is that you don’t have to render and shade everything-just exactly what the scene needs/what’s stochastically relevent. If Intel can get their Nanite for RT down to game-ready speeds, if we get improved denoisers, and if it can be implemented on AMD hardware, we could really see an industry shift.

[Rawalanche]

I think realtime PT is still some years away, but I can’t wait. Not only because of better quality but mainly because of feature deprecations. Unreal’s Actors, mainly lights, but also meshes are completely overloaded with many dozens of checkboxes to tweak individual parameters of many different “fake” components of rasterization, the configuration of which is highly circumstantial. All sorts of culling distances, biases, offsets, etc…

Offline path tracers don’t only have an edge in terms of quality, but also a huge advantage of usability, because scene objects such as light sources have only few necessary physically based parameters.

While Lumen and VSMs are great and scale across wide range of hardware, I hope they are just intermediate step before realtime path tracing with denoising is feasible at high framerates.

Then, we can start shoveling all the wonky checkboxes and value sliders into the deprecation trash bin by dozens

[jblackwell]

I can’t argue with that, there are an absolutely dizzying number of rendering-relevant checkboxes and configuration details and performance levers, and that’s leaving out the physics, networking, blueprint, and source-control parameters. The sheer number of permutations is absolutely staggering. And compared to offline renderers (Arnold, Cycles, etc), you do have massive numbers of settings.

Unfortunately, I do think some (but certainly not all) of the wonky checkboxes may be here to stay, especially with real-time path-tracing. If you open up Portal RTX’s developer options menu, you see hundreds of different variables, controlling everything from the SPP of specific effects, to the denoiser kernel, to culling distances for geo and more. To make real-time PT a reality for people is going to require scalability, no matter how much better the algorithms get. That could be resolution, # of bounces, whether to support complex features like caustics, etc. At the very least though, I believe you’re right: we’ll have way fewer settings to tune than we did in the raster path.

[Rawalanche]

I think realtime PT, if done right, will not have many performance knobs, but will still be very scalable.

This is a huge post so I've compacted it to not bloat the thread. Click if you want to read 🥹

It was that way with offline renderers as well. There were oldschool ones, like Vray, Mental Ray, and then Corona Renderer rolled over them by having fraction of settings, yet delivering better quality as well as performance. Turns out average users are extremely bad at balancing sampling parameters compared to heuristics.

I think that realtime game path tracer, if done with good care for UX, will have almost no parameters, and scalability will be matter of how heavily you trace, denoise and upscale. Let me explain:

Performance of non-branched path tracers (Cycles being great example) is usually not affected much by the amount of bounces, maybe with the exception of deep refractions, but for game purposes, I’d expect them to default to 2-3 bounces at most. With non-branched path tracers, individual sampling settings per individual light, material or volume are thing of the past for the most part.

I imagine next gen realtime path tracer to simply have a frame time budget you can define as developer (and expose as in-game option to player) which defines how much time can GPU spend casting rays each frame. So if you set it to let’s say 16ms, the GPU will start progressively path tracing the scene until this time limit, and then it denoises and upscales.

The denoising and upscaling will always manifest as some loss of accuracy. You will see smudginess of small sharp shadows, little detail loss on textures, glossy reflections not being as crisp, thin shapes in distance getting lost or flickering a little, etc… and as both developer, and also a player, you will be able to fine tune the trade off between how high framerate you get in your game (responsiveness) vs how smudgy and blurry your game looks.

To me, this sounds far better than the current option jungles most rasterized 3D games have, where you can configure them to look really hideous and at the same very far off of how developers intended them to look like.

This will be also main means of scalability:
I am thinking several years into the future, so I expect by then all mainstream GPUs having some ray tracing capability already. The very low end PCs would for example be able to squeeze out only 8-12 rays per pixel at 720p, which would be aggressively denoised to make up for the very noisy input and then upscaled to 1080p.

So the people on these low end PCs would play a game which looks as good as the Max/Ultra settings, meaning all the shadows, light bouncing, reflections and so on is there, except it would look a bit like smudgy youtube compressed 1080p video (as opposed to source quality) and running at about 30-50 FPS.

On the other hand people with cutting edge hardware would run the game at the exact same detail level, but at 4k, everything super crisp as if you had source footage from a RED camera, and at 144+FPS framerates. As both denoising and upscaling would need to be way less aggressive.

This is how I imagine the future. There’s really no reason to make it complicated. I really think so because for example Cycles in Blender is quite clean and elegant, without many technical knobs, yet there’s nearly no artistic flexibility missing compared to Unreal’s rasterizer.

[ZacD]

I’m hoping we’ll get AI to auto tune realtime PT per scene/project. Have it compare to ground truth and tune some nobs to increase performance and minimize noticeable artifacts.

[Rawalanche]

Yes, that’s also possible. The part of the reason of my optimism are rapid advances in AI. I expect the denoiser and upscalers to improve rapidly. Then maybe we get even some neural light transport, as opposed to path tracing, which will simply infer the scene lighting and reflection based on geometry and light sources location, rather than brute forcing towards it. And of course as you said, the possibility of AI fine tuning the samples spending based on scene analysis is also likely

[TheKJ]

I hate upscalers, even DLSS. I also hate almost all Temporal solutions except my tweaked TAAU and I still wouldn’t use it if Lumen wasn’t dependent on temporal solutions to hide its horrid artifacts (I would use high quality SMAA or slightly sharpened FXAA or crispy no AA).

These stupid looking, blurry as crap temporal methods are ruin raytracing and gaming for me.

Summary

Make no mistake, I believe the 30 series and any AMD/Intel equivalent competitors are the current gen for gaming. 1080p for 3060, 1440 for 3070, 4k on 3080, Plugin competitor’s vendors, etc. I think 30fps is unacceptable trash in this day and age and 60fps is a gold standard for a reason.

Take an amazing looking pre-dlss-days game. The 3060(a 1080p card) could run it at 4k 60fps and 144+fps at 1080p. This is because we had more innovative productions and development features.

Now games can barely run 50-60fps at 1080p on the 3060. I’m losing more than half of my performance for next gen games that barly look any better(exclude GI games)? WTF?!
This new trend in non-optimal, inefficient computing in games came from so called “innovations”(mistakes) like dlss and heavy procedural content design feautures. Lazy developers ask the majority of people(1080p players) to upscaling from an extremely crappy resolution like 540p or 720p.

An example of heavy procedural content design would be something like GI which is well worth it since it actually looks amazing in a dynamic scene. GI isn’t just easier/more simple way to light digital scenes.
But a bigger one is Nanite.
“No work being put into LODs and import extremely high poly meshes”

Okay but Nanite clearly doesn’t offer better performance than optimized, maybe lower poly mesh with LODS. You will never eliminate “pop in” because of world partitioning, shadow pop, and foliage culling. (All still present in FN 5.1). Importing those meshes would be cool but If I could trade in more traditinal (high) performance for pop in, I would.

Higher resolution textures are far cheaper than Nanite meshes and can still provide photorealistic results to regular lower poly meshes. The best way to optimize with 8k textures is to make sure it’s kept extremely small so you’re procedurally tiling the textures/UV correctly on the mesh to save VRAM.

Another thing about lower LODs with higher quality textures is they don’t break shadows like usual Nanite meshes so you can use regular shadows instead of VSMs. When you chop out Nanite, you can easily get to that native resolution your GPU card targets at a 60fps gold standard with great, photorealistic looking visuals with something like heavily scaled down Lumen or hopefully Intel’s new path tracing solution

Lumen can’t even run on integrated graphics. And any solution that can run on igpus will be extremely cheap on GPU cards so no blurry or ghosting upscaling should have to be used with that PT solution.

We need to move AWAY from upscalers. I get rendering smarter not harder which is what temporal solutions offer, but not if it looks like blurry **** . One step forward, two steps back.

Be dynamic, look photorealistic, compute fast, and look stable. That is real computer innovation.

I’m not suprised Intel is taking the first step in that future.

[jblackwell]

I didn’t even know you could compact posts like that. Did you do that with ‘hide details?’

Thoughts

I’m inclined to agree with you. What’s exciting about path-tracing is that scalability can exist in a much more targeted way- you don’t have to cull broad swaths of geometry or entire categories of effects, but you could reduce the resolution of certain effects, trade off between noise and ghosting. One of the SIGGRAPH papers digs into a developer’s implementation of VRS for ray-tracing, and that could be a very interesting idea- culling rays where it isn’t noticed, and supersampling the variance.

Of course, rendering isn’t the be-all-end-all of performance. Gameplay logic, AI, animation, VFX, physics, and network synchronization all need to be running well to get a game performing, but the future could look very bright for rendering.

[jblackwell]

Also, while I assume that the lumen devs have both A. Probably already seen/are thinking about this, and B. are likely at SIGGRAPH at the moment, I did want to share this resource on Ray-Aligned Occupancy Map Arrays (ROMA) (beware technical jargon):

(https://onlinelibrary.wiley.com/doi/full/10.1111/cgf.14882)

This is a link to a paper on Ray-aligned Occupancy Map Arrays (ROMA). Without getting too deep into it, they have created a novel tracing method that can resolve visibility (and extended to GI) in constant time. The unique bit of information they offer is that ROMA can support dynamic objects far better than distance field generation (under their schema at least) which makes me wonder if ROMA could be patched into software lumen as a way of supporting dynamic and deforming objects such as characters.

The single biggest limitation is quite similar to the limitations other voxel scene representations have- low resolution means that specular lighting/hard shadows aren’t really an option if you want good quality. Nonetheless, I feel like an option to support high-speed dynamic objects would be an asset to lumen SWRT, as in its’ current state there is only screen-space tracing, which has many large failure cases.

[Yaeko]

Soo much negativity around the new features, just because they arent meant for current gen.

Nanite is a godsend and can save weeks of tedious optimization work, while allowing you to use hundreds of thousands of meshes while paying 0 fps for them. (I tried with 100k, in addition to what was already there, zero difference in performance. (That was really a mind-blowing moment… seeing a 0 fps impact.))

Nanite scales on a level that was almost unachievable previously, while saving a lot of time… and now it works with Plants too… cant wait for the day it will work with translucent materials. (That would make me very happy )

Lumen is awesome for what it is, even with its limitations. Just imagine for a moment what we got: (relatively) Dynamic GI that runs just fine on last gen cards, while still achieving 30 fps on older midrange cards. And it doesnt even care what brand is written on your Graphics Card… AMD, Intel, nVidia, etc. - it just works.

No baking, you can just turn it on/off at runtime, adjust it at runtime etc… for decades we wanted something like Lumen, now we got it.

==========

Yes, yes, there are bugs, issues and missing parts - but think back a moment: just a year ago, I was poking the Lumen Team and Epic about the Foliage in Lumen, and it (fixes) barely made it into 5.0, just a day or so before the deadline was. (Its either in this thread, or the previous one)

Both of these features (Nanite and Lumen) are still very “young”, and they will need a lot more time and effort to support everything properly. (Afaik, Custom Stencils didnt work with Nanite until 5.2 for example…)

The “performance” side of things is to be expected, since Nanite trades GPU time for CPU time (in the end, it comes down to this), and Lumen does have a base-cost that you wont get rid of in terms of things that need to be calculated, thats just how it works - for everything else we have baked light.

The perceived performance issues on the stagnating “low end” (since 4050 is now 4060…) cards will disappear over time, as new hardware arrives - which is probably why Epic isnt investing too much effort into “scraping the barrel for fractions of a milisecond”.

The performance Delta between the Low and High-End cards is just bonkers nowadays… somewhere between 3x and 4x, depending on what you run, and that isnt even the fastest card nvidia could have made, there still is a larger chip that could have become the 4090 Ti if AMD had decided to also push its cards beyond the current power draw. (go and look, the coolers exist… ridiculous thing, uses actual copper plates to reroute the power input . https://cdn.videocardz.com/1/2023/07/TITAN-ADA-RTX-4090TI-COOLER-4-1.jpg )

What we currently see with “bad performance” of Lumen is completely normal and always happened when technology was pushed. Remember Crysis?

The sole reason we could run “modern” games on something like a 3060 at 4K was the fact that we didnt have had “next gen” games at that point. Whenever a console generation changes, a few years later we see system requirements go up a lot - which is what we see right now. (Remember how everyone was like: 8GB VRAM is enough, just a year ago?.. yeah, and suddenly it isnt anymore, especially not if you try to run RT - and it bites nvidias butt right now. (Meanwhile I am sitting here laughing, with my 16 GB 6900XT having no issues whatsoever.))

==========

Wall of Text ends here

I also recommend to not make games (right now) that require Lumen being enabled for gameplay reasons, it just wont work out. (Been there, considered that, calculated it, shelved it.)

[TheKJ]

I didn’t even know you could compact posts like that. Did you do that with ‘hide details?’ -jblackwell

Yeah, it’s going to be helpful.

I don’t think you read my post @Yaeko

Soo much negativity around the new features, just because they arent meant for current gen.

That is not true at all. It is meant for current gen (The resolution to GPU iteration chart to my post ). Who is actually hell is spreading this LIE. You are the 20th person I’ve had to correct on this.

There several papers from Epic Games referring to PS5 and Series X for millisecond timings. And real kicker is the papers, when referring to GPU rasterization, they referred to a 2080 ti. NOT EVEN A 30 series.

I tried with 100k, in addition to what was already there, zero difference in performance. (That was really a mind-blowing moment… seeing a 0 fps impact.

I already stated the problem with Nanite in my last post. If you are using a higher GPU iteration for its recommended resolution, you may not see a difference in fps.

Seeing 0fps impact? What was the millisecond timing on the Nanite base pass? Nanites biggest weakness is how many different meshes are in the scene(City Sample).

can save weeks of tedious optimization work

No, you are trading in those weeks for a base cost of 7 to 8 ms (on the resolution to gpu chart). And depending on how detailed they are, you have to use VSM.

Fortnite’s 5.1 Lumen/Nanite performance with barebones optimization(meduim VSMs, High Lumen, no AA, Post Process) at native 1080p only runs 51 fps on my 3060. Upscaling is NOT a solution.

I get the same performance I get in City Sample in editor with those same FN settings.
The biggest chuck in the GPU scene is Nanite Base pass and 2 other(I forget atm) Nanite related chucks.

6900XT

UE5 is probably using more optimized pipelines(console pipelines) for your card if your drivers expose the 16bit instructions newer AMD cards have (including new consoles).

I also recommend to not make games (right now) that require Lumen.

You can if you don’t depend on Nanite or VSMs. Native recommended resolution(remeber, determined by your GPU) at 60fps(real frames) should always be the goal.
40 series is a joke. the 4070 is just a cheaper 3080 and that’s is. No more generational leaps below that. The benchmarks prove it.

30 series is already a generation ahead of the Consoles.

Consoles on the other hand are Joke. We cannot keep comparing the past to the present, when the present is so unbelievable different. Back then, companies were not shamelessly (Nvidia, sony, mircosoft) giving the middle finger to customers.

[Yaeko]

I wasnt specifically aiming at you - I just think different about the new features, since they made my life a loooooooooot easier, after they initially made my life hell. (EA1/2 trauma…)

And real kicker is the papers, when referring to GPU rasterization, they referred to a 2080 ti. NOT EVEN A 30 series.

Which actually is fine, since a 2080 Ti is faster than a 3060 and equal to a 3070 or faster, depending on the game.

They could have just used a 1080 Ti too, its still faster than a 3060, despite having the “DX12 issues”.

In the end, all of this is just numbers.

You are the 20th person I’ve had to correct on this.

Fortnite runs at less than 900p at times on PS5 and Series X… then they slap TSR onto it and call it “4K” - as eurogamer cites Epic Games.

Idk… I wouldnt call “< 900p upscaled to 4k, at 60 fps”… “working fine on current gen”, especially given how simple Fortnite is in terms of visuals - which actually gives it a big advantage over more complex games.

I can only imagine what 900p upscaled to 4k looks like in motion… (I wouldnt try to use TSR at less than 80% of 1440p, to get to 1440p - it just doesnt look good in motion.)

Like, literally, Epic has somehow “made it work” but barely… which is unfeasable for anything more complex - and if a feature isnt really usable very well on current hardware, then we cant use it much → wait for next gen.

But I mean, if “it runs at X framerate, regardless of how it looks” is the baseline, then I can make it run on a smartfridge at 240p with TSR upscaled to 8K. (And say: see, its current gen viable. xD)

[TheKJ]

When they used the 2080 TI, they were testing rasterization above 1080. I think 1440p? So about a 3070 in power, That’s the target resolution to current gen gpu iteration scale I mentioned earlier.

Expand Post <-

Fortnite runs at less than 900p at times on PS5 and Series X… then they slap TSR onto it and call it “4K” - as eurogamer cites Epic Games.

I imagine so, when a character is overlooking the entire map. I’m pretty sure it’s running around 1080p most of the time. The PS5/Series X is basically current gen 1080p card with a little more rasterization head room for any upscaling method for “4k” output.
Not to mention the optimized pipelines the AMD hardware gets(16bit instruction sets in RDNA gpus).

I wasnt specifically aiming at you

I understand, maybe I miss understood your point. I don’t want to point my finger at you either. But I will point the finger at Epic Games\Engine Devs. They need to work on micro-optimizations or flat-out more scalability to Nanite, VSM’s and Lumen. When it comes to Nanite, it repeatedly shows that is should NOT be enabled on every static opaque object in the scene like the documentation suggest.

I don’t know what the most expensive part of Nanites’ is algorithm, is it the clusters? It is the monitoring of the camera’s positions to provide unnoticeable pop?
Because I would trade in some more visible pop from nanite meshes(As a togglable option) if it meant more performance.

The biggest thing I like about Nanite is the fact that is scales down the meshes in a better looking way than Unreal’s auto LOD maker.
I think there is a good idea and performance to presentation comprise in that statment(Cough, make the auto-lods resemble nanites scale down algorithm, bloody Cough)

The games using Nanite/ VSM speak for themselves.
I don’t want the Engine Devs working with UE5 to get comfortable with upscaling as a solution. Squeeze out every last innovation to optimize and pull 3 or .30ms off the table.

Because 60fps, NATIVE resolution is KING. Just because other people have crappy standards doesn’t mean all developers want to provided crappy results.

Epic also needs to stop relying on TAA since the majority of players can’t use it because of the immense blur vs no AA. Even my tweak TAA couldn’t fix the immense trash look in Tekken 8. That game looks absolutely stunning without any stupid temporal filter. FXAA or none brought out an insane experience of fidelity.

[ZacD]

Native resolution is already dead, consoles wanting to output 4k and DLSS2 proving upscaling can look good enough killed native. The industry is moving on, sorry.

[Yaeko]

TAA etc.

Epic also needs to stop relying on TAA since the majority of players can’t use it because of the immense blur vs no AA.

I mean, what are ya going to do?

No AA = everything is flickering
TAA = everything is blurry
TSR (at 80% res scale) = best visual result and you get some performance on top, but still not perfect.

Idk… TSR 80% runs circles around TAA and gives extra Performance (I just recently tested that) - and my straight lines are actually very much a nightmare for AA:

Lumen itself can also introduce “noise” to the scene, especially on not fully smooth materials, and that isnt even the worst “noise” it creates, since you will see how it updates in the distance.

We will see where Lumen goes in the next few engine versions, I am just glad we have it.

Meh, did anyone see someone from the Lumen Team in this thread since 5.1? (or have they been pushed away?)

I really liked how we could give Feedback and Bugs to them directly…

Upscaling

Just because other people have crappy standards

Its that weird race for “better graphics” while the hardware isnt ready for it yet. Remember previous console generation, where things ran below 720p at times, and every game practically was blurry?

Remember nvidias push for Raytracing? Guess what cards now have issues with it? exactly, the 2XXX cards, and any 3/4XXX card with 8GB vram, since it needs lots of that - it hurts them so much so, that even the (worse at raytracing) AMD cards can overtake them in some games with RT just because they are equipped better. (But hey, at least we now have RTX overdrive in Cyberpunk, that their own cards cant run without DLSS3 fake-frames etc…)

Its just the time we live in -