Localized-IBL implementation

[Chosker]

Coming over from the SVOGI thread and the Physical Lights thread I decided to make a new thread and see if there's any interest on this.
The point is an attempt to repurpose the Reflection Captures into Localized Image-Based Lighting.

The expected result would be a setup of baked ambient lighting (without using Lightmass) that goes beyond the limits of a globally-affecting skylight, that can co-exist in a dynamic lighting setup.

The most important use case would be indoor-outdoor ambient light differentiation, such as going inside a house or cave and not having the skylight affecting it. This is a commonly requested feature that is so far not addressed.

Up to UE4 4.11 there was a hidden feature called DiffuseFromCaptures which apparently was a way to contribute to lighting via reflection captures but it felt very short at it, and was removed anyway.

So I started a proof of concept which turned into a feature



What this does

TL;DR - Reflection Captures (Box & Sphere) are repurposed and used as Image-Based Lighting probes, which provide local ambient light while "blocking out" the skylight - allowing proper interior lighting when using a dynamic skylight.

Some comparison shots:
https://imgsli.com/MTYzMg
https://imgsli.com/MTYzMw
https://imgsli.com/MTYzNA
https://imgsli.com/MTYzNQ

How it looks if you smoothly change the ReflectionCapture's contribution at runtime:





Test on a production-like environment: UT4's awesome Outpost23 map



Outpost23 comparison screenshots here: https://imgur.com/a/EjEJ2zd


And the classic Sponza Atrium scene





How to set up

The engine changes were meant to go into the engine as a Pull Request, -integrated and maintained by Epic- and available for everyone directly from the binary engine. Because who wouldn't like having an additional ambient lighting choice when working with dynamic lights?
Unfortunately the Pull Request was declined by Epic, pretty much coming down to this:

This would add another option that has the potential to confuse artists about what is needed to achieve their visual goals.

While I understand this system is not a great fit for the engine's straightforward "Static / Stationary / Dynamic" mobility concept and the grasp of light-object interaction it produces, there are a number of things that don't fit that view either (i.e. translucency in general, screen-space reflections).

Moving forward the system is available to integrate manually, for which you'll need a custom UE4 build.
The code is available on my GitHub Fork, where I add branches corresponding to the major engine versions (ignore the 'master' and 'release' branches).
The expected workflow is to compare my fork with the original UE4 codebase, and then manually integrate the highlighted differences into your own build. I only update my branches to major engine versions, so if you use my fork directly you'll miss out on all engine hotfixes.
Supported versions are 4.20, 4.21, 4.22 and 4.23.

Disclaimer: I don't actively use the system and I cannot maintain it indefinitely. I try to keep up with major engine revisions but eventually a day will come when I won't support it.

How to use

1. set r.LocalIBLFromCaptures to 1 via console (set to 0 to disable again if needed). you probably want to add this into the ConsoleVariables ini
2. You'll need to re-compile all Translucent materials manually, since the flag is not propagated for those until the shader is compiled.
3. Make sure you have a Skylight that is set to Movable
4. Place ReflectionCapture Actors (sphere or box) encompassing the areas where you want Local IBL
5. Build the Reflection Captures to get instant results
6. Tweak your lights' "Indirect Lighting Intensity" to control how much each light contributes to the Local IBL (you probably should set it to 0.0 if the light will change during runtime). Build the Reflection Captures when finished
7. Optionally tweak your PostProcessVolume's 'Indirect Lighting Intensity' and 'Indirect Lighting Color' (applies as global changes) if needed. Build the Reflection Captures when finished
8. Optionally modify each ReflectionCapture's ContributionFactor value in cases where the full IBL isn't wanted (both at editor time and at runtime)

Limitations and considerations

• Since the ReflectionCaptures capture the scene from a single point and project it from there, it requires precise placement (just as it is for reflections, but here it's much more evident)
• Areas with a clear lighting directionality (i.e. bright spots on one side of a room) require multiple ReflectionCaptures to avoid the light getting projected unnaturally
• As IBL overrules skylighting (intended), anything outdoors that's affected by reflection captures won't be affected by skylighting. Therefore it's recommended to only use it in fully occluded areas (interiors) or semi-occluded areas (i.e. a forest) but not on clear outdoor areas
• On areas with "sharp" indoor/outdoor delimitations (i.e. building walls) it requires very precise placement to avoid indoor <--> outdoor leaking (not just keeping the skylight from coming in, but also avoiding darkening the outer walls), so thick walls are needed
• On areas without "sharp" indoor/outdoor delimitations the capture radius/size can be kept much bigger for a smoother transition
• As the engine is limited to 341 ReflectionCaptures there's a tradeoff between precision and level size (it's probably not feasable for really large levels that require precise placement like a city with building interiors)
• Translucent materials only work with local IBL when using the 'Surface' Lighting modes. The 'Volumetric' Lighting modes only ignore skylight altogether and do not make use of the reflection captures. You can get the old behavior by
  commenting out lines 562 and 567 on BasePassPixelShader.usf, but then they will again be affected by the skylight (and they will still not be affected by the reflection captures).
• If using the Forward Renderer, it requires to recompile all shaders (the toggle isn't instant as with deferred).

Additional Technical details

• The gist of it: Reflection Captures are passed over as IBL in the form of ambient light using the highest mip of the reflection.
• Localized IBL is applied in the same shader as the light from the SkyLight, but areas affected by IBL are no longer affected by the SkyLight (smoothly blended). This effectively allows using Reflection Captures as "skylight blocking areas" even if ambient light from Localized IBL isn't wanted.
• The new ContributionFactor property on ReflectionCaptures scale the contribution of that ReflectionCapture (both into ambient light as IBL and into the reflections)
• Dynamic lights are captured into IBL based on the existing 'Indirect Lighting Intensity' property of each light, which can be set to 0 to make it not contribute any light to IBL. This is useful to skip capturing lights that are expected to change during gameplay.
• The entire thing is toggleable via a new CVar "r.LocalIBLFromCaptures" and when disabled the code/shader footprint is next to nothing
• DFAO is applied also in IBL-affected areas, inherits all DFAO properties from the SkyLight
• All of the IBL light contribution will be affected globally with the PostProcessVolume's 'Indirect Lighting Intensity' and 'Indirect Lighting Color' properties
• Works only when using a Movable Skylight - gets properly disabled on Static or Stationary SkyLight - works even alongside a baked Static or Stationary DirectionalLight (but who would ever want that)
• Works with both Sphere and Box Reflection Captures
• Works using Reflection Captures' Source Type as 'Captured Scene' and 'Specified Cubemap' as well

The engine changes are relatively small (under 200 lines of code between C++ and HLSL) and the footprint when the feature is disabled is next to nothing. The pictures speak for themselves and local ambient light (or at least a way to block the skylight) is something we've been asking for a long time

Also I'd love to see this in actual game scenery so if someone integrates this into a custom build don't hesitate to post comparison screenshots

[Daedalus51]

I think this is pretty amazing but I am just gonna say it:

Why would you want to do this in the first place if not for fully dynamic lighting? So the fact it cant capture movable lights makes it in my eyes useless, because I would never use stationary lights for a dynamic light setup as they, as far as I understood, dont support shadow caching like the movable ones do and anyways, they are meant to be baked.

Dont get me wrong, I love the work...I am more playing devils advocate here^^

The complaint is that we cant work like in Cryengine for dynamic lighting, so if we cant capture movable lights, does this not mean this whole approach falls apart anyways?

Maybe am wrong, but in Cryengine all lights are always movable right?

Just some thoughts^^ But nevertheless...highly appreciated work man! I remember, when we worked on DI2, reflection capture actors were actually capturing dynamic lights too. But that was around 4.6 or so I think. DI2 was using a lightprobe system and fully movable lighting.

Daedalus

[Chosker]

fair points there. don't worry I won't get you wrong, I actually welcome the discussion

as much as I'd like this to work on a fully dynamic lighting scenario, as I mentioned reflection cubemaps are fully static on cooked. with this in mind the only use case that seemed useful is for lights that are "permanent" (even in a fully dynamic lighting scenario you'd hardly expect all lights to be toggled / moved). this would be the equivalent of adding fill lights.

I had the impression that static and stationary lights would act as fully dynamic if the project settings have "Allow Static Lighting" disabled (that was the behavior in UE3 when disabling "use precomputed lighting"). With this you'd have everything dynamic but you'd be able to control which lights are never meant to change and therefore have them opt in into IBL.
I found out too late that it seems it's not the case and static/stationary lights still show shadows with this 'Preview' watermark all over the place - so apparently they still expect to be baked even if the project doesn't allow it (and at this point if you Build Lighting it just fails)

so you're right. at this point it's almost useless... except it's still useful to block the sky light from coming through inside interiors (where DFAO falls short) which is already a nice improvement for indoor-outdoor games. it's something


so at this point I'd need to think of how to move forward and figure out what would be an actual useful use case. Suggestions are welcome
I could dive further into the engine code. I don't expect having dynamic lights captured into Reflection Captures should be a big issue to achieve.
but unless I can also make ReflectionCaptures re-capturable in cooked I don't see the point. as soon as the light changes the IBL data would not match anymore.
and even then triggering a recapture would cause a noticeable hitch. every time lights change. so unless I can also make ingame recapturing a deferred process it's still useless. but then it becomes such an enormous task (not sure I could handle all of that, and even then I doubt it would ever get accepted as Pull Request)
or is it useless?

I wonder how CryEngine behaves in all of this (I have no idea how it works, learning a bit of CryEngine is in my ToDo list). and I'd also be curious how it was in DI2 when any light would change

[Daedalus51]

Ah I see...good thoughts man!

So there must be a way to get the reflection captures to not be cooked as that's how it works when you use Enlighten for Unreal. The reflection captures have lower rez than Unreals default, but they all update on runtime during the game as well. So that can be done. Also, there must be code that tells the Engine to not consider movable lights but do it with Stationary ones. So it has to work with movable ones, they probably just don't sent the right signal as the other lights do.

Also, the reflections don't need to update at runtime to work for movable lights! The texture has to be generated in Editor and then saved and stored and then just loaded with the game. Problem is Unreal doesn't do that and recaptures the map when the map gets loaded...at least to my knowledge.

So I would say the first thing one needs to do is to make sure that when you hit capture, it actually creates a texture asset that's being used until you hit recapture. Sadly...UE captures on its own in the background when you change stuff and that needs to be turned off for sure

I actually hate the fact that UE updates captures without telling you! I wish it would never do so until you say it should.

Regarding DI2....well...that's a tricky one as we had a graphics programmer implementing stuff for us^^

In the end, we only worked with dynamic lighting because of the world size, but we didn't actually have changing lighting like TOD or stuff like that. So imagine GTA or Assassins Creed frozen at 2pm

We implemented our own probe solution that got baked on the GPU and solved GI for us. So, in the end, it was like this: Movable sun + movable skylight + probes for GI + DFAO to shadow the dynamic skylight + the GI from the probes. Since the probes are super low rez by nature, you need something like DFAO for more accurate occlusion and shadowing.

So that's the setup we used. We used only movable lights throughout the game and they got captured by the probes as well as by the reflection captures. Worked quite well for us, but it was a bit cumbersome to work with as it was pretty much a dirty alpha implementation by our progger back then^^

Daedalus

PS: I think reflection capture actors are streamable...so you should be able to have more than the 300 something if you have a proper streaming setup At least that's what we talked about with Epic back in the day but I don't fully remember. But we would have exceeded that limit with DI2 as well so that was the proposed solution.

[Chosker]

Ah I see...good thoughts man!

So there must be a way to get the reflection captures to not be cooked as that's how it works when you use Enlighten for Unreal. The reflection captures have lower rez than Unreals default, but they all update on runtime during the game as well. So that can be done.

well when triggering a recapture the code specifically checks if it's uncooked, and otherwise it skips it. that's the part I would need to try to overrule but I assume there's a reason for that (the code explains that only static textures are expected).

as for the resolution of the captures, that's something that's already exposed into reflection captures by unreal so that's not even a problem at this point. however since the probes are also used for reflections, there's only so low-res you can go before you ruin everything on the reflections side (which is why I keep the default resolution and simply use the highest mip for lighting)

Also, there must be code that tells the Engine to not consider movable lights but do it with Stationary ones. So it has to work with movable ones, they probably just don't sent the right signal as the other lights do.

yes that's the part I meant earlier. I don't think this would be a big problem to figure out

Also, the reflections don't need to update at runtime to work for movable lights!

they don't have to, but what happens if you turn off a light that was already part of the IBL? you still get indirect lighting from a light that should be turned off

The texture has to be generated in Editor and then saved and stored and then just loaded with the game. Problem is Unreal doesn't do that and recaptures the map when the map gets loaded...at least to my knowledge.

Unreal specifically skips any recapture on cooked, which means it really does work as you expect it to work - texture is generated in editor, saved and stored as a static texture on either saving or cooking [not sure], and then the cooked game only loads the saved static texture.

So I would say the first thing one needs to do is to make sure that when you hit capture, it actually creates a texture asset that's being used until you hit recapture. Sadly...UE captures on its own in the background when you change stuff and that needs to be turned off for sure

I actually hate the fact that UE updates captures without telling you! I wish it would never do so until you say it should.

I've never experienced such behavior actually. for me the captures only happen when I hit recapture. maybe also on -editor- map load but I'm not sure.
when you change stuff unreal doesn't recapture. you can move and even duplicate a reflection capture actor and it will not update anything, it just uses whatever captured data it had (if you duplicated it, it uses the data of the original one). if you move other actors the captures don't update. if you turn lights on and off the captures don't update. so I don't really see how this is a problem

Regarding DI2....well...that's a tricky one as we had a graphics programmer implementing stuff for us^^

In the end, we only worked with dynamic lighting because of the world size, but we didn't actually have changing lighting like TOD or stuff like that. So imagine GTA or Assassins Creed frozen at 2pm

We implemented our own probe solution that got baked on the GPU and solved GI for us. So, in the end, it was like this: Movable sun + movable skylight + probes for GI + DFAO to shadow the dynamic skylight + the GI from the probes. Since the probes are super low rez by nature, you need something like DFAO for more accurate occlusion and shadowing.

So that's the setup we used. We used only movable lights throughout the game and they got captured by the probes as well as by the reflection captures. Worked quite well for us, but it was a bit cumbersome to work with as it was pretty much a dirty alpha implementation by our progger back then^^

Daedalus

thanks for the insight. so with the DI2 setup, what would happen if a light is moved or turned off in-game? was there any recapturing at runtime or did the IBL not match the actual lighting anymore?
changing lights during gameplay would cause you to run into problems unless no light was ever supposed to change in any way, and using dynamic lighting was just for convenience (iteration times and memory savings).

In GTA or Assassins Creed even if you'd fix the time of day you can still have flashlights or torches. my main worry is how those get treated in relation to IBL (I guess any gameplay-spawned lights would just get ignored by the IBL, which would be acceptable). but then imagine a game like Thief where the artist-placed lights can be turned off via gameplay. maybe reflection capture Brightness can be changed at runtime which would fix artist-placed lights being turned off - but it wouldn't fix artist-placed lights that actually move.

also world size is still an issue as UE4 is limtied to 341 reflection probe textures. it's not a terribly small limit so I think it can be worked with, but it means you cannot go very granular and have multiple probes per room in each building (like the UE4 apartment thing) in a big map with a lot of buildings. you need something like 1 probe per room which is I think still okay.

[Daedalus51]

Awesome info, thanks again!

TBH, I think no game updates their IBLs on runtime for stuff like players holding a torch! Same goes for AC or GTA. Local lights that are moving only have teir own specular highlight but are not being updated into the reflection IBL. A game like Thief probably has updating reflection captures to account for someone shooting a light off OR...and that what I think they do, they have 2 kinds of lighsources. They have stuff like streetlights etc. I think those are baked in and cant be turned off during gameplay. They render into the reflection probes.

All the dynamic lights like candles etc DONT render into the reflection probes but only have their own specular. I think that's how they do it tbh.

Playing AC: origins....they don't update anything. Their TOD just lerps though incl. specular probes but nothing updates dynamically based on player input.

Yeah, for DI2 it was just about workflow and not about interactivity! So that's a super important point. If you would shoot out a light, it would still be there since the reflections were not recaptured.

Honestly, I think it works well and the illusion also holds up quite good. People generally don't notice if the reflections are accurate based on a light source you are waving around with your hand.

In Frostbite, there are only movable lights, but we can still bake down the indirect part and bake down reflection volumes. If you move the light, it breaks, but no one notices it. I am even reusing reflection textures from other rooms etc to make it more effieicnt...no one really cares in the end XD

-Daedalus

PS: do you know if the capture actors support streaming? Cause that should theoretically bypass the limitation

PPS: Yeah, maybe Epic changed the update behavior finally! Last time I really checked for it, an actor would instantly update when you move it or duplicate it for example.

[Chosker]

thanks again for the input

yeah I think ignoring gameplay-spawned lights is inevitable. updating the IBL on runtime is anyway too ambitious as I said, but it's good to hear that other games apparently get away with doing the same and it doesn't break the illusion significantly. it would be up to each developer to not break the illusion too much
so finding out how to capture dynamic lights would be the next step for me. however I would still need a way to tag lights to be ignored by the IBL to still have the option of artist-placed lights that might change at runtime, but it seems doable.


in AC: Origins the light from torches looks quite nice, I find it hard to believe it's a single pointlight specially because the shadows still get some color from the light (maybe they have a non-shadow-casting light on top of the shadow-casting-light to simulate some ambience). but I don't think they update their IBL at runtime for moving torches.
AC4 behaved like you describe (interpolated between 8 TOD probes). not sure how much their lighting has evolved since then



so going for this approach leads to this solution being a dynamic-lighting-only solution, which coincidentally is where it's needed the most (static and stationary are fine with their baking, and the place where it really lacks is ambient lighting on dynamic-only scenarios). so it deals with the "it's useless" problem

PS: I don't think capture actors support streaming. if I remember correctly all reflection captures are stored in a single texture-array but I could be wrong.

PPS: I only been properly testing with reflection captures since 4.19, and the behavior I described above is from 4.20 (not sure which preview version, I just got "the 4.20" from git)

[Daedalus51]

Sweet man! Super thanks for the nice info...I really appreciate conversations like this!

Your ideas sound great, so I am honestly really curious where this leads you to!

-Daedalus

[Daedalus51]

Oh btw....The Division showed off some realtime localized surfel update stuff in an old presentation about their GI and when players use flashlights. I believe they eventually didnt ship with it, but maybe thats the magic behind AC:Origins torch? There should still be an hour video somewhere on the interwebs with this presentation, maybe you or I will find it somewhere^^

PS: The old pointlight trick you mentioned works well too....especially if you have a light where you can turn off the specular component to give a soft diffuse light only. Then the bleeding on the character becomes almost invisible if the light is soft enough as the most visible non shadow light leaking issue usually comes from specular. At least thats how I would fake it without fancy tech

[TorQueMoD]

So I'm guessing you had to modify the C++ code to get this working? Is it it's own actor so to speak or does it replace reflection captures? It would be better if you could still use both.

[Chosker]

Sweet man! Super thanks for the nice info...I really appreciate conversations like this!

Your ideas sound great, so I am honestly really curious where this leads you to!

Much apreciated as well

Oh btw....The Division showed off some realtime localized surfel update stuff in an old presentation about their GI and when players use flashlights. I believe they eventually didnt ship with it, but maybe thats the magic behind AC:Origins torch? There should still be an hour video somewhere on the interwebs with this presentation, maybe you or I will find it somewhere^^

sounds like ue4's deprecated DFGI. if you find this video it's always interesting even if its purely for informative purposes

PS: The old pointlight trick you mentioned works well too....especially if you have a light where you can turn off the specular component to give a soft diffuse light only. Then the bleeding on the character becomes almost invisible if the light is soft enough as the most visible non shadow light leaking issue usually comes from specular. At least thats how I would fake it without fancy tech

You know most times its more time and performance -efficient to go for the trick that does the job - especially if its an isolated feature (a torch or a pointlight, while the rest of the game's lights are more static and predictable)
do you know if lights in ue4 can have their specular disabled? Cant check it atm but I have a vague memory of some option of the sort


Ill keep playing around with this when I get some more time. Yesterday I managed to capture all dynamic lights already (was very easy indeed) but I wont have time in the next few days

So I'm guessing you had to modify the C++ code to get this working? Is it it's own actor so to speak or does it replace reflection captures? It would be better if you could still use both.

Yes I'm having to modify the engine for this (c++ and engine shaders). The lighting system cant be decoupled into something that can be made as a plugin or as project-only code (both in terms of code and shaders)
Making it its own actor also doesnt make much sense. Im reusing the ReflectionCaptures because they already have a probe capture array (you wouldnt want 2 such big texture arrays) and they are so similar that it doesnt either justify having them separate. The point is not to hijack the ReflectionCaptures functionality, that still already works and I have no intention of getting rid of it.
If all goes well my IBL system will be non intrusive if you dont opt in, and if you use it it would be a matter or placing reflection probes as usual, but getting -also- IBL out of it

[Chosker]

didn't thought I would have time today for this but progress went much smoother than I expected

so I realized that the "Indirect Lighting Intensity" already magically works to control the IBL contribution for each light, and it works with every type of light except for the SkyLight (but it's kinda countered by the fact that the IBL projection -replaces- the skylight, so it only means that the skylight on areas not affected by the IBL will leak a bit and cause a bit of a light rebound effect)

with this I already have lights that opt out of the IBL contribution for cases where gameplay events might change them in any way
this also means I don't really need to decouple the ReflectionCapture's intensity to affect reflections and IBL differently anymore, since the IBL intensity is already controlled by the light.

since dynamic lights are now captured it means that the directional light is now also part of the IBL which also causes a bit of a sun light rebound effect. for dynamic Time of Day setups it's as simple as setting the Indirect Lighting Intensity to 0

here's the current result (sorry about the jpeg artifacts, it's actually butter-smooth inside unreal)

there's some weird white light leaking in the back room and at the moment I can only get rid by making the reflection probe in that area have a 0.0 brightness. I definitely need to figure out why this is happening


as this is shaping up nicely I think the next step should be opting in and out of the IBL system entirely


eh I'm surprised Maximum-Dev hasn't showed up around here

btw Daedalus51 I found that loading a map in the editor doesn't recapture the reflections, it actually uses the last saved data (and when you hit recapture it marks the LevelName_BuiltData asset as dirty)

also checked, and yes you can remove all specular from a light using the "Specular Scale" property. surely enough for the case of the torch/flashlight 'secondary' ambient light

oh god this forum's formatting is terrible

[Maximum-Dev]

eh I'm surprised Maximum-Dev hasn't showed up around here

*Reply is totally off-topic*

I've been fighting the auto exposure and PPV cubemap. You go hardly find an approximately correct value for one thing, something else goes off, you patch that up, something else goes wrong.
For now I just settled on leaving Auto Exposure settings on defaults and loading my cubemap into Skylight with lower hemisphere is black being unchecked and directional light intensity set to 1. Sounds crazy but that's the best spot for outdoor so far unless we don't use auto exposure at all. All in all, we'd be really happy if UE5 is going to address dynamic lighting issues. ^^

[Amiot]

this is awesome ! more screenshots or maybe an video of animated Time of day would be awesome !

[TorQueMoD]

...The point is not to hijack the ReflectionCaptures functionality, that still already works and I have no intention of getting rid of it.
If all goes well my IBL system will be non intrusive if you dont opt in, and if you use it it would be a matter or placing reflection probes as usual, but getting -also- IBL out of it

Oh that's awesome! You should load up a gumroad or something. I'd love to support you a bit as you work on something like this. Do you have plans to put it on the marketplace?