Localized-IBL implementation

[Chosker]

this is awesome ! more screenshots or maybe an video of animated Time of day would be awesome !

Ill post more screenshots as I progress
an animated time of day wouldnt be any impressive though, since the IBL does not get recaptured as lights move (you'd only see a moving sun no different than you can do now, only difference being the skylight not affecting it)

Oh that's awesome! You should load up a gumroad or something. I'd love to support you a bit as you work on something like this. Do you have plans to put it on the marketplace?

Thanks for the kind words
I cant put this on gumroad or the marketplace because they are engine changes. My intention is to release the code as open source (for anyone who uses UE4 as custom-built source) and also make a Pull Request to try to have Epic integrate it into the engine (but that would take a while if even they would accept it)

Im only looking to learn and improve my skills, and since I dont even have a project to use this IBL on atm, I thought why not try to improve the engine for everyone

[Tural]

an animated time of day wouldnt be any impressive though, since the IBL does not get recaptured as lights move (you'd only see a moving sun no different than you can do now, only difference being the skylight not affecting it)

Awesome work really. So what I understand, this is only for static light, but we will not need to build anything. Am I right? So for dynamic light condition we still have to stick with DFAO. Or will there be any way, manually recapture IBL like how we recapture reflection captures with additional cost?

[Daedalus51]

Soooo....I actually found the videos I meant! Chosker

This is the tech talk with a timestamp:

And here is proof that the Division actually shipped that **** as well! I guess PC only though: https://www.youtube.com/watch?v=ZfT0HIFZ37o

Enjoy

BTW: This talk is one of the best talks about open world global illumination so...Unreal Forum People...take a note, watch it and educate yourselves This is basically what you want. But also note what enormous task it is to build and maintain such a system...so thats the reason why youre not gonna get it^^

Cheers

[evilmrfrank]

We use localized IBL's on Ark Survival Evolved, the dev kit for it is freely downloadable if you wanted to check it out

[Chosker]

I've updated the main thread with progress. no new pictures for now though.

• I added a new CVar r.LocalIBLFromCaptures which turns the feature on or off, and passes a define to the shader to use the new behavior or the old one. This makes the entire feature's footprint next to nothing when it's not used
• I also fixed the weird white light leaking in the back room. it was actually the sphere (which I use to visualize reflections) which was metallic and on ReflectionCaptures UE4 captures metallic shaders as non-metals, therefore it was being captured as a solid full-white object and was contributing to IBL a lot - this behavior is now gone when using r.LocalIBLFromCaptures

by now I'd say the proof of concept works, and since it now has proper enabling/disabling I can almost call it a 'beta feature'

Awesome work really. So what I understand, this is only for static light, but we will not need to build anything. Am I right? So for dynamic light condition we still have to stick with DFAO. Or will there be any way, manually recapture IBL like how we recapture reflection captures with additional cost?

if you read the thread you'll understand that's not at all the intention.
Static lighting already has a good workflow (iteration time aside) and the quality will always be superior. Epic is also working on improving Lightmass once again (we don't know the details but we know they hired a few people specifically for it) - so I see no point.
The purpose of this is for scenes with dynamic lighting to have some sort of baked data on lights that do not move or change (you really don't want to bake static IBL data on lights that will change) - to use as localized ambient lighting instead of having skylight leaking indoors and no way to block it. even if all of your lights would change ingame and have them all opt out of IBL, it would still serve as a means to stop skylighting coming through into interiors (since DFAO falls short at it)
yes you need to build some data via the existing Build Reflection Captures functionality (pretty much instant), and as I said it is limited to editor-only capturing

Soooo....I actually found the videos I meant! Chosker

This is the tech talk with a timestamp:

And here is proof that the Division actually shipped that **** as well! I guess PC only though: https://www.youtube.com/watch?v=ZfT0HIFZ37o

Enjoy

BTW: This talk is one of the best talks about open world global illumination so...Unreal Forum People...take a note, watch it and educate yourselves This is basically what you want. But also note what enormous task it is to build and maintain such a system...so thats the reason why youre not gonna get it^^

Cheers

thanks, the talk is quite informative, sadly however out of my scope
IMO the 'proof' video doesn't really show any dynamic GI though, all I see is a light that moves and casts dynamic shadows - I don't really see the impact on GI in any way

We use localized IBL's on Ark Survival Evolved, the dev kit for it is freely downloadable if you wanted to check it out

oh awesome, thanks for that
the dev kit download will take a while but looking at the Lighting Documentation tells me that I'm quite on the right track.
as the doc doesn't quite describe the tech I'm confused if you use pre-baked cubemaps for the Master IBL Captures or if you actually use the capturing functionality. The Master-Child functionality seems like a great thing to save up on memory and/or overcome the 341 captures limit, as long as you can get away with something more generic (i.e. works on homogenous interiors or caves / forests, but wouldn't quite work if you want to capture colored lights as I have in my screenshots above)
great stuff. I'll wait to try it myself and see which concepts I could/should try to integrate

[Chosker]

Another update

Cleaned up the shader code a bit and re-added DFAO to the IBL-affected areas





A few more screenshots:
https://imgur.com/a/2R2mcIv

Also updated the main post to reflect the status

[Daedalus51]

Man, this is some super awesome progress! Need to try this out at some point

Regarding the Division stuff...yeah....totally out of scope XD But I just wanted to share it anyways as it has tons of info and we were still talking about local torch bounce light contribution^^
Also, not sure if my eyes are fooling me, but if you watch the proof video again, behind the light is a vertical beam that seems to be getting darker and brighter from the bounce light of the tube light as it moves...so I thought it was kinda working

Anyways...great updates man!

Daedalus

[Chosker]

Man, this is some super awesome progress! Need to try this out at some point

shouldn't be long until I make the pull request so you'll probably be able to try it soon if you integrate the engine changes yourself
I'm actually looking forward to seeing it in action in a real level!

Regarding the Division stuff...yeah....totally out of scope XD But I just wanted to share it anyways as it has tons of info and we were still talking about local torch bounce light contribution^^
Also, not sure if my eyes are fooling me, but if you watch the proof video again, behind the light is a vertical beam that seems to be getting darker and brighter from the bounce light of the tube light as it moves...so I thought it was kinda working

oh you might be right, didn't pay attention at the beam



so another small update
no actual changes this time around, just assembled a cave out of cubes and made a proper comparison screenshot
https://imgsli.com/MTYyNQ

it's almost literally day and night

from this comparison now I've noticed that SSAO is behaving quite differently, as if the IBL is getting rid of it. I'll investigate

[Tural]

[/SIZE][/FONT][/COLOR][/LEFT]

so another small update
no actual changes this time around, just assembled a cave out of cubes and made a proper comparison screenshot
https://imgsli.com/MTYyNQ

it's almost literally day and night

WOW. Maan... I am noob about all those stuff, but maaannnn... What a difference. I always thought, there is something not right with dynamic light interiors of UE4 but couldn't put the finger on, what exactly is not right. And this comparison explains many things to me now. One thing I didn't understand. Why, the far wall is more green than the wall near to the emissive lights? Is this how it must work? Or this is because of SSAO?

And about Division video. I think after 0:23 bounce light is visible where the beam connects to ceiling. When light goes far, ceiling became brighter, and when it comes close to beam, ceiling becomes darker. But there is static GI too. For example same guy uploaded this video too.

https://www.youtube.com/watch?v=j3N17_ZoPBg

[Maximum-Dev]

https://imgsli.com/MTYyNQ

Takes them very little effort to implement IBL. But the gain is a world of difference, as seen above.
Between that and this, you can hardly tell Epic supports dynamic lighting in UE4.

[Manoel.Neto]

Instead of Epic adding everything and the kitchen sink to UE4, I think they should make the render and lighting pipeline more extensible so things like this could be easily added as a plugin without requiring a engine fork.

[Chosker]

WOW. Maan... I am noob about all those stuff, but maaannnn... What a difference. I always thought, there is something not right with dynamic light interiors of UE4 but couldn't put the finger on, what exactly is not right. And this comparison explains many things to me now.

it's been discussed and requested in quite a few threads in this forum for quite a while now

One thing I didn't understand. Why, the far wall is more green than the wall near to the emissive lights? Is this how it must work? Or this is because of SSAO?

the reason is the directionality of it. the reflection is captured from the center of the room which captures the strong light to one side, and projects it back onto the other side (because it's a projected cubemap after all) - as of now I have 2 reflection captures in the fake cave
reflection captures suffer from the same, it's just much less noticeable. on scenes like this with a strong directionality factor into the expected ambient light it's best to use multiple reflection captures to make the ambient light even more "local"

Takes them very little effort to implement IBL. But the gain is a world of difference, as seen above.
Between that and this, you can hardly tell Epic supports dynamic lighting in UE4.

sadly it really seems to be the case
I've only added around 200 lines of code across C++ and HLSL to achieve what I have so far, and it's taken me roughly 2 weeks on 1-2 hr coding sessions, and it's only because: a) I'm not a graphics programmer and I had no idea how this system works, and b) iteration times are long because shader compilation takes a long time (which only gets worse because of point a))
Honestly I'd expect it would take a graphics expert from Epic less than a week if they'd wanted to do it

Instead of Epic adding everything and the kitchen sink to UE4, I think they should make the render and lighting pipeline more extensible so things like this could be easily added as a plugin without requiring a engine fork.

I wouldn't mind the graphics pipeline being more extensible as well




here's today's update:
I re-added the SSAO contribution, and also re-added the SkyLight DFAO Occlusion Tint into the mix. so for anyone who for some reason wants a colored AO it's now also being applied to IBL-affected areas just as it is for the SkyLight-affected areas

[doC]

Just wanted to chime in and say that this is fantastic work.

I can't understand why Epic wouldn't either do it on their own or take your push request (which I feel will be sadly the case). This technique may have it's issues, but also can help a lot, even if not in all situations. Until we have some form of at least semi-real time GI, or some other clever fake solution, this would do the trick for many people.

[Chosker]

Just wanted to chime in and say that this is fantastic work.

I can't understand why Epic wouldn't either do it on their own or take your push request (which I feel will be sadly the case). This technique may have it's issues, but also can help a lot, even if not in all situations. Until we have some form of at least semi-real time GI, or some other clever fake solution, this would do the trick for many people.

thanks
right, this technique will have its issues (mainly transition softness vs light leaking). it's not even semi-realtime (it's basically an extra layer of baked static ambient lighting to help in dynamic lighting scenarios). but I don't see a reason not to at least provide the option

I'm actually done with it by now. I just have to review all the changes again and make sure I didn't forget anything and I'll submit the Pull Request. will probably happen today or in the next few days
best part is that I made it using the UE4 4.20 source, so at least it won't be rejected right away simply because it not working in the latest engine version
I have a bit of hope on the Pull Request because the changes in code are actually really small and everything ties in nicely with the existing engine functionality and usage... but you never know

[Chosker]

it's done!







Some comparison shots:
https://imgsli.com/MTYzMg
https://imgsli.com/MTYzMw
https://imgsli.com/MTYzNA
https://imgsli.com/MTYzNQ


I'll post the code and will make the Pull Request as soon as I figure out how