New Dynamic GI Approach: Deep Illumination -- Possible in UE4?

Development Rendering
1

I’m not sure how many people are aware of this, but there’s a relatively recent (October 2017) research paper from Cornell University about a dynamic GI approach that I haven’t heard of before, called deep illumination.

Having read through the paper, it seems like a fascinating approach to both dynamic global illumination quality and performance issues. It essentially teaches a neural network to more efficiently discriminate between correct and incorrect lighting from global illumination in real-time at a very low cost.

The researchers state that they can get VXGI quality with only a fraction of the performance impact.

A quote from the paper giving an overview of how it works:

“There are two phases to our approach: 1) the training phase, and 2) the runtime phase. For the training phase, we extract G-buffers (depth map, normal map and diffuse map), the direct illumination buffer, and the output from an any expensive GI technique, which we use as our “ground truth,” from a 3D scene for different possible light-camera-object configurations in terms of position and direction. (We have used trained on the output of path tracing and Voxel-based Global Illumination, but our technique will work on any GI technique.) The GAN network is then trained in a semi-supervised fashion where the input to the network is the G-buffers and direct illumination buffer. The ground truth, that is, the output from the chosen GI technique, is used by the network to learn the distribution with the help of a loss function. Once training is complete, we can then use the trained network for the runtime phase. During the runtime phase, the network can generate indirect illuminations and soft shadows for new scene configurations at interactive rates, with a similar quality to the ground truth GI technique used for training, and using only image space buffer information.” (Page 5).

Has anybody looked into this technique, because it sounds like an amazing solution to dynamic GI issues, like quality in the case of LPV and performance-heavy calculations in the case of VXGI.

Here’s a link to the research paper:

[1710.09834] Deep Illumination: Approximating Dynamic Global Illumination with Generative Adversarial Network (Summary of paper)

https://arxiv.org/pdf/1710.09834.pdf (Actual PDF of the paper)

2

Well, in the essence, this is not a realtime solution.

3

Why do you say that?

If the world itself isn’t procedurally or randomly generated each time the game or level is started, I don’t see why it wouldn’t be a real-time solution.

4

Completely static environment, and there are plenty of ways to do GI for that.

5

Yeah, it would technically be a realtime solution because it just becomes a fancy algorithm. Do some research on how machine learning works and you’ll see what I mean.

“Once trained, our network can approximate global illumination for scene configurations it has never encountered before within the environment it was trained on.”

6

Technically, yep, it could be realtime. On practice, the more changes were done to environment after learning the worse will be the results. To me, this approach seems of zero practical value. At least for now.

7

I’m pretty sure that’s semantics for “pre-computed light baking” :confused:

8

Oh I definitely agree, for now, but with time, almost everything will use neural algorithms because it’s only “natural” for things to head into that direction.

9

What’s natural is for machine algorithms to hit a roadblock and become a dead end, bursting the “AI” bubble. It’s captured many people’s imaginations because it seems like a magical one-size-fits-all solution, and the lack of understanding leads to pinning exceedingly high hopes on impossibly small margins. Naturally, there will be no free lunch.

Besides, you’re still going to need an upgrade: 5ms for the cheapest network on the equivalent of a GTX 1080 is more than peanuts.

10

With time, these neural approaches will only get better and better. It’s getting to the point now where you can nest algorithms inside of algorithms and have really intricate cascades of networks, all interlinked. This is almost identical to how your brain functions and there definitely isn’t any “AI” bubble that’s going to burst. It only moves forward and as soon as quantum computing gets involved, it will exponentially take off… 5ms isn’t that horrible when you look at the cost of the same scene rendered with VXGI at similar visual quality.

11

I agree, and I don’t imagine it’ll be the all encompassing solution it may originally seem. But It seems that this. will likely provide avenues for getting a decent dynamic or semi-dynamic GI system in Unreal. Because we don’t have all that much atm moment except for VXGI, but it itself incredibly costly, so any improvement is welcome, imo.

​​​​​​

13

Here’s the video: Approximating Global Illumination using deep generative models
You can see that the indirect lighting is terrible. It loses all the surface detail of the model. Allowing for it being a static box with a static camera, it should be compared with other half-baked approaches, like enlighten, rather than VXGI. As far as I’m concerned, they outright lie about getting VXGI-quality output.

In the paper, they do mention increasing the layers from 32 (4.5ms) to 128 (22ms) to get results “indistinguishable” from VXGI (36ms). Not so cheap. Speaking of which, I think their values on VXGI are a bit off. 36ms to render a 256x256 pixel image? That must look glorious in a real scene.

14

And there’s another reason not to use it. Neural Nets look fantastic for approximating things that don’t need to be totally accurate in games. Fluid dynamics, hair/fur/cloth physics, and other stuff that can just approximate looking right can be run much much faster using a NN. Similarly AI (of course) and throwing it at confusing and frustrating as it is animation graphs could do wonders. But where precise mathematical returns are needed, like visuals or physics that make a large impact on gameplay (driving games) they’re not so good.

15

There are some really interesting Neural network GI research out there. Like this one. http://research.nvidia.com/sites/default/files/publications/dnn_denoise_author.pdf
Hack\Talks GRAPHICS-STAGE - YouTube

Not yet realtime but gap is closing with next generation of GPU’s.

16

It’s not hype-clickbait-sensationalism though and is used in almost every aspect of life now… You’re linking a bunch of cherry picked quotes, by someone that is probably an engineer that’s afraid of losing their job, and it sounds almost as bad as a climate change denier that doesn’t want to give up their 8mpg truck. From facebook picture recognition, to google predicting what you’re looking for as you type in the search bar, to ATMs reading handwriting on a scanned check written with pen, to big city traffic light control, to self-driving cars, etc etc. It’s everywhere and it’s not hype… You just need to sit down, truly research the topic and actually grasp the information. It’s not a matter of if it’s going to replace 90% of the workforce, it’s a matter of when and that when is rapidly approaching. AI capabilities are following an exponential curve, not a linear one.

As for making it difficult to see what’s going on, AI networks aren’t really meant to be fiddled with by humans. The big key, for now, is that they are only as good as the developers that program them though hence why companies like Google are making pushes to have AI write it’s own code. Also, there are dozens of primary types of ANNs and many different hybrid approaches.

17

I could not agree more. While the problem domain is confined meaning into a finite number of elements enclosed in a limited space (hair and fluid simulation fits a lot) the gains can be great.

21

NVidia gonna have a talk this GDC:

http://schedule.gdconf.com/session/advances-in-real-time-voxel-based-gi-temporal-super-resolution-presented-by-nvidia/856294

Maybe someone responsible for lighting programming takes something out of that.

22

Nice to hear they focused on optimizing VXGI. Maybe they got it to a point where it’s really practical for games.

23

Im also curious about the TSAA, they saying it is cheap to adapt into any game engine.

24

Sounds a lot like Temporal AA Upsampling in UE 4.19.