Announcement

Collapse
No announcement yet.

New Dynamic GI Approach: Deep Illumination -- Possible in UE4?

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

    New Dynamic GI Approach: Deep Illumination -- Possible in UE4?

    I'm not sure how many people are aware of this, but there's a relatively recent (October 2017) research paper from Cornell University about a dynamic GI approach that I haven't heard of before, called deep illumination.

    Having read through the paper, it seems like a fascinating approach to both dynamic global illumination quality and performance issues. It essentially teaches a neural network to more efficiently discriminate between correct and incorrect lighting from global illumination in real-time at a very low cost.

    The researchers state that they can get VXGI quality with only a fraction of the performance impact.

    A quote from the paper giving an overview of how it works:

    "There are two phases to our approach: 1) the training phase, and 2) the runtime phase. For the training phase, we extract G-buffers (depth map, normal map and diffuse map), the direct illumination buffer, and the output from an any expensive GI technique, which we use as our “ground truth,” from a 3D scene for different possible light-camera-object configurations in terms of position and direction. (We have used trained on the output of path tracing and Voxel-based Global Illumination, but our technique will work on any GI technique.) The GAN network is then trained in a semi-supervised fashion where the input to the network is the G-buffers and direct illumination buffer. The ground truth, that is, the output from the chosen GI technique, is used by the network to learn the distribution with the help of a loss function. Once training is complete, we can then use the trained network for the runtime phase. During the runtime phase, the network can generate indirect illuminations and soft shadows for new scene configurations at interactive rates, with a similar quality to the ground truth GI technique used for training, and using only image space buffer information." (Page 5).



    Has anybody looked into this technique, because it sounds like an amazing solution to dynamic GI issues, like quality in the case of LPV and performance-heavy calculations in the case of VXGI.




    Here's a link to the research paper:

    https://arxiv.org/abs/1710.09834 (Summary of paper)

    https://arxiv.org/pdf/1710.09834.pdf (Actual PDF of the paper)

    #2
    Well, in the essence, this is not a realtime solution.

    Comment


      #3
      Originally posted by Deathrey View Post
      Well, in the essence, this is not a realtime solution.
      Why do you say that?

      If the world itself isn't procedurally or randomly generated each time the game or level is started, I don't see why it wouldn't be a real-time solution.

      Comment


        #4
        Completely static environment, and there are plenty of ways to do GI for that.

        Comment


          #5
          Yeah, it would technically be a realtime solution because it just becomes a fancy algorithm. Do some research on how machine learning works and you'll see what I mean.

          "Once trained, our network can approximate global illumination for scene configurations it has never encountered before within the environment it was trained on."
          https://arxiv.org/abs/1710.09834

          Comment


            #6
            Technically, yep, it could be realtime. On practice, the more changes were done to environment after learning the worse will be the results. To me, this approach seems of zero practical value. At least for now.

            Comment


              #7
              the output from an any expensive GI technique, which we use as our “ground truth,” from a 3D scene for different possible light-camera-object configurations in terms of position and direction.
              I'm pretty sure that's semantics for "pre-computed light baking"
              | Savior | USQLite | FSM | Object Pool | Sound Occlusion | Property Transfer | Magic Nodes | MORE |

              Comment


                #8
                Originally posted by Deathrey View Post
                Technically, yep, it could be realtime. On practice, the more changes were done to environment after learning the worse will be the results. To me, this approach seems of zero practical value. At least for now.
                Oh I definitely agree, for now, but with time, almost everything will use neural algorithms because it's only "natural" for things to head into that direction.

                Comment


                  #9
                  Originally posted by IronicParadox View Post

                  Oh I definitely agree, for now, but with time, almost everything will use neural algorithms because it's only "natural" for things to head into that direction.
                  What's natural is for machine algorithms to hit a roadblock and become a dead end, bursting the "AI" bubble. It's captured many people's imaginations because it seems like a magical one-size-fits-all solution, and the lack of understanding leads to pinning exceedingly high hopes on impossibly small margins. Naturally, there will be no free lunch.

                  Besides, you're still going to need an upgrade: 5ms for the cheapest network on the equivalent of a GTX 1080 is more than peanuts.

                  Comment


                    #10
                    Originally posted by Dementiurge View Post
                    What's natural is for machine algorithms to hit a roadblock and become a dead end, bursting the "AI" bubble. It's captured many people's imaginations because it seems like a magical one-size-fits-all solution, and the lack of understanding leads to pinning exceedingly high hopes on impossibly small margins. Naturally, there will be no free lunch.

                    Besides, you're still going to need an upgrade: 5ms for the cheapest network on the equivalent of a GTX 1080 is more than peanuts.
                    With time, these neural approaches will only get better and better. It's getting to the point now where you can nest algorithms inside of algorithms and have really intricate cascades of networks, all interlinked. This is almost identical to how your brain functions and there definitely isn't any "AI" bubble that's going to burst. It only moves forward and as soon as quantum computing gets involved, it will exponentially take off... 5ms isn't that horrible when you look at the cost of the same scene rendered with VXGI at similar visual quality.

                    Comment


                      #11
                      Originally posted by IronicParadox View Post

                      With time, these neural approaches will only get better and better. It's getting to the point now where you can nest algorithms inside of algorithms and have really intricate cascades of networks, all interlinked. This is almost identical to how your brain functions and there definitely isn't any "AI" bubble that's going to burst. It only moves forward and as soon as quantum computing gets involved, it will exponentially take off... 5ms isn't that horrible when you look at the cost of the same scene rendered with VXGI at similar visual quality.
                      I agree, and I don't imagine it'll be the all encompassing solution it may originally seem. But It seems that this. will likely provide avenues for getting a decent dynamic or semi-dynamic GI system in Unreal. Because we don't have all that much atm moment except for VXGI, but it itself incredibly costly, so any improvement is welcome, imo.

                      ​​​​​​

                      Comment


                        #12
                        A Counter-view -

                        Neural nets may bring benefits to CG / Rendering, but its best not to drink too much of the AI koolaid!
                        NN / AI is hype being heavily driven by clickbait-chasing journalists / tech companies with too much $...

                        Adding nested layers makes complexity go up, often making it difficult to understand what's going on...
                        NN's are not good at inferring reasoning based on learning experience, just general pattern matching.
                        They can't see outlier exceptional cases, realize their importance and learn from that... Context is key:

                        Last edited by ClavosTech; 02-01-2018, 10:31 PM.

                        Comment


                          #13
                          Here's the video: Approximating Global Illumination using deep generative models
                          You can see that the indirect lighting is terrible. It loses all the surface detail of the model. Allowing for it being a static box with a static camera, it should be compared with other half-baked approaches, like enlighten, rather than VXGI. As far as I'm concerned, they outright lie about getting VXGI-quality output.

                          In the paper, they do mention increasing the layers from 32 (4.5ms) to 128 (22ms) to get results "indistinguishable" from VXGI (36ms). Not so cheap. Speaking of which, I think their values on VXGI are a bit off. 36ms to render a 256x256 pixel image? That must look glorious in a real scene.

                          Comment


                            #14
                            Originally posted by Dementiurge View Post
                            Here's the video: Approximating Global Illumination using deep generative models
                            You can see that the indirect lighting is terrible. It loses all the surface detail of the model. Allowing for it being a static box with a static camera, it should be compared with other half-baked approaches, like enlighten, rather than VXGI. As far as I'm concerned, they outright lie about getting VXGI-quality output.

                            In the paper, they do mention increasing the layers from 32 (4.5ms) to 128 (22ms) to get results "indistinguishable" from VXGI (36ms). Not so cheap. Speaking of which, I think their values on VXGI are a bit off. 36ms to render a 256x256 pixel image? That must look glorious in a real scene.
                            And there's another reason not to use it. Neural Nets look fantastic for approximating things that don't need to be totally accurate in games. Fluid dynamics, hair/fur/cloth physics, and other stuff that can just approximate looking right can be run much much faster using a NN. Similarly AI (of course) and throwing it at confusing and frustrating as it is animation graphs could do wonders. But where precise mathematical returns are needed, like visuals or physics that make a large impact on gameplay (driving games) they're not so good.

                            Comment


                              #15
                              There are some really interesting Neural network GI research out there. Like this one. http://research.nvidia.com/sites/def...ise_author.pdf
                              https://youtu.be/KmrodtJudLg?t=24m31s

                              Not yet realtime but gap is closing with next generation of GPU's.

                              Comment

                              Working...
                              X