Announcement

Collapse
No announcement yet.

[WIP] Anime/Toon Stylized Shading Model

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

    [WIP] Anime/Toon Stylized Shading Model

    Over the last few years I've had a lot of people ask me how I was achieving the shading in my game. Originally I was using the method outlined by Tom Looman here to retrieve the dynamic shadow information and use it within the material editor, however Epic broke this back in 4.20 and onward, and even when it worked it was a buggy hacky mess. I've spent the last couple months digging into the horrible undocumented depths of Unreal's shader system to go and achieve what I was doing before, but "properly" this time around. Here are the results.





    Features:



    I've exposed several shadow properties that I require for my project, they include the offset of Unreal's Vector based shadows, the hardness of the shadow, and the colour of the shadow. The latter of which can be used to do all kinds of ridiculous things, like this...



    or this,



    ...but I just use it to make sure that the colour of a shadow always complements the colour of the light, and surface nicely.

    Subsurface scattering also works, and has been slightly modified to work with the light levels I (and I imagine anyone else using this) would be working with.


    (ignore the garbage gif compression)

    Additionally, I've also added a control for dynamic shadow opacity. I use it on my character's faces to eliminate dynamic shadows and only leave the Vector based shadows. I only use this on my character's faces since they have edited normals to detach shading from the actual shape of the geometry, and dynamic shadows ruin this illusion.

    Limitations & Known issues:

    Only Forward Rendering
    is supported. Deferred shading is NOT supported.

    Lightmaps aren't supported at the moment, but I''m looking into integrating them in the future.

    Shader Complexity viewmode doesn't work correctly. I don't know if this is related to my changes or not.

    How it all works:



    Most of this works just as it does in vanilla Unreal, the important bits are:

    Shadow Range = this controls the softness of light vector shadows 0 is hard 1 is completely soft.

    Specular/Roughness = Unchanged, but you probably want to set it to 0/Fully Rough respectively unless you're making a material that utilizes reflections in some form or another.

    Shadow Colour = Colour of the pixel when in shadow. In vanilla Unreal, shadow colour is multiplied by base colour to get your result, however here the colour when in shadow is entirely decoupled from base colour. This allows you a lot of freedom since you can quite literally put whatever you want here. You can do the above Nicholas Cage effect, make shadows brighter than unshadowed areas, or... do what I do and multiply my shadow colour by my base colour. (probably that one tbh)

    Shadow Properties = Blue Channel is Vector Shadow Offset. Green Channel controls whether dynamic shadows are included in the shadow (1= Light Vector Shadows Only, 0= normal behaviour), Red channel is dynamic shadow range.
    Ambient Occlusion = Permanent shadows.

    For the sake of simplicity, this shading model hijacks the default lit shading model, so there's no faffing about with changing the shading model for every single material you make, and you can use the default shading models (like subsurface) without issue (I hope, I haven't tested anything other than subsurface :> ). The downside of this is that if you need the unmodified Default Lit shading model in addition to this you'll have to put in the extra footwork of segregating my changes off into its own shading model.

    Download & Getting Started:

    You can find my repository here: https://github.com/envieous/UnrealEngine-SelShader

    Before you ask why you got a 404 after clicking that link, read this: https://www.unrealengine.com/en-US/ue4-on-github

    There's a few changes that need to be made right from the get go to make this work in the way I intend it to.

    You can either download the example project I've made here, or make these changes yourself from scratch.

    First, in project settings, enable forward shading.



    next, disable static lighting.



    For point/spot lights to work correctly inverse square falloff needs to be disabled.



    The rest of these are optional, but chances are if you're using this you'll probably want to do these too.

    Disable auto exposure.



    In your post process set exposure compensation to 0



    In your directional light set it to moveable and give it an intensity of 1, the goal of this shading model is for colour values to be what you put in is what you get, and this results in that.



    It's also likely that if you're using this you probably don't want any sort of glossiness effects in most of your materials, so consider enabling this in materials you use.



    Follow/Support this project:

    You can follow me on twitter here & join my discord here.

    & if you like this and want to support further development of this shading model (and any other useful stylized rendering stuff I happen to make), you can donate to my Patreon by clicking the button below.



    Enjoy :>

    Acknowledgements:

    Tom Looman for his Dr Facilier shadow demonstration that I used initially as the basis for my shading system.
    Doomfest and Arnage, for their custom shading models. I studied a lot of their changes to figure out what was going on in Unreal's USF files.
    FelixK and Matt Hoffman for their articles on creating a custom shading model in Unreal. These articles despite being dated were still incredibly helpful for figuring all this out.
    Last edited by Envieous; 07-20-2020, 03:23 PM.
    Anime Shading Model - Twitter

    #2
    Interesting, congratulations!
    Does it also work with android (forward renderer)?

    Comment


      #3
      Originally posted by IppolitoGermer View Post
      Interesting, congratulations!
      Does it also work with android (forward renderer)?
      Not at the moment. My changes so far are limited to features that I'm actively using in my project, which is just the desktop forward renderer at the moment, once I finish my game I'll look into porting it, and this to mobile, but that's probably atleast a year or two out. However I did notice that Epic is planning to bring the desktop forward renderer to mobile in the future so perhaps in a future engine version this will work seamlessly with mobile. :>
      Anime Shading Model - Twitter

      Comment


        #4
        Added 4.25 version.
        Anime Shading Model - Twitter

        Comment


          #5
          So far, this shading model looks great in 4.25. I really love the option to change shading color altogether, as it gives more flexibility. That, and the shadow range and properties options are great to see.

          There's a few things that I wish could be fixed or added:
          • One thing specifically is how the default lit shading model is replaced with the toon shader. This causes some noticeable issues (such as the white shading that you mentioned, plus everything using toon shading). I think that having toon shading as it's own seperate shading model would be VERY helpful. Maybe for the default shading, have it multiply the base color by a fixed shading color unless overwritten in the shading texture area.
          • The second (Although a bit off-topic), would be a shading model that emulates unlit, but allows only shadows to be casted onto the material, as opposed to having full shading. This would be great for environments with hand-painted textures.
          • Support for point-lights (as currently, they don't work altogether, which would be useful for things like muzzleflashes and other kinds of dynamic lighting). Specularity also doesn't seem to work (Some sort of way to customize it would be great). It also looks like lighting channels don't work with this toon shader either (Which would be useful for characters having their own lighting vector, while still being able to cast shadows).
          Last edited by KingKrouch; 05-11-2020, 07:54 PM.

          Comment


            #6
            Originally posted by KingKrouch View Post
            So far, this shading model looks great in 4.25. I really love the option to change shading color altogether, as it gives more flexibility. That, and the shadow range and properties options are great to see.

            One thing specifically is how the default lit shading model is replaced with the toon shader. This causes some noticeable issues (such as the white shading that you mentioned, plus everything using toon shading). I think that having toon shading as it's own seperate shading model would be VERY helpful. Maybe for the default shading, have it multiply the base color by a fixed shading color unless overwritten in the shading texture area.
            :>

            I'll get around to implementing it properly alongside Default lit eventually, probably the next time I burn out on art because those are good times for me to just dig into technical stuff like this and I don't feel bad about sitting there for hours letting things compile.

            Originally posted by KingKrouch View Post
            The second (Although a bit off-topic), would be a shading model that emulates unlit, but allows only shadows to be casted onto the material, as opposed to having full shading. This would be great for environments with hand-painted textures.
            I'm not sure what you mean?

            Originally posted by KingKrouch View Post
            Support for point-lights (as currently, they don't work altogether, which would be useful for things like muzzleflashes and other kinds of dynamic lighting). Specularity also doesn't seem to work (Some sort of way to customize it would be great). It also looks like lighting channels don't work with this toon shader either (Which would be useful for characters having their own lighting vector, while still being able to cast shadows).
            Point/Spot lights work, but you have to turn off inverse square falloff in their settings (idk why). Their behavior is different than what you might expect though as they're not longer additive, but rather max instead.



            Lighting channels work as expected.



            I made no changes to specular because outside of reflections on certain materials it's always set to 0 for me, and because there's not really a one size fits all specular formula I could implement that would suit everybody's needs. However the reason I did shadow colour as a value that gets passed straight to the shader unmodified rather than multiplied was so specular formulas could be done in the material editor itself (I do this on the lamps in the above picture). The downside is that light positions being available to the material by default is lost, but they can always be inserted relatively painlessly into the material manually if you have a manageable amount of lights and they're really needed.
            Last edited by Envieous; 05-11-2020, 09:40 PM.
            Anime Shading Model - Twitter

            Comment


              #7
              Originally posted by Envieous View Post
              I'm not sure what you mean?

              Point/Spot lights work, but you have to turn off inverse square falloff in their settings (idk why). Their behavior is different than what you might expect though as they're not longer additive, but rather max instead.
              Thanks for telling me that. Regarding what I was saying earlier, I was referring to some sort of way to do unlit-esque shading that is the reverse of what you have for the shading scalar (think of instead of a scalar to remove dynamic shadows from the shading, which I believe you are using for faces, have some sort of way to only have dynamic shadows), but can still accept dynamic shadows. Such a feature would be useful for making hand-painted areas that can still have objects cast dynamic shadows onto it.

              Something else that I noticed with the materials is that UE4's own ambient occlusion isn't completely disabled with the toon shader, which leads to objects looking like they are lit regularly in shaded areas, as opposed to being flat like cel-shading. Perhaps there's a way to expose both UE4's own Ambient Occlusion and a custom node just for Toon Ambient Occlusion, so there's more control over this?

              Comment


                #8
                Originally posted by KingKrouch View Post
                Thanks for telling me that. Regarding what I was saying earlier, I was referring to some sort of way to do unlit-esque shading that is the reverse of what you have for the shading scalar (think of instead of a scalar to remove dynamic shadows from the shading, which I believe you are using for faces, have some sort of way to only have dynamic shadows), but can still accept dynamic shadows. Such a feature would be useful for making hand-painted areas that can still have objects cast dynamic shadows onto it.
                I still don't think I understand what you're asking, can you post a picture?

                All that scalar does is remove the true dynamic shadows while leaving the "fake" light vector shadows alone, doing the opposite doesn't really accomplish anything useful (though you could by having a vector offset value that pushes that gradient up enough that it's always >1.

                Unreal creates dynamic shadows by merging a simple light direction vs surface normal calculation and true dynamic shadows. All that's accomplished by removing the light direction component is that you lose normal map detail, and end up with ugly bumpy shadows because the dynamic shadows aren't entirely accurate so Unreal merges in the light direction component to mask that.



                Originally posted by KingKrouch View Post
                Something else that I noticed with the materials is that UE4's own ambient occlusion isn't completely disabled with the toon shader, which leads to objects looking like they are lit regularly in shaded areas, as opposed to being flat like cel-shading. Perhaps there's a way to expose both UE4's own Ambient Occlusion and a custom node just for Toon Ambient Occlusion, so there's more control over this?
                If you're talking about SSAO that's controlled through the post process which runs after the shader so isn't within its control afaik.
                Last edited by Envieous; 05-13-2020, 11:00 PM.
                Anime Shading Model - Twitter

                Comment


                  #9
                  this is pretty cool
                  can you control self shadow and casted color separately in the shader?

                  and do you think it would work with raytracing turned on?
                  https://github.com/unity3d-jp/Raytra...~/README_EN.md
                  like this one in unity?

                  Comment


                    #10
                    Originally posted by YuuJin View Post
                    this is pretty cool
                    can you control self shadow and casted color separately in the shader?
                    :>

                    Shadow colour is set in the material, so yes, just make the shadow colour value different for the self shadowing object, and whatever object is receiving the casted shadow.



                    Originally posted by YuuJin View Post
                    and do you think it would work with raytracing turned on?
                    https://github.com/unity3d-jp/Raytra...~/README_EN.md
                    like this one in unity?
                    Raytraced shadows are something I have a lot of interest in implementing since Unreal's default shadows for anything that isn't a directional light are just so... bad, but unfortunately I don't have 600$ to blow on an rtx card at the moment. While it's possible it works as is, it'd take someone with raytracing capable hardware to fire this up and find out to tell for sure.
                    Last edited by Envieous; 05-14-2020, 07:37 PM.
                    Anime Shading Model - Twitter

                    Comment


                      #11
                      I have an RTX card
                      but I don't really know how to build an engine, I'll take a crack at it at some point

                      Comment


                        #12
                        it's pretty simple, just gotta make sure you have all the visual studio components you need installed and you're pretty much good to go. gl
                        Anime Shading Model - Twitter

                        Comment


                          #13
                          Small change: the red channel of shadow properties now controls the range of dynamic shadows, while shadow range controls the light vector portion of shadows. While this won't do anything special with the actual shadow data (eg: blur hard shadows), it does give a bit more control since while light vector shadows are always a nice smooth gradient dynamic shadows can be all over the place.
                          Anime Shading Model - Twitter

                          Comment


                            #14
                            I've stopped hijacking the existing customdata pins and made new dedicated ones for Shadow Colour and Properties. This should mean that this plays nicely with all the existing shading models, but the focus of this change was to make the subsurface shading model work with my existing changes. Here's how it looks on my character's skin.

                            Right is with subsurface scattering, left is without.



                            There's a few minor changes that I made to make it fit my style. One, it only affects shadowed areas, surfaces that aren't in shadow are unaffected. Two, I amplified the result a bit to make it work well with 1.0 intensity lights.

                            Enjoy :>
                            Last edited by Envieous; 06-26-2020, 01:20 PM.
                            Anime Shading Model - Twitter

                            Comment


                              #15
                              Updated OP with an example project, and better instructions for how to start, and some more examples. :>
                              Anime Shading Model - Twitter

                              Comment

                              Working...
                              X