[WIP] Anime/Toon Stylized Shading Model

Development Rendering
, ,
1

This thread is very old. Find the UE5 version here.

Over the last few years I’ve had a lot of people ask me how I was achieving the shading in my game. Originally I was using the method outlined by Tom Looman here to retrieve the dynamic shadow information and use it within the material editor, however Epic broke this back in 4.20 and onward, and even when it worked it was a buggy hacky mess. I’ve spent the last couple months digging into the horrible undocumented depths of Unreal’s shader system to go and achieve what I was doing before, but “properly” this time around. Here are the results.

https://i.imgur.com/vr9tAgd.gif

https://i.imgur.com/EZ3Ps4J.gif

Features:

https://i.imgur.com/yBTC7mT.gif

I’ve exposed several shadow properties that I require for my project, they include the offset of Unreal’s light vector shadows, the hardness of the shadow, and the colour of the shadow. The latter of which can be used to do all kinds of ridiculous things, like this…

https://i.imgur.com/7JjU3Oe.png

or this,

https://i.imgur.com/JjHRFzu.gif

…but I just use it to make sure that the colour of a shadow always complements the colour of the light, and surface nicely.

Subsurface scattering also works, and has been slightly modified to work with the light levels I (and I imagine anyone else using this) would be working with.

https://i.imgur.com/ZKuFFQb.gif

(ignore the garbage gif compression)^

Left character is without, right is with.

Additionally, I’ve also added a control for dynamic shadow opacity. I use it on my character’s faces to eliminate dynamic shadows and only leave the Vector based shadows. I only use this on my character’s faces since they have edited normals to detach shading from the actual shape of the geometry, and dynamic shadows ruin this illusion.

**Limitations & Known issues:

Only Forward Rendering** is supported. Deferred shading is NOT supported.

Lightmaps aren’t supported at the moment, but I’'m interested in looking into integrating them in the future.

Shader Complexity viewmode doesn’t work correctly. I don’t know if this is related to my changes or not.

How it all works:

https://i.imgur.com/MR7B7K0.png

Most of this works just as it does in vanilla Unreal, the important bits are:

Shadow Range = this controls the softness of light vector shadows 0 is hard 1 is completely soft.

Specular/Roughness = Unchanged, but you probably want to set it to 0/Fully Rough respectively unless you’re making a material that utilizes reflections in some form or another.

Shadow Colour = Colour of the pixel when in shadow. In vanilla Unreal, shadow colour is multiplied by base colour to get your result, however here the colour when in shadow is entirely decoupled from base colour. This allows you a lot of freedom since you can quite literally put whatever you want here. You can do the above Nicholas Cage effect, make shadows brighter than unshadowed areas, or… do what I do and multiply my shadow colour by my base colour. (probably that one tbh)

Shadow Properties = Blue Channel is Vector Shadow Offset. Green Channel controls whether dynamic shadows are included in the shadow (1= Light Vector Shadows Only, 0= normal behaviour), Red channel is dynamic shadow range.
Ambient Occlusion = Permanent shadows.

For the sake of simplicity, this shading model hijacks the default lit shading model, so there’s no faffing about with changing the shading model for every single material you make, and you can use the default shading models (like subsurface) without issue (I hope, I haven’t tested anything other than subsurface :> ). The downside of this is that if you need the unmodified Default Lit shading model in addition to this you’ll have to put in the extra footwork of segregating my changes off into its own shading model.

Download & Getting Started:

You can find my repository here: https://github.com/envieous/UnrealEngine-SelShader

Before you ask why you got a 404 after clicking that link, read this: Unreal Engine on GitHub

There’s a few changes that need to be made right from the get go to make this work in the way I intend it to.

You can either download the example project I’ve made here, or make these changes yourself from scratch.

First, in project settings, enable forward shading.

https://i.imgur.com/7EqRR8U.jpg

next, disable static lighting.

https://i.imgur.com/VHe3ie9.jpg

For point/spot lights to work correctly inverse square falloff needs to be disabled.

https://i.imgur.com/pzay2Di.png

The rest of these are optional, but chances are if you’re using this you’ll probably want to do these too.

Disable auto exposure.

https://i.imgur.com/1woWbqY.jpg

In your post process set exposure compensation to 0

https://i.imgur.com/bXhWRvG.jpg

In your directional light set it to moveable and give it an intensity of 1, the goal of this shading model is for colour values to be what you put in is what you get, and this results in that.

https://i.imgur.com/whP2Rsr.jpg

It’s also likely that if you’re using this you probably don’t want any sort of glossiness effects in most of your materials, so consider enabling this in materials you use.

https://i.imgur.com/aSCg7rN.jpg

Follow/Support this project:

You can follow me on twitter here & join my discord here.

& if you like this and want to support further development of this shading model (and any other useful stylized rendering stuff I happen to make), you can donate to my Patreon by clicking the button below.

https://i.imgur.com/nDEYUjw.png

Enjoy :>

Acknowledgements:

Tom Looman for his Dr Facilier shadow demonstration that I used initially as the basis for my shading system.
Doomfest and Arnage, for their custom shading models. I studied a lot of their changes to figure out what was going on in Unreal’s USF files.
FelixK and Matt Hoffman for their articles on creating a custom shading model in Unreal. These articles despite being dated were still incredibly helpful for figuring all this out.

4 Likes
4

Interesting, congratulations!
Does it also work with android (forward renderer)?

5

Not at the moment. My changes so far are limited to features that I’m actively using in my project, which is just the desktop forward renderer at the moment, once I finish my game I’ll look into porting it, and this to mobile, but that’s probably atleast a year or two out. However I did notice that Epic is planning to bring the desktop forward renderer to mobile in the futureso perhaps in a future engine version this will work seamlessly with mobile. :>

6

Added 4.25 version.

7

So far, this shading model looks great in 4.25. I really love the option to change shading color altogether, as it gives more flexibility. That, and the shadow range and properties options are great to see.

There’s a few things that I wish could be fixed or added:

  • One thing specifically is how the default lit shading model is replaced with the toon shader. This causes some noticeable issues (such as the white shading that you mentioned, plus everything using toon shading). I think that having toon shading as it’s own seperate shading model would be VERY helpful. Maybe for the default shading, have it multiply the base color by a fixed shading color unless overwritten in the shading texture area.
  • The second (Although a bit off-topic), would be a shading model that emulates unlit, but allows only shadows to be casted onto the material, as opposed to having full shading. This would be great for environments with hand-painted textures.
  • Support for point-lights (as currently, they don’t work altogether, which would be useful for things like muzzleflashes and other kinds of dynamic lighting). Specularity also doesn’t seem to work (Some sort of way to customize it would be great). It also looks like lighting channels don’t work with this toon shader either (Which would be useful for characters having their own lighting vector, while still being able to cast shadows).
8

:>

I’ll get around to implementing it properly alongside Default lit eventually, probably the next time I burn out on art because those are good times for me to just dig into technical stuff like this and I don’t feel bad about sitting there for hours letting things compile.

I’m not sure what you mean?

Point/Spot lights work, but you have to turn off inverse square falloff in their settings (idk why). Their behavior is different than what you might expect though as they’re not longer additive, but rather max instead.

https://i.imgur.com/0fKTuIX.png

Lighting channels work as expected.

https://i.imgur.com/cQWErl0.png

I made no changes to specular because outside of reflections on certain materials it’s always set to 0 for me, and because there’s not really a one size fits all specular formula I could implement that would suit everybody’s needs. However the reason I did shadow colour as a value that gets passed straight to the shader unmodified rather than multiplied was so specular formulas could be done in the material editor itself (I do this on the lamps in the above picture). The downside is that light positions being available to the material by default is lost, but they can always be inserted relatively painlessly into the material manually if you have a manageable amount of lights and they’re really needed.

9

Thanks for telling me that. Regarding what I was saying earlier, I was referring to some sort of way to do unlit-esque shading that is the reverse of what you have for the shading scalar (think of instead of a scalar to remove dynamic shadows from the shading, which I believe you are using for faces, have some sort of way to only have dynamic shadows), but can still accept dynamic shadows. Such a feature would be useful for making hand-painted areas that can still have objects cast dynamic shadows onto it.

Something else that I noticed with the materials is that UE4’s own ambient occlusion isn’t completely disabled with the toon shader, which leads to objects looking like they are lit regularly in shaded areas, as opposed to being flat like cel-shading. Perhaps there’s a way to expose both UE4’s own Ambient Occlusion and a custom node just for Toon Ambient Occlusion, so there’s more control over this?

10

I still don’t think I understand what you’re asking, can you post a picture?

All that scalar does is remove the true dynamic shadows while leaving the “fake” light vector shadows alone, doing the opposite doesn’t really accomplish anything useful (though you could by having a vector offset value that pushes that gradient up enough that it’s always >1.

Unreal creates dynamic shadows by merging a simple light direction vs surface normal calculation and true dynamic shadows. All that’s accomplished by removing the light direction component is that you lose normal map detail, and end up with ugly bumpy shadows because the dynamic shadows aren’t entirely accurate so Unreal merges in the light direction component to mask that.

https://i.imgur.com/SQKgHUa.png

If you’re talking about SSAO that’s controlled through the post process which runs after the shader so isn’t within its control afaik.

11

this is pretty cool
can you control self shadow and casted color separately in the shader?

and do you think it would work with raytracing turned on?
https://github.com/unity3d-jp/Raytra…~/README_EN.md
like this one in unity?

12

:>

Shadow colour is set in the material, so yes, just make the shadow colour value different for the self shadowing object, and whatever object is receiving the casted shadow.

https://i.imgur.com/7JjU3Oe.png

Raytraced shadows are something I have a lot of interest in implementing since Unreal’s default shadows for anything that isn’t a directional light are just so… bad, but unfortunately I don’t have 600$ to blow on an rtx card at the moment. While it’s possible it works as is, it’d take someone with raytracing capable hardware to fire this up and find out to tell for sure.

13

I have an RTX card
but I don’t really know how to build an engine, I’ll take a crack at it at some point

14

it’s pretty simple, just gotta make sure you have all the visual studio components you need installed and you’re pretty much good to go. gl

16

Small change: the red channel of shadow properties now controls the range of dynamic shadows, while shadow range controls the light vector portion of shadows. While this won’t do anything special with the actual shadow data (eg: blur hard shadows), it does give a bit more control since while light vector shadows are always a nice smooth gradient dynamic shadows can be all over the place.

17

I’ve stopped hijacking the existing customdata pins and made new dedicated ones for Shadow Colour and Properties. This should mean that this plays nicely with all the existing shading models, but the focus of this change was to make the subsurface shading model work with my existing changes. Here’s how it looks on my character’s skin.

Right is with subsurface scattering, left is without.

There’s a few minor changes that I made to make it fit my style. One, it only affects shadowed areas, surfaces that aren’t in shadow are unaffected. Two, I amplified the result a bit to make it work well with 1.0 intensity lights.

Enjoy :>

18

Updated OP with an example project, and better instructions for how to start, and some more examples. :>

19

I’ve added support for controlling shadows on DBuffer Decals in the same fashion as regular geometry.

For now it works by hijacking the Shadow Range, Spec, and Roughness pins and using them as the shadow r, g, and b values respectively.

20

Wow this looks great. :cool:

21

reporting that this build doesn’t work with raytracing
https://forums.unrealengine.com/unre…aytracing-4-23
probably because this ? it’s not set to render with forward

would have been nice to have these features with raytraced accurate shadows

22

:>

Disappointing but not exactly surprising. Unfortunately the deferred rendering anti aliasing “solutions” are just so awful that I’ve got very little interest in putting effort into figuring out how to do this in that pipeline, but then again the forward renderer and msaa seem kinda doomed given the glaring issues they’ve faced for years that Epic doesn’t seem interested in moving on, so I may have to make the switch someday anyway.

vOv

23

TAA is pretty nice for certain effects, maked dither and whatnot, hair cards, there’s also settings to control the TAA I saw in a quixel video
https://www.justtodosomethingbad.com…/7/skunk-works
my next attempt at anime type shading will probbaly be based off this