Toon Shading Models, Stylized Rendering Experiments


Repos & Downloads:
IMPORTANT: Unreal’s source code is not a public repository, so all forks of it (such as this one) also are not public. You HAVE to read Downloading Unreal Engine Source Code to link your Epic account to Github or the links will give you a 404 error.

@kusogaki77’s fork (4.27): https://github.com/kusogaki77/UnrealEngine/tree/4.27Toon
Github fork link (4.20) (Outdated): https://github.com/SaimonMa/UnrealEngine

Example Project: ToonMaterialProjects - Google Drive


Hello! I just wanted to share some custom shading models I’ve been working on. I’m aware of the shading models that have already been posted on the forum (1)](Toon shading model - Game Development - Epic Developer Community Forums) (2)](Toon shading model for UE 4.20 - Rendering - Epic Developer Community Forums), but I wanted more control over the shading on a per-material or per-pixel basis, so I decided to fork the engine, teach myself HLSL, and take a stab at it. The idea for these models is to have soft indirect lighting but controllable direct lighting so it’s possible to mix the default materials with the custom stylized ones. Note that the code does include a version of stororokw’s anisotropic shader ported to 4.20 since I wanted to build on top of the anisotropic shading to create a toon version.

I’m going to post my upcoming experiments as I try making some manga-style stuff in it, but I’d appreciate any feedback.

Since Offset is a parameter in every Toon material, you can drive it with some screen-oriented masks for halftone or crosshatch effects:

Also, materials have separate controls for specular and diffuse, which I find useful for stylized stuff:


If GBuffers weren’t limited, I could have just made a ubershader with all the features in one shader. Since there’s not many unused GBuffer channels, here’s a couple shading models that (hopefully) cover a wide range of uses, and their controls:

Toon:
General-purpose toon shader. Most control over the shadow and highlight locations.

  • Offset: controls the location of the shadow terminator. For faces, a vertex color channel or a texture can be used to push shadows away from the front of the face, or to increase how quickly the shadow appears in crevices, etc.
  • Roughness: controls the range of the shadow terminator. Roughness has been decoupled from specular. Max hardness for the terminator is reached at 0.5 roughness. Reflections are still rendered and will be visible at values below 0.5, if you want them.
  • Shadow Lightener: a float that lightens the shadow towards the base color of the material.
  • Specular Offset: Same as Offset, but for the specular highlight.
  • Specular Range: Same as Offset, but for the specular highlight.

Skin:
Skin toon shader with an adjustable SSS color. SSS calculation is based on the Subsurface shading model.

  • Subdermal Color: Takes a color. Note that the value for this color drives the strength of the effect, not the Opacity slot like in the standard Subsurface shading mode. I recommend multiplying your color or color texture by a scalar for easy iteration. If you are using a texture to drive the color of the SSS, I recommend multiplying it by a scalar, especially if you want really low SSS values. You’ll get more control this way without texture compression crunching your values too much at low value ranges in the texture.
  • Offset: Same as Toon.
  • Roughness: Same as Toon.
  • Specular: For skin, the specular intensity and sharpness is linked, higher specular values also tighten the specular lobe, and vice versa.
  • Specular Offset: Same as Toon.

Hair:
Based on the built in Hair shading model for Unreal, so it will be fairly similar- though it lacks a Backlit parameter (since we’re out of GBuffers again…). Has dual anisotropic highlights.

  • Roughness: Same as Toon.
  • Specular: Same as Toon.
  • Specular Lobe2 Strength: The second lobe is broader and more saturated. This parameter drives the strength of it
  • Tighten Specular: Higher values will make the specular highlight narrower.
  • Scatter: Works the same way as the built-in Hair shading model. Increase this value to lighten the hair.
  • Tangent: Takes the first two values of a vector to assign the anisotropic direction of the hair in UV space, same as the built-in Hair model.

Toon Anisotropic:
Anisotropic shader based on this stororokw’s code, but ported to 4.20.

  • Roughness: Same as Toon, but also drives the range of the specular highlight.
  • Offset: Same as Toon.
  • Specular: Same as Toon.
  • Anisotropy: Controls the amount of anisotropy of the material.
  • Anisotropic Rotation: Rotates the anisotropy of the material.
  • Anisotropic Roughness: Drives the roughness of the anisotropic specular independently from the material’s lighting range.
  • Aniso Tangent: 2D vector that controls the Anisotropy using maps like these. This page shows what kind of data the shader expects.

Some of the known issues include:

  • odd behavior with point lights with inverse square falloff and area lights with 0 width and height dimensions. I’m not sure what causes this, so if you have insight it would greatly help. Current workaround is to make sure point lights have inverse square falloff turned off.
  • for Toon shading model, having a metallic material makes the specular color unable to to be turned off. I’m sure I can figure out where it is multiplying metallicity by specular color somewhere, so I’ll fix this in the future.
  • completely untested with the forward renderer, but it almost certainly going to be broken. If you’re interested in this and want forward rendering support, I would appreciate some pointers in the right direction as to what files to poke around in.

Also, I couldn’t have done this without these guys’ tutorials paving the way:

17 Likes

@kusogaki77 brought up a couple use cases where being able to fully put a character in shadow or light using the material would be useful, so I extended the range of Offset to allow for this. (thanks!) This is the new range for Offset, note that if you’ve made materials you might need to update your scalar values for offset. if you used a texture to drive offset, I recommend using a Lerp to adjust the range of offset for that texture (I use that on my test cases):

1 Like

I’ve added some controls for flatter shading for ToonSkin and ToonHair. Note that this changes some pins in the material editor, and will require the engine to be partially rebuilt. Some parameters for ToonSkin might need to be moved around, since SpecularOffset’s pin is moved to another material pin for ToonSkin.

Skin:
Specular Offset and Specular Range are now unlinked and can be controlled separately. Since SpecularRange and SpecularOffset actually share the same GBuffer, SpecularRange only has 5 steps. More info on how the encoding works in a later post. SSSMode is now a switch that allows standard SSS shading or simple SSS shading that only affects the shadow color. Personally I find adding a low emissive value to the entire material or a fresnel helps this mode a little, since the lit areas receives no scattering, but it depends on how much scattering you want to emulate for skin.

Hair:
Roughness now affects the hardness of the specular and surface shadow rendering.

1 Like

  • Surface shadows (this was broken by accident before): Note: artifacts might pop up if you use cast shadows, low roughness values (hard surface shading), and a shadow offset greater than 0.5. This is because cast surface shadows on objects can’t be pushed farther back than the point where offset = 0.5 when using hard shadows. However, an offset value of 1 will still turn off cast surface shadows and surface shading on the object, if that is what you are going for. Will document this better later.
  • Inverse square falloff: I haven’t tested this with spotlights yet, but inverse square falloff now works with point lights.
  • SSS mode 1 update: SSS mode 1 now uniformly applies transmission lighting over the entire object instead of just the shadows. I think this looks better for toon skin, and removes the need for emissive lighting hacks.
1 Like

@IOchair
Hey IO,
As you mentioned, you added a Custom Material Node to modify normal to reduce the face shadow, I am very interested in that since the project I am doing is also facing the face shadow problem. If you can explain your method to modify the face normal in Engine, I will be very much appreciated. Attached is picture showing the face shadow I currently have for my toon Shaded character.

1 Like

After a few days of attempting to push away the face shadow, I found a quite easy way to achieve a more 2D like effect. I modified the skin material shading mode to subsurface and added an extra direction light which will rotate according to the position of player camera. Since my project is a VR project so this method is quite handy.
I am now trying to understand the reason causing this shading difference, maybe is due to the extra light reflection under subsurface shading mode?

Followings are the pictures comparing the results.

1 Like

What do I do with this? Using forward renderer and a project upgraded from 4.19 I think to this. I’ll try a fresh one I guess.

edit: tried a fresh project and same thing

Hey all, just wondering if anyone else has had this issue
I’m getting weird shadow lines with the shader, even without textures
The shading also falls apart completely on large objects, getting these polygonal facets
Its interesting that these issues don’t happen at all with the specular highlight
Any ideas why?

Hi! I recently got the toon shader compiled in 4.22, and I was going to do some tests to try and get raytracing to run properly, and for some reason, none of the toon shading is appearing in reflections.

Any idea on a fix?

Is there a way to get uniform flat shadows like the light? The models look great in the light or in half shadow where there’s lots of clean contrast, but in mostly dark areas the model’s geometery is visible dirtying up the shadow

1 Like

Well, it sorta works, very Killer7, I tried modifying the Shadow Color parameter to make the shadows something other than pure black but it just washes out the entire character

1 Like

I’m I going crazy or a lot of replies disappeared? I hope this is not the end of this Branch.

1 Like

The same as you.I hope they can come back ASAP

1 Like

I dunno what happened to all the pages, but I noticed the link to the active branch is gone, so here it is for anyone looking
https://github.com/kusogaki77/UnrealEngine

1 Like

Is there a discord or something where more active conversation takes place? I’m really excited for a potential fork for ue5 down the line

4 Likes

I too would love to follow these convos, a fork starting off early in UE5 sounds pretty dang tasty to me.

1 Like

This custom solution works perfectly. Thanks so much for this @doomfest and all the other contributors, seriously it works great and I really appreciate you sharing it and all the info online. It has been a great learning experience diving into a custom engine solution for this.

1 Like

■■■■ our several replies disappeared when ue4 forum changed.
include mine.
anyway How can i import this toon shader for ES3.1 mobile?
kurosaki’s git work perfect but i want android render like genshin game.
anybody know how?

1 Like

@kusogaki77 sorry sir bump! i want mobile shader! and our comments are disappeared!

1 Like

@poiqwe

I’m still in the middle of moving, and will be for another month or 2, so I won’t be updating the fork any time soon. I definitely plan to, though.

I’m not sure there’s an easy way to get these shaders working on mobile, as mobile simply doesn’t have access to the full array of parameters that these shaders use.

I messaged VictorLerp and Amanda Shade in regards to why so many posts have been removed, and whether they could add them in. We technically could manually add them back by using internet archives I believe, but that would be a lot of tedious work.

The last time this engine was updated, to 4.26.2, was by drydev. After merging the changes I realized all the binaries were included. They should not be. When I have time I will remove the binaries. Lastly, it seems like 4.26.2 was the last update. So after I, or someone removes the binaries and verifies the changes on 4.26.2, I’ll look into adding the toon shader models into 4.27.

1 Like