Toon Shading Models, Stylized Rendering Experiments

Hey! I was wondering if this shader is still available on GitHub? I linked my epic account to my Github account and I’m still getting the 404 error, does anyone have any suggestions?

This shader looks absolutely incredibly by the way! It really goes into depth compared to a lot of toon shaders I have used, it is just what I was looking for.

Hi, is it possible to get multiple tones shading with this? I don’t really understand how the materials in the example project work and I’m not sure if that project is outdated since it’s from 2018. Would be really good if I could have a 3rd tone for shading as it really helps with depth on characters when used right.

Can the UE5 version be supported? QAQ

1 Like

I’m currently working on getting it work in 5.1.

I managed to adapt the code to new way of UE5 handling GBuffers, resolving issue of custom shading model appearing as solid black in deferred rendering.

However I still can’t get the toon shading to display correctly just yet. There are also some errors with vector truncations, which I managed to resolve, because some functions returning scalar float value is using vector types for some reason.

1 Like

In case you’re wondering, or anyone else willing to give some input while I’m wrapping my head around it, here’s my fork implementing it:

https://github.com/EscapeEntertainmentTeam/UnrealEngine

The fork has the custom shading model work the same way as kusogaki77’s 4.27 branch. Specular attribute doesn’t work yet, however it can be remedied by adjusting specular offset/scale values.

2 Likes

In my custom 5.1 branch, I’m currently stumbling upon a puzzling issue where toon shading model won’t appear correctly, as in it only reacting to indirect lighting.

I had a suspicion this was because LightAccumulator function in GetDynamicLightingSplit() in DeferredLightingCommon.ush got split into its own function, AccumulateDynamicLighting(), per commit 7b4f4de. This wasn’t the case with 5.0 and prior versions, so other examples of custom shading models in 5.0 are close to what this Toon shading model is doing. However this new function eventually ended up called in GetDynamicLightingSplit() anyway, so it doesn’t makes sense for me if that’s the failure point.

I’ve tried to closely follow the commit for 4.27, and trying to adapt to 5.1’s code changes. i think I have iron out all C++ code changes (at least ones documented for 5.0), so I don’t know any more probable bad spot that stopped this from working properly. Any help is appreciated.

EDIT: I was far off. Turns out I goofed up the code to include Toon shading models to use the custom data. Not to mention additional material attributes have to be reconnected if you’re migrating from UE4.

Now it works like it should, all the nodes, except the vanilla Specular attribute doesn’t seem to work yet. But since it’s not too problematic, I’m going to call it there for now.

1 Like

hi, the softer bands or the ambient occlusion artifact thats happening kinda breaks alot. Here are some implementations which have no softer shadows: Reddit - Dive into anything

however those are 4.25 and 4.26 and nobody made a fork for 4.27… ( i dont know how to do it myself) however, would be awesome if there was a way to get a solid shading here.

As far as engine code goes, I don’t think there’s not much change in regards to the shader codes from 4.25/4.26 to 4.27. So you could simply fork UE repo and merge/cherrypick SirCodalot’s custom shader model that you linked. It should be going without major hiccups.

As for 5.x, there are major changes to shader related codes, but since you’re asking for 4.27, I won’t go into details for 5.x on this reply.

5.0 https://github.com/envieous/UnrealEngine-SelShader
4.27 https://github.com/envieous/UnrealEngine-SelShader/tree/4.27-SelShader-Deferred

Nice Ichinose! I stumbled trying to implement this in UE5 so thanks for the port. :slight_smile: It seems like projects should transfer over fine. I still use the original specular in very small amounts, a little torn on it but sometimes it can add some nice subtle gradients in the original.

I’m no .ush expert but one thing I did using this shader recently is fighting off self-shadowing on characters while keeping a HQ environment shadow and mesh light colors with no self shadow interference. It relies on toon subsurface color to do the coloring on top of existing base color+like 0.05 emissive or so.

Sorry if it’s a little long and off topic as a whole, but I think the results are nice when used in a megasaturated 2000s-ambient style lighting setup, and somewhat relevant as a way to use this shader.

Guide:

  • Make a copy of mesh that casts shadows on lighting channel 0 and set it to hidden shadow/not visible. I use enable dynamic inset on skel mesh, looks great on otherwise not so great CSM. (Not trying VSM yet)
  • Original mesh on light channel 1 only.
  • All point/spot lights now need to be split if shadows are desired. (!)
  • Place two lights in the same location: One for channel 0 (environment, shadows), and one for channel 1 (character, no shadows).
  • I’m using many movable lights for all this and it’s ok as long as one writes good enable/disable distance check functions since lights on the other side of the planet are adding up 0.02ms calls.

Now you’ll have really pretty colored lighting on meshes with subsurface colors as well as a HQ shadow with no self shadow artifacts on the mesh itself :slight_smile: the good part about this is that you can tweak each light nicely to fit with your toon character and how it plays into the environment.

Double mesh hidden shadow is nothing new, but one can extend it to the environment as well:

Usually one would do sun-line trace+emissive modifier like in genshin impact to change the toon char brightness. This method is fine if used with good art direction and manual control, but for the sake of this setup, let’s try two directional shadowed lights (!)

Split the dir-light into two equally powered lights operating on channel 0 and 1+2, respectively. I know two shadowed dir-lights is a big yikes, but performance has been surprisingly fair and worth it compared to nasty csm shadows in my project.

  • Directional light 1: Intensity 0.5, channel 0, shadows on.
  • Directional light 2: Intensity 0.5, channel 1+2, shadows on.
  • (One can switch ratios up slightly to alter shadow strengths between characters/environment. If you want slightly less intense player shadow, use lower intens for d1, like d1=0.4, d2=0.6)

Fully lit areas will be doubly lit than the player operating on channel 1 only, but with a good setup in the toon material and all the world materials one can get around this rather easily and it will blend together nicely.

The trick to the open world shadows is in channel 2 on large meshes. The player shadow will be equal in strength to the ground below if using either 0 only OR 0+2.
(Use channel 0+2 on landscape).

Mesh light channel settings:

  • Channel 0: Shadows the environment (half), light passes through characters. Use on environment and hidden character shadow meshes. Not bad for small meshes like a fence if you don’t want aggressive shadows on a character’s legs
  • Channel 1: Use only for characters to receive lights, disable shadow casting on c1 always unless you managed to make it look good - I couldn’t.
  • Channel 2: Shadow both player and environment, by half.
  • Channel 0+2: Fully lit mesh, strong shadow, player shadow.

Hope it’s helpful to anyone else who likes using this shader. :sweat_smile:

1 Like

My previous approach was not that great, but I found a way to remove self-shadows while keeping the light color influence as well as a shadow on the ground, all on one light channel. The downside of this is of course that no shadows effect the toon mesh anymore, so it is best used with point lights only or in combination with some sort of trace manager that checks for sunlight.

(Point lights only, cast shadows, channel 0, no directional light, toon offset 0.5):
celstatue2

Lowering the subdermal color can create a more shadow-y style, and tweaking roughness and toon specular also looks pretty good with this tweak, in my opinion:

In DeferredLightingCommon.ush, line 351 (4.27 branch):

// Only get shadow terms if not toon shader
BRANCH
if (!( GBuffer.ShadingModelID == SHADINGMODELID_TOON || GBuffer.ShadingModelID == SHADINGMODELID_TOON_SKIN || GBuffer.ShadingModelID == SHADINGMODELID_TOON_ANISO || GBuffer.ShadingModelID == SHADINGMODELID_TOON_HAIR)) {

		GetShadowTerms(GBuffer, LightData, WorldPosition, L, LightAttenuation, Dither, Shadow);
		SurfaceShadow = Shadow.SurfaceShadow;
		LightAccumulator.EstimatedCost += 0.3f;		// add the cost of getting the shadow terms 
}
2 Likes

this is awesome, will try to replicate it, thanks!

1 Like

Since I recently have access to realtime ray tracing capable hardware, I decided to test the toon shading with RT. However, with RT shadows, the shading model results in incorrect (and janky) shading:

Lumen GI does help a little bit, but not by much:

For comparison, with ray traced shadows disabled:

I’m inclined to think it’s due to how the shading model itself worked with the lighting that it fall apart with raytraced shadows. A workaround for this would be to disable RT shadows, which also affect RT shadows on all object, as there seem to be no way around isolating it to just the shading model.

And why even ray tracing? Because why not. :yum:

2 Likes

Hi, thanks for UE5 integration to bring this toon shader.

I have question related to ShadingModelID limit.
As far as I know, this shading model ID is stored in 4 bit in gbuffer. (Together with Selective output mask) → it make up for 1 channel in one of the gbuffer.
Now if you extend this beyond 15, it will have some complication especially ray tracing, anisotropic material.
So far have you encounter this issue? I am curious if there is better way to extends this multiple Toon shading model without breaking limit.

1 Like

Hi,

This may also be achievable with our new Strata tool as you develop your art style, details here - Strata Materials | Course

Thanks,
Jon

1 Like

@Jon.Cain
Hi, I decided to give a tried Strata Material, it’s definitely possible, but it has some notes from my side. (besides shader compilation for strata is quite long).

  • Adding STRATA_BSDF_TYPE_[xxxx] is currently limited to 7, increasing this definitely will change how HEADER store BSDF type. So I don’t think it’s a good idea.
  • Adding property to STRATA_BSDF node, something like enable toon. So you can branch all BSDF with toon flag. Example: BSDF_Slab with Toon, BSDF_Eye with Toon, BSDF_Hair with Toon. This flag can be easily expanded and stored on STRATA BSDF Context State.

This is something that legacy material has a hard time doing since material mask is only limited to 4 bit and it’s fully occupied.

So the only big drawback that I think as for now if it’s required to extend STRATA_BSDF_TYPE beyond 7, the implication on pack/unpack Header and other stuff.

Did you try using VSMs?

Hey @Ichinose ! Thank you for updating the branch to 5.1, I have been looking through it and can see all the hard work you have done!

For anyone who is interested, I found some articles about Toon shaders whilst doing a bit of a Google search. Only problem, they are in Chinese :stuck_out_tongue: Here is one I found: https://zhuanlan.zhihu.com/p/551343308

I’m running into an error with this on 5.3. The shadow is incredibly sharp and does not have the smooth falloff from doomfests example.