Toon shading model

@motorsep
I checked that one out, and the vids on it look cool but it’s got some issues (jagged shadows, no access to shadow color, and it’s no longer being updated). He said he made it just for fun and has no plans to add more features.

I’ve never made a plugin before, but I’d undertake that task if someone smart could fix all the issues and add some more features to one of those custom shader models that are available.

Hey so I’m running into weird visual bug whenever my game drops in framerate, it happens as high as 55-60 fps so is not an acceptable bug. I have the 4.18 version of Toon Shader and I’ve never seen bug on any other version of Unreal so I assume is where it’s coming from. When the framerate drops there are missing/black pixels that pop up randomly over the screen. They stay in the same location if you don’t move the camera, however new ones jump up and old ones disappear when you move the camera. I have a video here demonstrating it Weird Toon Bug - YouTube Hopefully someone can help me, thanks!

EDIT: So I found the problem, it isn’t toon shader at all and I’m sorry I posted here about it. I had my Pixel Depth Offset constantly set with a scalar parameter to zero, and that would cause problems. I figured out by moving the project to vanilla unreal and realizing the problem still happened, so it’s all good now! Sorry to bother

Anyone else having probs with Marynate’s 4.19-toon-release? Tried building it from scratch twice, and it runs so bad it’s unusable.

Edit: I integrated marnate’s 4.19-toon-release changelist into the official 4.19.2 release (https://github.com/EpicGames/UnrealE…4.19.2-release) and the terrible performance issue is gone.

Dynamic skylights and reflections have been stylized now(in fact, removing the normal influence in the relevant shader and removing the specular color)
It has just been found that using deferred decals can achieve better results than any shadow casting technique. makes me feel that it is necessary to think more about the shadow of stylized rendering.

Im not sure if better with decals (maybe cheaper?), because the right foot and knee are not showing the correct shadowing.

In fact, the shadow of cel-shading has nothing to do with “correct”.
Check out link

Look at the results of those “wrong way to do it”.
They are so similar to the results of using shadow casting techniques in rendering.

The result of the deferred decal is closer to what you would see in the anime when the character leaves the shadow of the building: a nearly parallel line.
(Yes, the projection shadow of the character on the ground is missing, and I’m trying to use other techniques to simulate it)

Does still exist? If so where do I get it? Wherever I get it, does it support 4.18? Does only modify engine code? Can you use within custom rendering so it only applies to certain models? A lot of questions – if doesn’t support 4.18 and above, is there another way to achieve an effect like ? Perhaps make it into a plugin?

Yeah, Nate kindly ported it over to 4.18 and 4.19 for us. I recommend one: https://github.com/marynate/UnrealEn…8-release-toon

4.19 is unusable. Last I checked, if you want it for 4.19 you have to manually port over his 4.19 changelist (here: https://github.com/marynate/UnrealEn…9-release-toon) to the latest stable 4.19 release (here:https://github.com/EpicGames/UnrealE…4.19.2-release).

I recommend these simple settings to start:
Metallic: 0
Specular: 0
Roughness: 1

Toon Blend: 1
Toon Roughness: 0
Toon Step: 1

Ambient Occlusion: v3(0,0,0) way you can stick a texture map or v3 or whatever you want into emissive to get different color for shaded areas.

Edit: I’m not exactly sure what you mean by custom rendering. It’s not a post process. It’s a simple material, so you can use it on whatever you want just as you would any other material. It doesn’t work as a plugin, because, simply put, the code changes are too extensive.

Is still available? Links take me to Page not Found :frowning:

Yeah, it is. You want to use Nate’s version of it, though, because he ported it to all the latest versions. (Read my post above one to find out where his github fork is.)
The reason why you’re getting page not found is because your UE4 account isn’t linked to a github account.

  1. First create a github account.
  2. Click on your UE4 user name (in the forums here), and select Developer Information.
  3. Click on Synced accounts on the left.
  4. Click on the Github icon and add the github user name you registered in step 1.

You’ll then get a confirmation from github, and will now be able to access all the forks people have made of UE4.

Discovered some techniques to make the self-shadow on the character razor-sharp.
Through some minor modifications to the capsule’s shadow, not only can the sharpness and uniformities of the character’s self-shadows be achieved, but also makes the projection shadow of the character on the ground more closely conforms to what we often see in anime—a combination of several ellipses.

any chance you’d want to share the set up you had to make the Wind Waker HD look?

It’s pretty simple: I added a gradient to the emissive channel that lerps between a sky and a ground bounce color based on the world Z-axis. gradient is then masked by a fresnel term. (And I enabled sharp shadows on the light)

@Arnage If you have time I was wondering if you could direct me on how to use the CustomData1 pin (toon roughness) for something else.
I wanted to use it instead to determine how lit the object is based on the light direction to the surface normal.

I’m not sure exactly which NoL I should modify, but I’m pretty sure it’s in one (or some) of these files:
ShadingModels.ush,DeferredLightingCommon.ush BRDF.ush

I tried changing ShadingModels.ush and DeferredLightingCommon.ush (multiplying NoL by CustomData1) but got some strange results.

It’s my understanding that, currently Lit and Unlit areas are set based on a realistic 0.5 hard-coded threshold. So 90 degrees within the light direction is lit, over 90 and it’s not.

Basically, in the end, I want to be able to change the NoL threshold to anywhere between 0 - 1 at runtime, 0 = fully lit, 1 = fully unlit, .5 half-lit half-unlit, etc…

Could you or someone else advise where exactly I need to multiply NoL by customdata1 (and consequently, how to hardcode the toon roughness to 0 or .0001 if necessary)?

So you want to do a NoL offset?
Directly modify the Falloff_Toon_Stepped function in ToonShading.ush.

The use of NoL offset is really necessary.
You want to increase the Shadow bias if you want to alleviate the shadow problem.
After Shadow bias is raised, there will be some light leakage problems.
So you need to use the NoL offset to blacken the umbra

@IOchair Hey thanks a lot for the help, and sorry for the laggy reply. The shadow isn’t really a problem. I mean, it’s realistic right now.
I was just looking for a little more control over the amount of shadow in an attempt to convey character emotion.

See, I’ve been watching some animation lately (specifically old school Dragonball) and have noticed that sometimes only a small portion of characters are shaded (as if the NoL is .9) when they are mad or angry, and sometimes they’re fully lit (as if the NoL is 0) when they’re more happy. I just wanted to be able to control that.

I’ll have a look at Toonshading.ush. I really appreciate the help!

Edit: I don’t think I’m doing right. If I set CustomData y to 0 (in material editor) then the whole mesh is shaded, which is good. But, setting CustomData.y to any value over .01 will give me the default .5 (half shaded, half lit):

(I commented out the branch because I don’t care what Toon Roughness is. I want it always at its default 0.



float Falloff_Toon_Stepped ( float NoL, float3 CustomData )
{
    float Blend = CustomData.x;
    //float Roughness = CustomData.y;
    float Amount = CustomData.y * NoL;
    float Steps = 1 / CustomData.z;
    //float NoLS = NoL * Steps;
    float NoLS = Amount * Steps;
    float Base = ceil(NoLS) / Steps;

    //Branch to prevent divide by zero and save some instructions when Smoothing = 0
    //BRANCH
    //if (Roughness != 0)
    //{
    //    float iR = 1 / Roughness;
    //    float SmoothEdge = (1 - saturate((NoLS * iR) - (clamp(floor(NoLS), 0, Steps) * iR))) / Steps;
    //    Base -= SmoothEdge;
    //}

    Base = lerp(NoL, saturate(Base), Blend * Blend);
    return saturate(Base);
}

Hi,

I researched 3 days before finally saw wonderful thread! I couldn’t wait integrated it. However I have 2 problems that seem to be deal break for our game:

  1. About the ambient indirect light being smooth. The solution given by the forum is to set Ambient Occlusion to 0, and set some color in Emissive to fake indirect light. Do you guy really do ? Our game is mainly night stealth scene, always having an Emissive is not an option, I really want it completely dark in shadows. In night scene if player stays in the dark, the ambient shading is really strong and gives out the low poly of the model, see the character face in the picture. Where could I completely disable the ambient indirect light? I turned off ambient occlusion in world settings, post process volume, and delete sky light actor. It removes the indirect light on static objects. But the character (skeletal mesh) always have indirect light.

  2. Self shadow seems to cause problem in certain angle. It has artifacts on the faces that are parallel to light, see the picture of pink geometries. Post #348 by erodann also had problem but no one answered, see picture of white geometries taken from erodann’s post. Setting the light’s Shadow Filter Sharpen to full, as suggested by the forum, help a little, but still get very noticeable artifacts. By the way, If I turn Cast Shadow off, the artifacts are gone.

Toon render is really crucial for our small budget game, I’d appreciate if anyone could help out.

I tested in both version 4.18 and 4.19(merged to 4.19.2) from marynate, they both have the problems.

Thanks.

@
​​​​​​​
The meaning of NoL offset is of course ADD offset to NoL, not multiplication.
Try add something to NoLS

@renwei2
To solve the problem of indirect light, you still need to change the source code.
The problem of shadow edges is even more difficult. The artifacts on the edge of the umbra are caused by the limitations of the shadow mapping technology itself.
Switching the shadow filter to PCSS with the console r.Shadow.FilterMethod 1 can solve some umbral problems, but at the same time the shadow edges will also be blurred
Or you can increase the shadow bias a lot, but the resulting light leakage problem needs to be solved using the NoL offset I mentioned above.
The problem of shadow blurring can be solved by increasing the shadow map resolution and reducing the dynamic shadow distance

Considering that everyone is actually facing similar problems, I have made a list of the problems that I have encountered that have been resolved and have not yet been resolved.

  1. Character Face Shadows (resolved): Added a Custom Material Node to modify normal in the engine.
  2. Skylight, indirect light, and reflection are not stylized (resolved): The effect of normal on skylight, indirect light, and reflection is removed in the source code.
  3. Outline and inner line (resolved): Add Custom data control node, specifically use two Custom data to control the post-process material.
  4. Self-shadowing accuracy (resolved): Use ray-traced capsule shadows instead of shadow maps, combined with normal edits.
  5. Character receiving shadow accuracy (resolved): Use a decal instead of a shadow map (remove the fade-out of the deferred decal X-axis).
  6. Character contact shadow (unresolved): Shadows on small areas of the character, such as the projection of hair on the forehead.
  7. Stylized eyes (unresolved): Porting some properties of the default eye material to the Toon Shading Model, adding a number of controllable fake highlights.
  8. Stylized Perspective (unresolved): Achieve a few common perspective errors in Anime.
  9. Stylized metal (unresolved): Makes metal objects in the scene more stylized.
  10. Stylized edge light (unresolved): Rim light around static objects in Anime.
  11. (And more to discover)

@IOchair
Thanks for your answer, and the list! I’ll try while still learning some of the terms (graphic programming noob).

By the way, do you think all these effort we’re doing is the right direction to go production? It looks like the more we dig into it, the more problem we discover. It worries me because I’m working for a commercial project and not just for experiment and learning. It’d be great to hear your opinion as you seem rather experienced.

Thanks.