Cel shading mods to M_PostProcess_Base from Epic's Stylized demo project

Here’s a post on some modifications I’ve been fiddling with using M_PostProcess_Base, the post-process outline in Stylized, to make it closer to the effects I want (more outline, vary line thickness and detail with distance, red facial shadows).

I removed the part in the lower right that creates the paper-like material, and moved the PostProcessInput0, Mask, and Blank Paper Color over to the Lerp connected to the Emissive Color, deleting the Post Process Blend Weight and other Lerp. Keeping the Blank Paper Color allows you to connect either that or the Mask to the Lerp, so you can see the final render or just see the outline:

The original M_PostProcess_Base keeps the same line thickness regardless of distance, so distant objects get a heavy-looking outline.
The output of Min replaces the Constant “Line Render Width” of 1 in the “Square Samples” segment:

I did a similar thing with the (unlabeled) depth calculation in the center (swapping the Subtract arguments so the Edge Angle Fallout is no longer a negative number), and replaced the Edge Angle Fallout with another distance modifier, so closer objects will need less of a depth difference to generate an outline and farther objects will need more. This gives close objects more outline detail and prevents far objects (like fingers on a hand) from having too many dark lines:

Then, taking an idea from 3D Modeling & Texturing: [Unreal Engine 4] Post Processing , I added what I think is a similar edge detector using normals:

They connect up like this to combine depth & normals outlines – for testing, you can hook up either of the first clamps to the 1-x to see just depth outlines or just normals outlines:

Finally, here’s a complete hack to change shading on faces from gray to reddish – the “Face Color” is the skin color being used (though, for some reason, the saturation needed to be doubled; I’m still trying to figure out the difference, it doesn’t seem to be due to lighting).

What this does is:
Takes the Base Color and subtracts it from the skin color, to see if this looks like it’s part of a person’s skin color;
Takes the Base Color and subtracts it from the PostProcessInput0, to see if the something like a shadow is making the pixel darker;
Create a “redder” version of the shadow difference calculation by scaling down the redness (since shadows are darkening pixels, scaling red down has it remove less redness);
Check if both the diff calc shows the red being reduced (i.e. a shadow) and the Base Color looks like a skin color pixel – if so, output the redder shading for red shading, otherwise output the original for the original shading.
The Add output feeds into the Lerp that goes into Emissive Color:

Here is a comparison of the original shader vs. my modifications. Also note that she has a chin line and inner blouse lines:

Looking good man, thanks for sharing!

This is excellent. I’ve been playing with the stylized showcase’s outline renderer a lot, and I disliked many things about it (such as how very close objects eventually see their outlines fade, outlines don’t scale very well with distance, etc), definitely going to try implementing some of this!

You should also check out the cel shader being worked on (in the Content Creation forum); they go really well together.

Here’s a slight improvement to the reddish facial shadows – it uses a lerp to fade from red shadows to gray shadows instead of the first “if”. The “if” made it change from red to gray instantly at the pixel level, which can create gray pixels near things like mouth lines. The lerp avoids such artifacting by making it fade from red to gray gradually.

Also, I’ve moved this over to this forum post, combining it with flat shading: