Trying to make Toon Shader Material on UE4

I’ve copied some of the earlier code from Headstub (which he said was buggy) but I actually quite liked. And am getting the same issues as described by (as in, everything in my scene just appears as if there is no lighting) anyone else from the original “cel shader crew” have their cel shaders broken?

I’ve haven’t got this bizarre effect though!

So, I’ve been trying to re-create the celshaded effects that have been posted in this thread. I’ve gone through and re-created almost everyone’s different shaders at different stages and haven’t really managed to achieve anything approaching a cel shaded effect and I am not exactly sure what I am doing wrong. To be honest I don’t really understand shaders very well so when I’m altering these material graphs I do feel I’m guessing. Anyway, posted below are some my attempts and results that I got. If you can spot any errors I’ve made I would greatly appreciate it. I’ve checked them agianst the images people posted multiple times so all I can assume is that there is some editor setting I have missed that I have no knowledge of or I have somehow messed up some sort of setting.

This is a re-creation of DevilboxGames’ network (post #72) - I take this network, create a material instance and apply it as a blendable in the misc section of the post-process volume. It turns the screen black.

Network:

(Doing it in multiple posts due to upload size issues)

Then I tried to re-create Headstub’s network (post #50) - Same procedure as above, but this one doesn’t use textures just the band numbers from JvTheWanderer. This doesn’t turn the entire screen black at least, but just makes object with setcustomdepthbuffer enabled to be a sort of flat grey color.

Network:

Result of above network:

Finally I tried breaking the cel shader effect off entirely as it its own thing and it sort of achieves the start of the right effect but the light is reversed and the rest of the scene is dark. I’m not exactly sure where to take this.

Network:

Result:

I managed to get something sort of working but not entirely sure WHY it works…

I’ve mixed the cel shader from before with the specular/emissive custom depth buffer from earlier in the thread. However, weirdly, nothing is actually set in that depth buffer but it still manages to produce the cel shaded effect on these particular objects. I also found that if I set each object to just a flat colored texture and turned specularity way down it had a much better effect.

Interestingly I found in the above one that if I remove all light sources in the scene everything still remains lit. And if things are actually added to the custom depth buffer they get shaded in the same way as if there were no light in the scene. So removing them from the depth buffer just allows them to receive extra light. Is there a way to fix it so they actually only receive light that is there?

Oh, nice work.
And now this is fixed?

Continue to update your post please to see where you are.
Thanks.

If you copy the last network I posted you’ll get a cel shaded effect but the cel shading I have been able to achieve seems to have baked in lighting. Even with zero light sources in a scene you will still have illuminated objects. And the bright / dark sides of those objects will remains consistent regardless of where light sources are in the scene. I don’t know why this happens, I think it has something to do with multiplying by world normal but, to be honest I am stumbling around in the dark here, the documentation seems very lackluster.

I can’t for the life of me work out what all the options in SceneTexture mean.

For example how exactly does the DiffuseLight part of scene texture work. It displays positive data whether there are lights in the scene or not. Even when I remove all lights and plug it into the emissive outlet of my post process material everything gets colored. Surely that shouldn’t happen if there is no light in the scene to be diffused. Also, all the calculations are down using World Normal which I think might be the cause of the unitary direction for all the light as the vector normal for the world will always cause the light to shine in a single direction.

If anyone has some insight on explaining these parts of the SceneTexture it would be great.

Bonus points if you can tell me what PostProcessInputs do!

Having done a bit little bit more experimenting (probably my last for tonight) I noticed that actually world normal and diffuse light are completely direction agnostic so I don’t understanding how any of the shaders up until now have corresponding to a positioned light source at all. So far I’ve only been able to light from a single direction as the diffuse light and world normal stay static.

The global illumination happens because of a clamp to a 0.1-1 range you’re doing shortly after the dot product in the “light banding” portion of the material.

By clamping the light band range above 0, you are assuring that the very lowest amount of lighting is NOT totally dark. This produces the cheap “global illumination” effect that you’re seeing; no matter how unlit your scene it has some lighting since you’ve set the brightness of “total darkness” to some non-zero number.

Having changed the clamp I don’t think it was the cause as when I re-applied the shader there was still low level illumination (see below). Also, I’ve linked here a postprocess with JUST diffuse plugged in and you can see that there are no lights in the scene and yet there is still diffuse light color data being displayed. My next intuition is that the postprocess material must be providing SOME illumination but if I remove the shader the scene displays as if there were no light, so even an incredibly simply shader that just uses diffuse light color data as its only node still artificially lights.

With Clamp 0:

With Just Diffuse:

Also, my above point about origination of light seems a bit worrying I don’t see how previous people in this thread could have been calculating the direction of the diffuse light using only that and the world normal (which is static as far as I can tell).

So nobody can solve this problem?

This is awesome stuff! I hope somebody can help you with your problems.

Yes, that would be nice.

So nobody have the solution?

Not a solution to your problem, but I have an alternative method for the cartoon shader.
It’s not really finished, but I won’t do much on it, because I just did this for fun on the side. Maybe someone can improve on it or gain some ideas with this.

The basic Idea was to take the Scene and the Base Color of the Scene and divide them so that you can get a multiplicator for the base color.
everything below 1 is darker than the base color (shadows), everything above 1 is lighter (highlights, emissive)

So the first step is
(PostProcessingInput0 / Base Color)

This output now needs to be modified. I seperate the shadow and Highlight components of this output by having one branch go through a Clamp (Min=0, Max = 1) node. Then I multiply it by the number ob bands I want to have for shadows, use Ceil and then Divide them to undo the Multiplication. This would create your Multiplier for the toon shadows. I also didn’t like to have completly black shadows so I used a Max (0.1) node to make everything below 0.1 to 0.1. You can change that number untill the really dark shadows are the way you like them.

For the Highlight part I subtracted 1 from the output of the division and then used Max (,0) to remove the negative part. So now he highlights go from 0 to 1. This way I can later just add them to the shadow multiplier before I multiply it back to the base color. This Highlight part also needs to be modified. The way you do this is similar to the shadows. Multiply (this time make this number below 1 because the highlight multiplier can have high values, just tweak it untill you like it) and them convert them to int using Ceil or Floow, then divide with the same number you used for the multiplication to undo the previous Multiplication. I then use a upper limit for the Highlights using Min.
Since the Highlights add color I had some strange looking behaviour, so I had to desaturate the output before I combined the shadows and highlights.

The last step is to add the two components and then Multiply them with the Base Color. This is your toon shader.
Just to give it more plasticity I also added a Lerp so I can blend the original shading and the toon shader, so get some of the more detailed shadows for edges. And finally I also used Custom Depth to be able to control which Assets should use the toon shader.

The result looks like this:

http://91.228.52.204/~Tomura/images/ue4/toonshader03.png

Problems:
I think for the shadows this works very good, but the emissive part definetly has problems since I had to desaturate in order to keep the colors from going crazy. If someone can figure out a way to keep that from happening while still keeping the color information that would be great. Another problem would be parts that are completly Black in their base color since this would mean a 0/0 division. Clamping can help to make this look ok, but the is a loss in information. Maybe using addition instead of Multiplication could be a better way for the highlights/emissive parts.
Another solution to make a toon shader might also be to just limit the color palette and use the nearest color. I haven’t tried this, yet.

Another solution I’m thinking of could be converting the colors into HSV format. With this RGB solution regardless of additive or multiplicative there were problems when I had different colored Highlights (e.g. white highlight on Red surface means it just adds blue and green). With HSV you’d have the lighter or darker than Base Color information in the V component. So you could just modify that then use the toon V and convert it back to RGB.
This way there could be way less problems with new colors being added in a strange way after using ceil or floor.

I hope this approach that is a bit different to the other one that was posted here is helpful. Maybe someone an find a good solution for this.

A post-process effect is how I’ve done it. I can confirm it works really well with edge detection.

It looks like you’re trying to achieve posterization. This is actually easier than it seems. This is the method I used: [Shader Library] Posterization Post Processing Effect (GLSL) | Geeks3D

That can be converted to blueprint in 4 or 5 nodes.

Thanks you very mush!