Cell shading without post processing or forward rendering

I’ve been thinking of a way to get cell shading working in unreal without expensive post process effects, code plugins or forward rendering so I can take advantage of unreal’s deferred pipeline (multiple lights, etc.)

After much experimentation I came up with a method that alters the vertex normals of any mesh to achieve a cell shaded effect. It’s a bit of a hack and not quite perfect yet but it’s a lot cheaper than doing full screen post processing for just a character. Right now there are two different shading styles i’m torn between:

Without Cell Shader

With Cell Shader

Style 1:

Style 2:

Multiple Lights

Any feedback would be much appreciated! :slight_smile:

This model is really low poly and messy with pretty bad normals (It was a free download from some obscure site) so I imagine it will look better on a cleaner mesh. I haven’t got around to the outline stuff yet but that should be fairly straightforward I think. The model’s texture is a very basic flat color map, the cel shading simply comes from the vertex normal tweaks in the shader.

Will post more info tomorrow. :slight_smile:

Please check our Character Cel Shading Pack in the marketplace.

https://www.unrealengine.com/marketplace/character-cel-shading-pack

Uh… That’s the most shameless plug I’ve ever seen. The guy asks for feedback and you tell him to buy your pack?

Hi Jonimake. Please read our comment again. We didn’t tell him to buy our pack. Don’t misunderstand our intention. Thanks.

Well as a developer who is also making a cell shaded game, I think it terrific! Having support for multiple dynamic lights at once is always great :slight_smile:

Hi bud!
May I ask you to share some hints about the tweaking of the vertex normals of your great cel shader?
I’m trying every possible way to have cel shading, both postprocess and material… I personally prefer to have full control on my object instead of using a postprocess. I now have a shader that performs the shading but, as you surely imagine, with just one light vector replicating the main directional light of the scene.
Be able to keep all the ligths would be fantastic and your system sounds an interesting idea :slight_smile:
Thanks in advance for any help or suggestion you would like to share :slight_smile:

Daniele

Hi guys, sorry I totally forgot to come back and post my findings here. So basically my solution is not ideal visually but comes close to emulating that cell-shaded look without being too involved or restrictive.

Below is what I do to the vertex normals of my shader to achieve the look:

Simple way:
0e109bdabf749bce0d947ca511fa9207d058f516.jpeg
This gives you one subdivision of cell shading and is the most basic stylized look

This graph gives you control over the number of shading layers and also allows you to blend between the smooth normals and stylized ones:
60c56b8d7c7a9727b719d9d629e2e67a38724e6f.jpeg
7b4cdf1ccfda175a3f169b83f74a88992ccd680d.jpeg



so it’s not really true cell shading but a close emulation that works with any number of dynamic lights and light propagation volumes. This can also be used on lightmapped objects too but is less noticeable.

Have Fun! :slight_smile:

2 Likes

Ok so I decided to come back to this today and do a little more research and see if I could improve things a bit and I think I found a nicer solution:

654899ef02a905ee9a229e540d8bcf8d7a43e35f.jpeg

Overall I’m pretty pleased with the results - again this works with any lighting, LPV’s or static lighting, the only downside to this is that you can’t tweak how many cell shade levels you have but I think it’s a good alternative for a more typical cell shaded style:

Let me know what you think or if you have any questions!

4 Likes

This is incredible. I will definitely be looking into using this for my next project, as using custom scripts in the engine (which is what I’m doing now) can make things very complicated.

Having a fully working cel shaded system in vanilla unreal would definitely make things much easier.

Hi!
Your result is really good… I should give it a try :slight_smile:
Mi solution at the moment was to take the abs values of the VertexNormals and use each channel as UV for a band/ink textures, make a new float3 from the 3 TextureSamplesand then multiply it for (VertexNormal/abs(VertexNormals)) to retain the sign of the components.
This simply makes all the VertexNormals as discrete values so the surface can react to the light as if polygons share the same normals… the final look is very similar to a cel shading.
Pros: it’s very easy to adjust number,thickness and smoothness of the ink-bands… And of course it reacts to all lights in scene.
Cons:it gives me strange result somewhere because it tends to generate squared shaded areas.
I don’t have the engine at home but I’ll study your solution and try it on mondey as soon I reach the studio :slight_smile:
Great job bud! :slight_smile:

Hi!
I’ve tried your solutions (both of them) this morning and my opinion is that your last shader gives you nicer results but very different to the previous in terms of “what polygons” are shaded and how.
Your method with the ShadingLevels parameters gives the mesh almost the same look of mine.
It’s hard to tell wich one is the most correct regarding to the lights in scene…
BTW, i’ts a great idea! :slight_smile:

Hi Berna, glad you found it useful. Your technique sounds interesting, care to share your shader graph? I’ve love to see what results you got with it :slight_smile:

Yep!
This is the graph of my function (I love to use material functions and create a library so I can always mix them up and reuse useful shader code)

While this is a comparison between my code and yours (please note that now I’m referring to your “second to last” solution): as you can see the result is almost the same. I slightly prefer my approach because a texture for the ink levels allows you to control better the amount of inks, their intensity, transitions… and it’s possible too to not shade equally (you can reduce the dark part to a small section and then keep the rest light).

I also tried your last solution (of course :slight_smile: ) but on my mesh it looks a bit strange.


The look is good with some tweaking but the surface reacts to lights very differently in comparison with the other two systems.
So I’m not really sure about wich one is the best :slight_smile:
I have the feeling to not have full control on the look of the object with your last idea… but maybe it’s just a matter of understanding better how to set parameters :slight_smile:

Hope you could find useful my experiments :slight_smile:

Looks great and thanks for sharing! that’s a really interesting approach, I especially like how you implemented the use of a detail normal map and ramp for shading control - I think i’m going to merge some of our ideas and see what the result is!

Can you explain a little more about how your ramp texture works? I’m struggling to understand and recreate your results here.

The idea comes from this great tutorial about a cel shading material (but with a single and explicit light vector to fake a directional light)
https://www.youtube.com/watch?v=9YQEAvvzh0k&t=1153s
Basically he uses a grayscale gradient as UV for the ramp texture, so I did the same thing to round the normal values: each channel value of the normal vector becomes one UV coordinate for the ramp.
I’ll try a guide :
40e410f8f4c648d1198509e6cea4aa795d1f171c.jpeg
Painting different areas in the ramp textures allows you to choose how to round the normals. So, instead of dividing the normal into 2, 3 or more equal steps you can adjust the ratio between dark and light areas and have smoothed transitions.
So a value for the normal of (for example) 0,27 as UV corresponds to, let’s say, a black color from the ramp (or 0)… or it could be 0,25 if you paint more gray bands instead of simply half black and half white.
Hope I was clear… sometimes I find quite hard to explain what I’m doing :slight_smile:

Hello!

I’m also having problems recreating the shader…
Those red nodes (Input Normal Map, Input Normal Map Multiplier and Input Inkband Texture), I couldn’t find it. Could someone tell me what kind of node that is?

Also: Great work in this thread, thank you guys :slight_smile:

Those nodes are “Input nodes” used inside a material function. My graph is actually inside a function because I like to encapsulate useful parts of code/nodes, use it elsewhere in different ways and share it between projects and people of the team :slight_smile:
If you’re doing a simple material, you can use a standard texture sample with a normal texture inside instead of my input Normal Map. The other one (multiplier) is a float1 number while the last is a TextureObject Input: again, if you’re doing a material simply delete it and directly put the ramp texture inside the 3 sampler. :slight_smile:

OK, got it. Thank you very much!

I must be missing something, the steps in the normals just step the normals in each axis, resulting in a WS grid aligned normals.