Announcement

Collapse
No announcement yet.

Cell shading without post processing or forward rendering

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

    Cell shading without post processing or forward rendering

    I've been thinking of a way to get cell shading working in unreal without expensive post process effects, code plugins or forward rendering so I can take advantage of unreal's deferred pipeline (multiple lights, etc.)

    After much experimentation I came up with a method that alters the vertex normals of any mesh to achieve a cell shaded effect. It's a bit of a hack and not quite perfect yet but it's a lot cheaper than doing full screen post processing for just a character. Right now there are two different shading styles i'm torn between:

    Without Cell Shader
    Click image for larger version

Name:	2zJhfb7.jpg
Views:	1
Size:	131.7 KB
ID:	1196105

    With Cell Shader

    Style 1:
    Click image for larger version

Name:	tq9XwSf.jpg
Views:	1
Size:	134.7 KB
ID:	1196104

    Style 2:
    Click image for larger version

Name:	nNn7fdL.jpg
Views:	1
Size:	133.7 KB
ID:	1196103


    Multiple Lights
    Click image for larger version

Name:	rgfxFiu.jpg
Views:	1
Size:	168.9 KB
ID:	1196106

    Any feedback would be much appreciated!

    This model is really low poly and messy with pretty bad normals (It was a free download from some obscure site) so I imagine it will look better on a cleaner mesh. I haven't got around to the outline stuff yet but that should be fairly straightforward I think. The model's texture is a very basic flat color map, the cel shading simply comes from the vertex normal tweaks in the shader.

    Will post more info tomorrow.
    Last edited by brisck1; 09-29-2016, 10:07 PM.

    #2
    Please check our Character Cel Shading Pack in the marketplace.

    https://www.unrealengine.com/marketp...l-shading-pack
    https://jiffycrew.com

    Comment


      #3
      Originally posted by Jiffycrew View Post
      Please check our Character Cel Shading Pack in the marketplace.

      https://www.unrealengine.com/marketp...l-shading-pack
      Uh... That's the most shameless plug I've ever seen. The guy asks for feedback and you tell him to buy your pack?

      Comment


        #4
        Originally posted by jonimake View Post
        Uh... That's the most shameless plug I've ever seen. The guy asks for feedback and you tell him to buy your pack?
        Hi Jonimake. Please read our comment again. We didn't tell him to buy our pack. Don't misunderstand our intention. Thanks.
        https://jiffycrew.com

        Comment


          #5
          Well as a developer who is also making a cell shaded game, I think it terrific! Having support for multiple dynamic lights at once is always great
          Check out our game Arcus!

          Comment


            #6
            Originally posted by brisck1 View Post
            After much experimentation I came up with a method that alters the vertex normals of any mesh to achieve a cell shaded effect.
            Hi bud!
            May I ask you to share some hints about the tweaking of the vertex normals of your great cel shader?
            I'm trying every possible way to have cel shading, both postprocess and material... I personally prefer to have full control on my object instead of using a postprocess. I now have a shader that performs the shading but, as you surely imagine, with just one light vector replicating the main directional light of the scene.
            Be able to keep all the ligths would be fantastic and your system sounds an interesting idea
            Thanks in advance for any help or suggestion you would like to share

            Daniele

            Comment


              #7
              Hi guys, sorry I totally forgot to come back and post my findings here. So basically my solution is not ideal visually but comes close to emulating that cell-shaded look without being too involved or restrictive.

              Below is what I do to the vertex normals of my shader to achieve the look:

              Simple way:
              Click image for larger version

Name:	cell_001.JPG
Views:	1
Size:	72.0 KB
ID:	1118937
              This gives you one subdivision of cell shading and is the most basic stylized look

              This graph gives you control over the number of shading layers and also allows you to blend between the smooth normals and stylized ones:
              Click image for larger version

Name:	cell_001.jpg
Views:	1
Size:	488.2 KB
ID:	1118938
              Click image for larger version

Name:	cell_002.jpg
Views:	1
Size:	495.3 KB
ID:	1118939
              Click image for larger version

Name:	cell_003.jpg
Views:	1
Size:	486.6 KB
ID:	1118940
              Click image for larger version

Name:	cell_004.jpg
Views:	1
Size:	493.8 KB
ID:	1118941
              Click image for larger version

Name:	cell_005.jpg
Views:	1
Size:	492.0 KB
ID:	1118942

              so it's not really true cell shading but a close emulation that works with any number of dynamic lights and light propagation volumes. This can also be used on lightmapped objects too but is less noticeable.

              Have Fun!
              Last edited by brisck1; 11-25-2016, 05:45 PM.

              Comment


                #8
                Ok so I decided to come back to this today and do a little more research and see if I could improve things a bit and I think I found a nicer solution:

                Click image for larger version

Name:	cellB-001.jpg
Views:	1
Size:	339.9 KB
ID:	1118949

                Click image for larger version

Name:	cellB-002.jpg
Views:	1
Size:	106.3 KB
ID:	1118950

                Click image for larger version

Name:	cellB-003.jpg
Views:	1
Size:	110.4 KB
ID:	1118951

                Click image for larger version

Name:	cellB-004.jpg
Views:	1
Size:	113.7 KB
ID:	1118952

                Click image for larger version

Name:	cellB-005.jpg
Views:	1
Size:	103.6 KB
ID:	1118953

                Click image for larger version

Name:	cellB-006.jpg
Views:	1
Size:	102.4 KB
ID:	1118954

                Overall I'm pretty pleased with the results - again this works with any lighting, LPV's or static lighting, the only downside to this is that you can't tweak how many cell shade levels you have but I think it's a good alternative for a more typical cell shaded style:

                Click image for larger version

Name:	cellB-007.jpg
Views:	1
Size:	122.1 KB
ID:	1118955

                Let me know what you think or if you have any questions!

                Comment


                  #9
                  This is incredible. I will definitely be looking into using this for my next project, as using custom scripts in the engine (which is what I'm doing now) can make things very complicated.

                  Having a fully working cel shaded system in vanilla unreal would definitely make things much easier.
                  Check out our game Arcus!

                  Comment


                    #10
                    Hi!
                    Your result is really good... I should give it a try
                    Mi solution at the moment was to take the abs values of the VertexNormals and use each channel as UV for a band/ink textures, make a new float3 from the 3 TextureSamplesand then multiply it for (VertexNormal/abs(VertexNormals)) to retain the sign of the components.
                    This simply makes all the VertexNormals as discrete values so the surface can react to the light as if polygons share the same normals... the final look is very similar to a cel shading.
                    Pros: it's very easy to adjust number,thickness and smoothness of the ink-bands... And of course it reacts to all lights in scene.
                    Cons:it gives me strange result somewhere because it tends to generate squared shaded areas.
                    I don't have the engine at home but I'll study your solution and try it on mondey as soon I reach the studio
                    Great job bud!
                    Last edited by berna; 11-26-2016, 10:25 AM.

                    Comment


                      #11
                      Hi!
                      I've tried your solutions (both of them) this morning and my opinion is that your last shader gives you nicer results but very different to the previous in terms of "what polygons" are shaded and how.
                      Your method with the ShadingLevels parameters gives the mesh almost the same look of mine.
                      It's hard to tell wich one is the most correct regarding to the lights in scene...
                      BTW, i'ts a great idea!

                      Comment


                        #12
                        Hi Berna, glad you found it useful. Your technique sounds interesting, care to share your shader graph? I've love to see what results you got with it

                        Comment


                          #13
                          Originally posted by brisck1 View Post
                          Hi Berna, glad you found it useful. Your technique sounds interesting, care to share your shader graph? I've love to see what results you got with it
                          Yep!
                          This is the graph of my function (I love to use material functions and create a library so I can always mix them up and reuse useful shader code)
                          Click image for larger version

Name:	ToonDeferredGraph.jpg
Views:	1
Size:	327.8 KB
ID:	1119092

                          While this is a comparison between my code and yours (please note that now I'm referring to your "second to last" solution): as you can see the result is almost the same. I slightly prefer my approach because a texture for the ink levels allows you to control better the amount of inks, their intensity, transitions... and it's possible too to not shade equally (you can reduce the dark part to a small section and then keep the rest light).
                          Click image for larger version

Name:	ToonDeferredComparison.jpg
Views:	1
Size:	180.9 KB
ID:	1119093

                          I also tried your last solution (of course ) but on my mesh it looks a bit strange.
                          Click image for larger version

Name:	ToonDeferredLightTweak.jpg
Views:	1
Size:	86.4 KB
ID:	1119094
                          The look is good with some tweaking but the surface reacts to lights very differently in comparison with the other two systems.
                          So I'm not really sure about wich one is the best
                          I have the feeling to not have full control on the look of the object with your last idea... but maybe it's just a matter of understanding better how to set parameters

                          Hope you could find useful my experiments

                          Comment


                            #14
                            Looks great and thanks for sharing! that's a really interesting approach, I especially like how you implemented the use of a detail normal map and ramp for shading control - I think i'm going to merge some of our ideas and see what the result is!

                            Comment


                              #15
                              Can you explain a little more about how your ramp texture works? I'm struggling to understand and recreate your results here.

                              Comment

                              Working...
                              X