Announcement

Collapse
No announcement yet.

trying to make Toon Shader Material on UE4

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

    Hi again. It's been a while. I've encountered a few new problems after the 4.4.3 update and was hoping some of you could spare some time to help me.

    After the 4.4.3 update broke my old shader, I started experimenting in order to fix it. What I came up with fixed the skybox problem and black screens, but now I'm faced with a new problem. The shadows of the cel shade seems to be bound to the brightness of the diffuse texture for some reason. I've been trying for days to find a way to get around this with no luck finding a solution. I was hoping some more experienced shader programmers could shed some light on this and help find a way to fix it. In the images bellow I've compared different diffuse textures to each other in order to demonstrate my problem.





    As you can see, at a RGB of 0.5 it looks perfect.



    On this image you can see that the hair has no shadow and is completely black.



    Here's the same image without the shader.



    I put 5 models side by side, each with a different diffuse color in order to see the changes in the shader as I experimented


    Here's how my current shader is built:











    And here's the cel texture I'm using:




    If someone could help me fix this, it would be much appreciated! Thanks for your time.

    -Headstub

    Comment


      Originally posted by Headstub View Post
      Hi again. It's been a while. I've encountered a few new problems after the 4.4.3 update and was hoping some of you could spare some time to help me.

      After the 4.4.3 update broke my old shader, I started experimenting in order to fix it. What I came up with fixed the skybox problem and black screens, but now I'm faced with a new problem. The shadows of the cel shade seems to be bound to the brightness of the diffuse texture for some reason. I've been trying for days to find a way to get around this with no luck finding a solution. I was hoping some more experienced shader programmers could shed some light on this and help find a way to fix it. In the images bellow I've compared different diffuse textures to each other in order to demonstrate my problem.

      -Headstub
      Hey Headstub!

      I really like the way you put the CelShader together, but, as you mentioned, the borders of the brightness scale made some difficulties.
      But I might have figure something out that could help you. It definitely isn't the best solution, but it works, at least for me.

      Click image for larger version

Name:	83f97cb39d.png
Views:	1
Size:	339.5 KB
ID:	1056950
      As you can see you got the CelShading even at black and white, although the outer Stones aren't completely the colour they are supposed to be.
      You can improve this a little bit if you like to, by changing your cel texture to a light grey and a complete white.

      Click image for larger version

Name:	b41c893d6f.png
Views:	1
Size:	313.0 KB
ID:	1056951
      But I guess you want to see how I solve your real Problem, so let's start:

      Click image for larger version

Name:	5aa11e862e.png
Views:	1
Size:	118.3 KB
ID:	1056952
      By adding this Max node, you solve your problem with the black colour. The reason for that is that no matter whether the CelShader decided that the colour is going to be the right or the left one of your texture, it always will be multiplied by 0. Therefore the result is also always 0. So by adding a border, so that any value beyond 0.03 becomes 0.03, you do not multiply by 0.
      You can also change the value to something smaller, so the black gets darker, but in opinion you shouldn't get beyond 0.02, because then the same Problem will appear again or you just at least can't keep the dark colours apart

      Click image for larger version

Name:	a087540fee.png
Views:	1
Size:	5.9 KB
ID:	1056953
      Solving the problem for the white colour took me much longer. I tried several different ways, but none of them seems to work.
      But somehow I could solve it anyway by happenstance. I noticed that the colours were kind of noisy, as if the antialiasing was broken. Therefore I changed the Post Process Material to 'Before Tonemapping', because that is what you do in such case. Yeah and somehow it also fix the problem with the bright colours.

      Yeah and that is all. Nothing more behind it
      I hope I could help you, because I read this thread for a while now, but I could never add something that advances this.

      - Merlin

      PS: Just so you can see that I actually used your CelShader with my tweaks
      Click image for larger version

Name:	ce4567110f.png
Views:	1
Size:	155.4 KB
ID:	1056954

      Comment


        Originally posted by Marki217q View Post
        Hey Headstub!

        I really like the way you put the CelShader together, but, as you mentioned, the borders of the brightness scale made some difficulties.
        But I might have figure something out that could help you. It definitely isn't the best solution, but it works, at least for me.

        As you can see you got the CelShading even at black and white, although the outer Stones aren't completely the colour they are supposed to be.
        You can improve this a little bit if you like to, by changing your cel texture to a light grey and a complete white.

        But I guess you want to see how I solve your real Problem, so let's start:

        By adding this Max node, you solve your problem with the black colour. The reason for that is that no matter whether the CelShader decided that the colour is going to be the right or the left one of your texture, it always will be multiplied by 0. Therefore the result is also always 0. So by adding a border, so that any value beyond 0.03 becomes 0.03, you do not multiply by 0.
        You can also change the value to something smaller, so the black gets darker, but in opinion you shouldn't get beyond 0.02, because then the same Problem will appear again or you just at least can't keep the dark colours apart

        Solving the problem for the white colour took me much longer. I tried several different ways, but none of them seems to work.
        But somehow I could solve it anyway by happenstance. I noticed that the colours were kind of noisy, as if the antialiasing was broken. Therefore I changed the Post Process Material to 'Before Tonemapping', because that is what you do in such case. Yeah and somehow it also fix the problem with the bright colours.

        Yeah and that is all. Nothing more behind it
        I hope I could help you, because I read this thread for a while now, but I could never add something that advances this.

        - Merlin

        PS: Just so you can see that I actually used your CelShader with my tweaks
        Alright cool, but it doesn't seem to work for me:/ The black textures still have close to no shadows on them. Same with the white. I can't notice any difference with it on or off:/

        For the record, the current shader I'm using was mostly made by DevilBoxGames. I've just been playing around with it trying to come up with a fix:P

        But to add my own touch to it I made a set of controls to blend in soft shadows with the cel shade and adjusting the cel shade brightness:



        It gives more control if you want completely solid shading or a mix between the two.







        But back to the problem at hand. If you look at the screenshot I posted a while back, you can see that the shadow on the black hair is completely normal:



        As you can see by the model on the right, the black texture on the hair doesn't interfere with the shadows at all.

        I appreciate your help Marki and hope we can figure out a good solution for this. It's cool that this shader has become a group effort by different people in this thread

        -Headstub
        Last edited by Headstub; 09-22-2014, 06:15 PM.

        Comment


          Hi again guys. I managed to get the shader to work perfectly today. I was on the fence about sharing the fix since I wanted people to figure it out themselves, but I ended up just thinking "Eh, **** it. I'd like to see more games using a proper cel shader". My fix was to remove all unnecessary information from the postprocessinput0 that would mess with the shadows of the cel by subtracting it from the main input.

          I've tried documenting as much as I can about what each section does. I'm not sure if everything I said is correct, so please correct me if I've misunderstood something.














          RGB 1.0



          RGB 0.75



          RGB 0.5



          RGB 0.25



          RGB 0.0 (Right)

          RGB 0.01 (Left)

          So don't go completely black


          Problems can occur with the shader if the material is reflective, so make sure to set roughness to 1.



          Roughness 1



          Roughness 0


          One last thing for people who are new to UE4(like me). Remember to check "Render Custom Depth" in the Rendering tab of the static mesh you want the shader to apply to.



          That is what this whole module in the shader does:



          If you want the shader to apply to everything in the scene you just remove that module. The problem is that the sky will be completely black if you do that because those channels will be masked out by the cel banding. People could probably find a workaround for that in the future. Right now I enjoy having cel shading on characters and leaving the environment realistic, which is the case in most anime if you look at the realistic background paintings and the simple shaded characters.

          Thanks to everyone who helped put this shader together and I hope this will be of use to more people who come by here in the future.

          -Headstub

          Comment


            I'm going to try implementing some of the ideas from this thread; in case anyone in this thread is interested, I posted an article in Rendering on a post-process outline (vary thickness with distance) and a hack to make facial shadows reddish instead of gray, here:

            https://forums.unrealengine.com/show...d-demo-project

            Note that my shader is acting on everything that does NOT use custom depth, same as Epic's Stylized demo, instead of everything WITH custom depth, which seems to be the rule here. So invert it.

            Comment


              OK, here's an example of no postprocessing, just outline+red facial shadows, just flat color shading, and both.
              Some small adjustments of lighting and red facial shadows were done for different shaders.

              Unfortunately, a few outline pixels sometime strobe. That seems to be due to the original algorithm, and I think it might be due to feedback from the outline from one frame to the next. If you slow the framerate way down when it happens you can see pixels in the outline changing through the same 4 states or so.

              Here's no post-processing:
              Click image for larger version

Name:	IXS7G2N.jpg
Views:	1
Size:	60.6 KB
ID:	1060212

              Here's just outline + red facial shadows:
              Click image for larger version

Name:	Uwe7QXi.jpg
Views:	1
Size:	67.7 KB
ID:	1060213

              Here's just flat color shading:
              Click image for larger version

Name:	YNz4wyW.jpg
Views:	1
Size:	49.7 KB
ID:	1060214

              And here's both:
              Click image for larger version

Name:	w8cCNwo.jpg
Views:	1
Size:	61.0 KB
ID:	1060215
              Attached Files
              Last edited by Brian Westley; 10-26-2014, 10:06 PM.

              Comment


                I've figured out the cause of the strobing pixels; it's feedback from the combination of using pixel near-neighbor algorithms in PP and TemporalAA, where the algorithm looks at various pixels, some of which have & some have not gone through the algorithm yet. For example, the outline algorithm that strobes through an 8-frame pattern looks at all 4 pixel neighbors, but using the x+1,y and x,y+1 pixels can create the feedback. When I change the algorithm to only use x-1,y and x,y-1, the feedback is gone.

                I'll post a general warning post about this later, as this can affect all pixel near-neighbor algorithms in PP with TemporalAA.

                Comment


                  cool we are going toon/cel later in our game, and it is good to read some of the things people have done. We will see where ours land. Maybe just simple color, three shades and an outline.

                  Comment


                    Can you guys pleeeeaaase update your materials uasset files and save me from going through all those pages ? I will appreciate it very much !
                    Check out my game OldSchool Nightmare : http://www.indiedb.com/games/oldschool-nightmare

                    Comment


                      I'm working on a version that uses params, so it will be easier to customize. I also changed the "edge detection by normals", which was pretty bogus. I'll post the file once I have it all hashed out. The pixel strobing might have been due to differences in the length of the pipeline for x+1,y x,y+1 vs. x-1,y x,y-1 pixel calculations, and 4.5.1 seems to have also improved the situation, so I'm no longer seeing parts of the outline flicker in extreme cases.

                      Also, one thing I need to mention, for the post-process, you need to reduce the ambient lighting (default is 1, should be 0.1 - 0.3) or the red facial shadows wash out and don't really show up.
                      Last edited by Brian Westley; 10-29-2014, 09:53 AM.

                      Comment


                        Attached here is an improved version of my shader combined with the latest from Headstub.

                        I added a few things:

                        1) the outline by distance can make flat walls turn black when seen at an extreme angle, because the distance between each pixel can be quite large and fool the algorithm into thinking it should create an outline pixel. I fixed this by not drawing a pixel if all the neighboring pixels were pointing in the same direction (by checking normals).

                        2) Headstub's flat shader darkens the skybox, so I added a large distance check to skip it.

                        3) It's set up to only apply the anime shader to meshes using CustomDepth. Change the connection to skip it and apply it to everything.

                        4) Added more labels for tweaking (how to e.g. only see outline by normals, etc)

                        5) Headstub's shader doesn't need the soft shadow blend, since that's essentially the same as blending with red facial shader, which is needed anyway.

                        6) Added separate outline fade with distance.

                        7) As above, reduce the ambient lighting in PostProcessing from the default of 1 to 0.1-0.3, otherwise the red facial shadows will wash out.

                        8) Decided against using params for now.

                        EDIT: use this link for the latest version:
                        https://forums.unrealengine.com/atta...8&d=1445222643
                        Last edited by Brian Westley; 03-11-2017, 02:40 PM.

                        Comment


                          Brian Westley Awesome thank you very much !
                          Check out my game OldSchool Nightmare : http://www.indiedb.com/games/oldschool-nightmare

                          Comment


                            A couple of things I still need to check:

                            The fix in (1) includes the alpha in the normals, I'm not sure if it should. Likewise, right now it uses a rather narrow comparison (0.99) to determine if all the normals are pointing the same way, but textures with bumpmaps might justify reducing the 0.99 value to ignore surfaces that are mostly flat. I'll also put a note in the shader itself about how high ambient lighting washes out the red facial shadows.

                            Also, if you change the line color from a constant black color to the output of the add that goes into Lerp A multiplied by 0.4 or so, the outlines will be darker versions of the color of the material instead of black.
                            Last edited by Brian Westley; 10-31-2014, 06:32 PM.

                            Comment


                              The shader doesn't work for me, the scene looks exactly the same with or without it.

                              This is the mesh material :
                              Click image for larger version

Name:	MaterialGrapth.PNG
Views:	1
Size:	88.6 KB
ID:	1060479
                              This is the scene :
                              Click image for larger version

Name:	Scene.PNG
Views:	1
Size:	724.9 KB
ID:	1060480

                              What am i missing ?
                              Check out my game OldSchool Nightmare : http://www.indiedb.com/games/oldschool-nightmare

                              Comment


                                The default configuration is to only affect meshes that have Custom Depth turned on. Does your brick thing have "Render Custom Depth" checked in the extras section of its Rendering?

                                Alternatively, you can connect the rightmost Lerp directly into Emissive Color in the shader, to bypass the part that only applies it to meshes with CustomDepth turned on. Then it will be used on everything.

                                Comment

                                Working...
                                X