Care to elaborate? I’m on the art/animation side of things and slowly getting a grasp of the blueprint system. I think with just a hint at the first node I could figure out the rest…
A slight modification to Tomura’s shader.
I’ve removed all of the “above one” lighting path that he created; it turns out that emissives work just fine without it (they don’t bloom, but they do remain perpetually lit even in total blackness) and I don’t feel it added much beyond extra banding that I didn’t want.
What I DID do, however, was figure out how to properly re-add the skybox that went missing after 4.3!
It seems the Skybox no longer had a “base color” value, even if there was one attached to its material. This means that the only way to get the skybox back in is to grab it directly from the Postprocess Input… but this means taking back in all lighting! So what I did was grab the SceneDepth value and multiply it by the lowest allowable amount; this makes everything in the scene more or less completely black, except the skybox (which is at an essentially infinite depth value, evidently?) which shows up bright white. I multiply this by the PostProcess input to get a scene which is all black except the skybox, then add that to the cel shader path’s output.
The end result is we have a cel-shaded scene with a black sky and a normal sky with a black scene, added to produce a cel-shaded scene with a sky! (note that I further multiplied my skybox feed by 0.25, but that’s a personal thing since I like a dull skybox and it’s easier for me that tweaking skybox settings to get the result I’m after).
This is the end result, and many thanks to Tomura for creating a cel-shader setup that was easy enough for someone like me to understand at a conceptual level!
Just throwing this out there, but if somebody were to put a nice, posterized cel shader up on the Marketplace, they’d have a customer…
The same 
Same! I’d love a good Posterized Cel Shader on the Marketplace!
Hi again. It’s been a while. I’ve encountered a few new problems after the 4.4.3 update and was hoping some of you could spare some time to help me.
After the 4.4.3 update broke my old shader, I started experimenting in order to fix it. What I came up with fixed the skybox problem and black screens, but now I’m faced with a new problem. The shadows of the cel shade seems to be bound to the brightness of the diffuse texture for some reason. I’ve been trying for days to find a way to get around this with no luck finding a solution. I was hoping some more experienced shader programmers could shed some light on this and help find a way to fix it. In the images bellow I’ve compared different diffuse textures to each other in order to demonstrate my problem.
As you can see, at a RGB of 0.5 it looks perfect.
On this image you can see that the hair has no shadow and is completely black.
Here’s the same image without the shader.
I put 5 models side by side, each with a different diffuse color in order to see the changes in the shader as I experimented
Here’s how my current shader is built:
And here’s the cel texture I’m using:
If someone could help me fix this, it would be much appreciated! Thanks for your time.
-Headstub
Hey Headstub!
I really like the way you put the CelShader together, but, as you mentioned, the borders of the brightness scale made some difficulties.
But I might have figure something out that could help you. It definitely isn’t the best solution, but it works, at least for me.
As you can see you got the CelShading even at black and white, although the outer Stones aren’t completely the colour they are supposed to be.
You can improve this a little bit if you like to, by changing your cel texture to a light grey and a complete white.
But I guess you want to see how I solve your real Problem, so let’s start:
By adding this Max node, you solve your problem with the black colour. The reason for that is that no matter whether the CelShader decided that the colour is going to be the right or the left one of your texture, it always will be multiplied by 0. Therefore the result is also always 0. So by adding a border, so that any value beyond 0.03 becomes 0.03, you do not multiply by 0.
You can also change the value to something smaller, so the black gets darker, but in opinion you shouldn’t get beyond 0.02, because then the same Problem will appear again or you just at least can’t keep the dark colours apart
Solving the problem for the white colour took me much longer. I tried several different ways, but none of them seems to work.
But somehow I could solve it anyway by happenstance. I noticed that the colours were kind of noisy, as if the antialiasing was broken. Therefore I changed the Post Process Material to ‘Before Tonemapping’, because that is what you do in such case. Yeah and somehow it also fix the problem with the bright colours.
Yeah and that is all. Nothing more behind it
I hope I could help you, because I read this thread for a while now, but I could never add something that advances this.
- Merlin
PS: Just so you can see that I actually used your CelShader with my tweaks
Alright cool, but it doesn’t seem to work for me:/ The black textures still have close to no shadows on them. Same with the white. I can’t notice any difference with it on or off:/
For the record, the current shader I’m using was mostly made by DevilBoxGames. I’ve just been playing around with it trying to come up with a fix:P
But to add my own touch to it I made a set of controls to blend in soft shadows with the cel shade and adjusting the cel shade brightness:
It gives more control if you want completely solid shading or a mix between the two.
But back to the problem at hand. If you look at the screenshot I posted a while back, you can see that the shadow on the black hair is completely normal:
As you can see by the model on the right, the black texture on the hair doesn’t interfere with the shadows at all.
I appreciate your help Marki and hope we can figure out a good solution for this. It’s cool that this shader has become a group effort by different people in this thread:)
-Headstub
Hi again guys. I managed to get the shader to work perfectly today. I was on the fence about sharing the fix since I wanted people to figure it out themselves, but I ended up just thinking “Eh, **** it. I’d like to see more games using a proper cel shader”. My fix was to remove all unnecessary information from the postprocessinput0 that would mess with the shadows of the cel by subtracting it from the main input.
I’ve tried documenting as much as I can about what each section does. I’m not sure if everything I said is correct, so please correct me if I’ve misunderstood something.
RGB 1.0
RGB 0.75
RGB 0.5
RGB 0.25
RGB 0.0 (Right)
RGB 0.01 (Left)
So don’t go completely black
Problems can occur with the shader if the material is reflective, so make sure to set roughness to 1.
Roughness 1
Roughness 0
One last thing for people who are new to UE4(like me). Remember to check “Render Custom Depth” in the Rendering tab of the static mesh you want the shader to apply to.
That is what this whole module in the shader does:
If you want the shader to apply to everything in the scene you just remove that module. The problem is that the sky will be completely black if you do that because those channels will be masked out by the cel banding. People could probably find a workaround for that in the future. Right now I enjoy having cel shading on characters and leaving the environment realistic, which is the case in most anime if you look at the realistic background paintings and the simple shaded characters.
Thanks to everyone who helped put this shader together and I hope this will be of use to more people who come by here in the future.
-Headstub
I’m going to try implementing some of the ideas from this thread; in case anyone in this thread is interested, I posted an article in Rendering on a post-process outline (vary thickness with distance) and a hack to make facial shadows reddish instead of gray, here:
Note that my shader is acting on everything that does NOT use custom depth, same as Epic’s Stylized demo, instead of everything WITH custom depth, which seems to be the rule here. So invert it.
OK, here’s an example of no postprocessing, just outline+red facial shadows, just flat color shading, and both.
Some small adjustments of lighting and red facial shadows were done for different shaders.
Unfortunately, a few outline pixels sometime strobe. That seems to be due to the original algorithm, and I think it might be due to feedback from the outline from one frame to the next. If you slow the framerate way down when it happens you can see pixels in the outline changing through the same 4 states or so.
Here’s no post-processing:
Here’s just outline + red facial shadows:
Here’s just flat color shading:
And here’s both:
I’ve figured out the cause of the strobing pixels; it’s feedback from the combination of using pixel near-neighbor algorithms in PP and TemporalAA, where the algorithm looks at various pixels, some of which have & some have not gone through the algorithm yet. For example, the outline algorithm that strobes through an 8-frame pattern looks at all 4 pixel neighbors, but using the x+1,y and x,y+1 pixels can create the feedback. When I change the algorithm to only use x-1,y and x,y-1, the feedback is gone.
I’ll post a general warning post about this later, as this can affect all pixel near-neighbor algorithms in PP with TemporalAA.
cool we are going toon/cel later in our game, and it is good to read some of the things people have done. We will see where ours land. Maybe just simple color, three shades and an outline.
Can you guys pleeeeaaase update your materials uasset files and save me from going through all those pages ? I will appreciate it very much !
I’m working on a version that uses params, so it will be easier to customize. I also changed the “edge detection by normals”, which was pretty bogus. I’ll post the file once I have it all hashed out. The pixel strobing might have been due to differences in the length of the pipeline for x+1,y x,y+1 vs. x-1,y x,y-1 pixel calculations, and 4.5.1 seems to have also improved the situation, so I’m no longer seeing parts of the outline flicker in extreme cases.
Also, one thing I need to mention, for the post-process, you need to reduce the ambient lighting (default is 1, should be 0.1 - 0.3) or the red facial shadows wash out and don’t really show up.
Attached here is an improved version of my shader combined with the latest from Headstub.
I added a few things:
-
the outline by distance can make flat walls turn black when seen at an extreme angle, because the distance between each pixel can be quite large and fool the algorithm into thinking it should create an outline pixel. I fixed this by not drawing a pixel if all the neighboring pixels were pointing in the same direction (by checking normals).
-
Headstub’s flat shader darkens the skybox, so I added a large distance check to skip it.
-
It’s set up to only apply the anime shader to meshes using CustomDepth. Change the connection to skip it and apply it to everything.
-
Added more labels for tweaking (how to e.g. only see outline by normals, etc)
-
Headstub’s shader doesn’t need the soft shadow blend, since that’s essentially the same as blending with red facial shader, which is needed anyway.
-
Added separate outline fade with distance.
-
As above, reduce the ambient lighting in PostProcessing from the default of 1 to 0.1-0.3, otherwise the red facial shadows will wash out.
-
Decided against using params for now.
EDIT: use this link for the latest version:
AnimeShaderFiles.zip (89 KB)&d=1445222643
Brian Westley Awesome thank you very much !
A couple of things I still need to check:
The fix in (1) includes the alpha in the normals, I’m not sure if it should. Likewise, right now it uses a rather narrow comparison (0.99) to determine if all the normals are pointing the same way, but textures with bumpmaps might justify reducing the 0.99 value to ignore surfaces that are mostly flat. I’ll also put a note in the shader itself about how high ambient lighting washes out the red facial shadows.
Also, if you change the line color from a constant black color to the output of the add that goes into Lerp A multiplied by 0.4 or so, the outlines will be darker versions of the color of the material instead of black.
The shader doesn’t work for me, the scene looks exactly the same with or without it.
This is the mesh material :
This is the scene :
What am i missing ?
The default configuration is to only affect meshes that have Custom Depth turned on. Does your brick thing have “Render Custom Depth” checked in the extras section of its Rendering?
Alternatively, you can connect the rightmost Lerp directly into Emissive Color in the shader, to bypass the part that only applies it to meshes with CustomDepth turned on. Then it will be used on everything.