I love talking about this! And it’s also much more complicated than that!
A bit of background: Unreal Engine’s render pipeline uses deferred translucency. This means that first it renders everything that’s opaque (or masked), then it renders everything translucent, then it combines them together like a post-process effect. What complicates things is that the cel shader post-process layer can happen BETWEEN these steps. By default it does because that looks better (although you can change this and cel-shade translucent materials too.)
When I was developing in XNA way back in 2009 I was also doing deferred translucency and I made use of this issue! When you’re cel shading, water looks better if you cel shade the ground behind it then apply the translucency over top. Then when I remade the shader this year in UE I made the same accidental discovery all over again. Go me.
The end result is this:
The water can be made more or less opaque as far as the cel shading lines are concerned. It can also be made more or less opaque in relation to the colour behind it as a totally independent control. This water shader IS included in the Advanced Cel Shading Pack because it’s dope.
So how does this apply to particles? Well, depending on your particle material you’ll either be using lights, a masked material or a translucent material. Like mentions above, translucent materials aren’t cel-shaded normally! Fortunately most situations in which you’d use a translucent materials, the edge detection wouldn’t pick up the edges and shade them anyway. By changing how you do opacity, you can choose if you want your particle to be heavily cel shaded like a normal object, or lightly shaded behind it like water.
Masked:
Translucent:
So finally, you CAN tell the cel-shader post process material to run AFTER the deferred translucency. I experimented with this a bit in the beginning but haven’t gone back to it as I didn’t want to have to tweak my settings all over again to make it look good. I think the issue was that the lighting is ALSO deferred and it was really messing with my settings at the time. It’s certainly worth another look.
Also worth mentioning: I AM writing a user guide for the cel shader which will have screenshots demonstrating what each control does. I’m currently reducing the number of cel shader controls a bit as some of the options are best left as static values in the blueprint, but Epic say that we can do updates post-release! Hooray!
Edit: the settings are “Before tonemapping”, “After tonemapping” and “Before translucency”. Not hugely clear what each one means in relation to the others. Anyway:
Before tonemapping (normal):
After tonemapping:
That looks evil as hell. I think the colour is coming from the Distant Overbright control. Yup, it’s the overbright. You can do some surprisingly cool stuff with it, e.g. make it bright and make it start way further back:
Our pirate looks particularly badass:
Things can get pretty funky:
So that all looked pretty good. But what about “Before translucency”? Here’s a shot:
This just looks terrible. The water still looks good because there’s a lot that goes into the water shader - an approximation of specularity (UE doesn’t do this for translucent materials), some distortion, some depth-based colour, some depth-based opacity, etc. The smoke material has had no such effort applied to it as it’s just meant to be simple and fast. End result, looks bad.