I kept seeing all these posts on various forums claiming this can’t be done because Unreal 4 uses deferred rendering, but really you can write whatever shading model you want in an unlit material and light the surface however you want. It’s a shame someone at Epic decided to label it unlit rather than constant. This is entirely doable in the material editor without additional programming and is actually a lot more simple than most shaders. My node graph has more nodes creating parameters and managing the material interface than nodes that generate the look.
How it’s done is by treating the final output like layers of paint on a acetate cel rather than thinking of it as a lit 3D surface. In shading the surface what you are actually trying to do is build up a series of black and white masks that you fill with whatever texture or color you want, and you composite them together.
This is a really great talk from Junya Christopher Motomura about Guilty Gear XRD that is clear and I think very insightful for anyone interested in non-photorealistic rendering. The core point beyond the nuts and bolts of the technical implementation is that when doing something like this you want to control every aspect of it and not leave it to the shader to generate.
For this shader you can start with something like a Phong or Blinn shading model. So get your vertex or pixel normal and a light vector, normalize them and do a dot product. This will give you a 0-1 value. You take that value and based on a threshold parameter that is either a simple scalar or angle if you prefer, fill it with color or a texel representing a lit area or a shaded area. Lots of attempts at this sort of shader use one texture or surface color and let the shader modify it’s HSV for shaded areas which is not going to give the result you want. Very important that the artist choose what colors fill the area explicitly. You can get pretty nutty with your look if you think of it as color compositing a 2D image rather than shading a 3D model. This cel look might be the most simple thing to do with the concept.
The outline is even more simple. It’s an inverted surface that scales the normals based on camera distance.
You can build up from there, but really that’s all there is to the shader itself.
The second part of it is modeling and texturing. The artist has to model for the shader. This means paying attention to silhouettes and probably using a lot of extra geometry to ensure nice smooth surfaces. It also means using less geometry in some areas. The artist should avoid continuous surfaces that go from convex to concave over short periods if the camera is going to be close to the surface. This means for something like a character’s mouth, the artist will model a shallow divot with teeth and a tongue rather than a deep mouth cavity. The artist will also have to manipulate normals by hand. This is covered a bit in the video. You will want to control what areas are always in shadow and probably try to prevent some areas from shadowing. UVs are taken advantage of to create interior ink lines. Also covered in the video. Basically the artist has to be careful about how the model is atlased to determine where the interior lines are and how thick they are. The UV atlases are squared off so the edges always run parallel to texel boundaries in the texture map. This will give you crisp interior lines.
I’ve got a bunch of other non-photorealistic shaders and some post processes I’m working on as well. I was planning to put them on the marketplace with some documentation.