The normal adustment refers to manually changing the direction of the normals to alter how the lighting hits. That’s done in something like Maya or 3dsmax (or in their case, Softimage). The custom AO is just a mask that’s used to offset the lighting calc at runtime. The formula is pretty simple, take the dot product of the direction of the normal(adjusted in a DCC package) and the direction of the light + the offset mask(value from the custom AO mask).
The outlines are done by taking a copy of the mesh, expanding it a bit and flipping the normals. It’s actually an old skool technique. The downside of this method is that it inceases the amount of tri’s the system has to process. The upside for me is that it allows me to vary the color of the outline as seen in the attached image.
You can see that the outline for the hair is a different colour than that on the body. I haven’t been able to achieve this using the post processing methods. The added cost of having additional geometry to render doesn’t bother me since I’m currently building an anime web series using UE4 and a game later, so I don’t have to worry about performance right now.