[UE5] Anime/Toon/Cel Shading Model (WORKS WITH LAUNCHER ENGINE VERSIONS)

[Envieous]

This is a UE5 version of my shading model for UE4 that I built for my military academy dating simulator/military warfare tactical rpg/tank washing simulator.

A fair amount of stuff has changed since the previous thread, including the first number in the engine version it’s for so may as well make a new thread to contain that information.

In summary, this shading model allows you to specify exactly what unreal displays in shadow and lit areas. This doesnt have to follow any sort of realistic lighting logic, you can create materials that are brighter in shadow than where they are actually lit. Whatever you put in the shadow colour pin is what is displayed in shadow.

Here is Nicholas Cage to demonstrate.

and some basic spheres

Here’s how it works.

Material and Lighting Setup

This Shading Model hijacks the Clear Coat (default lit in source versions) Shading Model.

There are two options that this is geared towards.

#1 Standard material workflow

Most of this works the same as stock Unreal.

Base Colour works the same as stock Unreal, however I do not use the colour of light actors on the shading side. So if you have a surface and you want to apply a light colour to it you need to multiply that onto the base colour on the material side. I might change this back later but I’m working on some effect ideas that require this kind of setup. It’s also a nonissue unless you want to use more than just a single directional light which isn’t a purpose this shading model is really geared towards.

Metallic is shadow range, this defines how hard your shadow is. 1 is soft, 0 is hard.

Custom Data 0 is Shadow Colour. This is what renders when a surface is in shadow.

Custom Data 1 isn’t used for anything but it is exposed and will write a single channel to the gbuffer so it can be used to provide information to a post process material.

All that is needed on the lighting front is a single white moveable directional light with an intensity of 1. You can affect the size of the shadow by raising/lowering this, but this value is what I use.

Multiple lights is SORTA supported but they are tricky, their influence must not overlap or else you will get incorrect results. HOWEVER, if you enable the command r.UseClusteredDeferredShading you can stack point and spot lights to your hearts content. This isn’t a magic bullet right now unfortunately, they will still misbehave in certain circumstances, and if they touch a directional light/rect light and you should use lighting channels to prevent this from happening, but this opens up many lighting possibilities that were not possible to do correctly in previous versions.

Left is without clustered shading enabled, right is with.

This requires a post process to correct unlit pixels, it is included in the example project as PP_PointLightCorrection.

#2 Replace final output with a true lighting buffer for more advanced post process shaders.

This approach works by turning the final output Unreal creates into just the lighting information, and as such should be compatible with all lighting features, Lumen, Ray Tracing, etc. The downside, is that you’re still stuck with the limitations of working in the post process space, but if the first approach’s ability to control lighting is too limited for you, this should cover your needs.

In DeferredLightPixelShader.usf uncomment this line:

OutColor.rgb = Radiance.rgb;

and comment out

ShadowMask = smoothstep(0.5 - ShadowRange, 0.5 + ShadowRange, Luminance(Radiance));
OutColor.rgb = lerp(ShadowColour, PixelColour, ShadowMask);

I’m not using this approach, but people interested in creating their own custom shading through post processes will find this interesting given the significant drawbacks of the traditional Scenecolour divided by diffuse colour to get a crude approximation of the lighting information approach. This is just outright better in every single way than trying to do this with stock Unreal.

Installation

You can find my repository here: https://github.com/envieous/UnrealEngine-SelShader

Before you ask why you got a 404 after clicking that link, read this: Unreal Engine 4 on GitHub - Unreal Engine

For 5.3(Launcher branch) and onwards

Download my repository, and overwrite Engine/Shaders/Private in your LAUNCHER engine installation folder with the Engine/Shaders/Private folder from MY REPOSITORY.

For 5.3 (Source branch) and previous

Build engine from source.

Setup

These changes aren’t explicitly required, but if you’re trying to achieve a look similar to mine this is how I’m doing it.

Disable Static Lighting

Disable exposure adjustment here.

You’ll also likely want to look at your other post process settings, bloom and vignette in particular and pick values you like.

The tonemapper in Unreal is notorious for washing out/desaturating/and just generally ruining your colours in the name of physical accuracy. I have disabled the tonemapper and Unreal now outputs colours very close to what is inputted.

If for some reason you actually want Unreal to tOnEmAp your colours you can undo this change by changing this line in PostProcessTonemap.usf

half3 OutDeviceColor = LinearColor.rgb;

back to

half3 OutDeviceColor = ColorLookupTable(LinearColor);

Unreal also tonemaps UMG elements. I have no idea why ANYONE would desire this behavior, and unfortunately I can not remove this functionality without severely compromising the usability of the editor as it affects the ENTIRE editor UI as well.

For now the best way I’ve found to correct this is to create this material function to correct UMG colours.

This isn’t an ideal way of doing this and will probably cause problems, but it’s good enough for development, and can be removed at the end while also removing UMG tonemaping for the final build to achieve correct results.

To remove UMG tonemapping open SlateElementPixelShader.usf and comment out the lines:

OutColor = ApplyContrast(OutColor);
OutColor.rgb = GammaCorrect(OutColor.rgb);

Misc

I should probably mention Lumen. I’ve played with it a bit but it doesn’t seem feasible from a both a performance and art standpoint to utilize it in my project so if you want to use it I can’t really offer any suggestions or help, but if you make something cool with it in this shading model I’m very interested to see what people come up with.

Example Project(s)

This project has all the settings setup and contains further example content. There’s two folders in the archive, use launcher example for launcher version, source for source.

Download here.

Acknowledgements

Tom Looman for his Dr Facilier shadow demonstration that I used initially as the basis for my shading system.
Doomfest and Arnage , for their custom shading models. I studied a lot of their changes to figure out what was going on in Unreal’s USF files.
FelixK and Matt Hoffman for their articles on creating a custom shading model in Unreal. These articles despite being dated were still incredibly helpful for figuring all this out.

---

I’m pretty sure that covers everything. Enjoy.

You can find my Twitter here, and my Discord here.

Also if you’re interested, you can wishlist my game on Steam here, each wishlist helps <3

Also consider leaving a post and vote here in the off chance of getting Epic to make custom shading models not require wasting hours of everyone’s time building the engine from source just to enable a couple material pins.

Finally ,if you want to support further development of this shading model, you can do so on Patreon here.

[Envieous]

Small update: I’ve disabled the tonemapper in the shader files and replaced the section in the OP about converting colours within the material since this should be a more direct and better way of getting correct colours.

[kriNon]

@Envieous

DeferredLightPixelShader.usf.

Out of curiosity, have you looked in strata materials to see if there’s a way of adding a custom shading model there? It might be extendable without needing to edit the engine source.

[Envieous]

This is the first I’ve heard of it.

My city recently got pummeled by a storm and I’ve been out of power for the last 5 days, so I watched a video about it and it doesn’t really look applicable, but I’ll give it a proper look when the power comes back on.

[kriNon]

I hope your power situation gets fixed soon! Power outages really suck.

Yeah, so the intention of strata materials is for materials that can have aspects of multiple shading models.

I don’t have a full understanding of how it works, I’ve been trying to understand the source but it’s taking me some time.

I believe that strata materials internally replicate the shader models (I think they get compiled in to some sort of strata model). Either it might be possible to add a custom material shader to expand upon the strata system, or it might be possible to create a custom strata model and run it directly???

I’m really not confident, but i think there is a slight chance that it would be possible.

[Envieous]

I gave it a look. I can’t really make heads or tails of what the code it’s got is supposed to be doing, but maybe it’ll have proper documentation that’ll help us all make more sense of it when it’s no longer an experimental feature.

For now I guess just bump/vote this thread in the hopes that Epic gives us some kind of proper response about making custom shading models viable without wasting everyone’s time with recompiling the engine.

[Envieous]

I added an example project to the OP with all the settings set up as I use them, and some example content.

[Frenetic]

That’s not Nic Cage, that’s John Travolta.

[Envieous]

Added 5.1 version.

[LoafingShiba]

Hi @Envieous, thank you for sharing the shader. I saw that you had simulated subsurface scattering in the older versions of the shader from [WIP] Anime/Toon Stylized Shading Model, specifically up to UE 4.26.

Out of curiosity, was there a reason you stopped supporting subsurface scattering in your shader?

[Envieous]

I moved away from forward rendering after 4.26, and this meant I needed a Gbuffer to hold the shadow colour data, and since I wasn’t really making any use of subsurface scattering for my game it was first up on the chopping block to make space.

I also saw some documentation (that I don’t have on hand, but I’m pretty sure it was in softmode’s shading model thread) on Honkai Impact’s rendering where they had some method of faking subsurface lighting that just looked a lot better and could be replicated easily within the material graph. The only real advantage of doing it through the shading model was that dynamic shadows would effect it.

[LoafingShiba]

Changing engine code can be pretty nasty each time a new UE version releases. There’s also the potential of needing to re-implement your shading if the shading code changes too much.

Have you considered using a post process cel shaded material in combination with some sort of object masking, like a Custom Stencil Buffer? This cel shader from the Unreal Marketplace pretty much combines all of those concepts.

I haven’t investigated the performance of using a masked post process shader vs. modifying the engine code, but if you’re using deferred shading…that’s pretty much just adding one extra layer to the whole stack of G-buffer layers anyway? So it wouldn’t be too much of a performance hit? The gains in maintainability would be tremendous if the masked post process shader works out.

@Envieous

…Honkai Impact’s rendering where they had some method of faking subsurface lighting that just looked a lot better and could be replicated easily within the material graph. The only real advantage of doing it through the shading model was that dynamic shadows would effect it.

I thought Genshin/Honkai Impact’s subsurface scattering is just color/shadow banding? Pretty bleh. Your dynamic subsurface scattering looks substantially better , where it “turns on” when the light vector points more towards the camera.

[Envieous]

@LoafingShiba

Changing engine code can be pretty nasty each time a new UE version releases. There’s also the potential of needing to re-implement your shading if the shading code changes too much.

This is why I keep the changes as simple as possible. I change literally 5 lines of C++ code, and just hijack the existing shading model. It’s trivially easy to just copy paste them over between engine versions as long as Epic doesn’t throw in any curveballs, and the only one that was troublesome to figure out was them changing where the CustomData buffer is enabled between 4.27 and 5, otherwise they just seem to pointlessly shuffle shader code around with each engine version without making any actual changes which is usually not a big deal to figure out since the functionality didn’t actually change.

At this point the most obnoxious thing about this is that it requires wasting everyone’s time recompiling the engine for HOURS just to make basic C++ changes to open a material pin, and as of 5, enable the CustomData buffer, which you could do before in the USF/USH files.

The worst part is that a little before 5 came out I had made a version that worked without touching the engine source and worked on launcher builds (by using the compression method I use in the metallic pin), and then Epic moved the ability to enable the CustomData gbuffer into the C++ code.

@LoafingShiba

Have you considered using a post process cel shaded material in combination with some sort of object masking, like a Custom Stencil Buffer? This cel shader from the Unreal Marketplace pretty much combines all of those concepts.

I have, and they’re so limited that they’re borderline useless. Anyone selling them in the marketplace pushing them as a good solution is, at best just marginally less clueless than the people buying them or at worst taking advantage of the ignorance of newbie developers to make a quick buck. Their sole benefit is that people who don’t know what they’re doing can implement them in their projects easily.

That said I get why they’re so prevalent, there’s a big leap in knowledge required between throwing a post process into unreal and modifying unreal’s shader code with the only middleground inbetween being like 2 people’s shading models that may or may not fit your needs, and on top of that the only information on how to tackle this stuff is a few extremely out of date community tutorials, and studying the changes existing custom shading models have made.

@LoafingShiba

I haven’t investigated the performance of using a masked post process shader vs. modifying the engine code, but if you’re using deferred shading…that’s pretty much just adding one extra layer to the whole stack of G-buffer layers anyway? So it wouldn’t be too much of a performance hit? The gains in maintainability would be tremendous if the masked post process shader works out.

Up until recently I was using an RX580 which was basically a mid end card at best when it came out in 2017, and I’ve used both approaches without every running into any noticeable difference in performance. As I understand it using a post process does come with an added cost, but I’ve never found it to be significant.

@LoafingShiba

I thought Genshin/Honkai Impact’s subsurface scattering is just color/shadow banding? Pretty bleh. Your dynamic subsurface scattering looks substantially better , where it “turns on” when the light vector points more towards the camera.

I took a quick glance and couldn’t find the article again and I’m not really sure how genshin does it, but if all you want is for it to fade based on whether the camera is looking at the light source that’s very easy to accomplish in the material editor.

This is the main reason I made shadow colour a literal value instead of just multiplying it in the shader code. A lot of the shader code is is just the exact same logic you would make in the material graph, it just has access to additional information that isn’t available in the material graph, most importantly for our needs, the true lighting data (light colour, dynamic shadows, cubemaps, etc). For effects that don’t need this information (though with my shading model you do get to indirectly define behavior in shadow/lit) you can make your own on the material level and not be reliant on whatever hardcoded formulas are in the shadercode.

[Envieous]

Fixed decals disappearing when in shadow.

Set up decal shadow colour inputs like so to control their colour in shadow.

[zebrrush]

Do you prefer mesh outlines instead of post-process? Why?

[Arkiras]

Not the OP but the main benefit is that you get variable line weight. Using a post process gets extremely expensive when you want thick lines and if you want the line weight to get thinner based on distance you pretty much get that for free with mesh outlines.

[Envieous]

I use both. I’m running a post process that uses a sobel filter on normals, depth, and diffuse with sensitivity controls in the material. This works great for outlining surface details but has trouble with certain areas like fingers, or pleated skirts. Mesh outlines on the other hand capture these areas just fine.

[zebrrush]

I’m kinda sad after moving from unity (where I’ve done a shitload of custom lighting).
Who knew that unreal did not give us ability to customize lighting response in the editor (even if it would be HLSL/GLSL/whateverSL)

What I did here is adjust world space normals to get more or less a cell-shaded look. It kinda messes up with normal edge detection and is not really accurate… Final color / baseColor doesn’t work properly also. It looks like there is no other way than to adjust the engine code

This is the closest I’ve got to my style:

This is what I have in Unity, which I’m trying to repeat here:

[Envieous]

@zebrrush

I’m kinda sad after moving from unity (where I’ve done a shitload of custom lighting).
Who knew that unreal did not give us ability to customize lighting response in the editor (even if it would be HLSL/GLSL/whateverSL)

Yeah this has been a long running complaint since ue4 came out, maybe before that too, but I was busy doing photorealism on cryengine at the time so I wouldn’t know lol.

Hopefully Epic builds up the will to do something about it eventually vOv

As for your thing, try my model if you haven’t already. As long as you’re dealing with just one light it looks like it should do what you want. Otherwise you could try playing with the post processing lighting buffer curiosity I mention in the OP to try to build your own shading off of that if you need something involving multiple lights.

[zebrrush]

Yea, I’ve read through your changes already, very cool
setting up my fork rn to try to fiddle with shading

Do you know if it’s possible to separate directional light intensity from everything else in the lighting buffer? (would be funny to put it into alpha channel)

I kinda want to try and keep lumen for the slight colorization of caves/etc, but still have separate colors for light and shadow of directional light
c:

As a last resort I would just reimplement my directional only setup from unity :v