Community Tutorial: New shading models and changing the GBuffer

[0ne3y3]

Implementing a Celshading model directly into UE5.1 source. This celshading use a linear color curve atlas to drive all the values.
Learn how to set your own shading model, add a render target to GBuffer, storing information in the View struct, creating a global node for material graph. Some tips and tricks
We will see restriction about mobile and raytracing.

https://dev.epicgames.com/community/learning/tutorials/2R5x/unreal-engine-new-shading-models-and-changing-the-gbuffer

[pemjahat]

Hello @One3y3.
Great post, haven’t dig deep into gbuffer section first but the rest of it is just great.
So I want to point out something that bothers me:
AFAIK the only communication between base pass (gbuffer) and deferred pass(lighting) is through gbuffer shading model.
The problem is shading model encoded in gbuffer (4byte), sharing 4byte with selective output mask. (Check EncodeShadingModelIDAndSelectiveOutputMask).
Now the code in tutorial using up to 16 (with SHADINGMODEL_NUM assigned to 16).
It might be works in this case, if it’s not involving raytrace, but it might be harder to extend it beyond this. (For example Toon model for hair, etc).
Changing the number of bits allocated to store GBuffer shading model is not preferrable (due to the amount of engine files touched - integration nightmare).
So I wonder if anyone has better approach to extends this beyond shading model id.

[0ne3y3]

From what I saw in the gbuffer generation, Shadingmodelid is a int value packed into a uint4 with Outputmask, which then is converted into a float in the alpha channel (because textures are only 0-1 range).

I don’t have any problem when enabling raytracing shadow / raytracing skylight and hardware Lumen in my case.

GBuffer is the only way to communicate specific material properties to the deffered light rendering pass. There is some global struct like the view structure, but it’s common to the whole scene. The only way would be to add your own subpass in between basepass and light pass. But that’s like re-coding the whole engine and raytracing implementation

[AnaWilliam850]

Hi! great post and thank you for the tutorial

[pemjahat]

Hello.

Here is the comment taken from ShadingCommon.ush

// SHADINGMODELID_* occupy the 4 low bits of an 8bit channel and SKIP_* occupy the 4 high bits

so in essence, ShadingModelID uses 1 channel on GBuffer, and in this channel shared half of them with OutputMask.
So effectively only 4 bits are allowed, which is 15 only.
I haven’t found any issue so far, it might be I haven’t used OutputMask extensively yet.

I am still struggling to find the proper solution to extend this ShaderModelID beyond 15 without not much gbuffer encoding modification. Still have so many things to learn and explore

[shinryuk3n1]

@0ne3y3 thanks for this awesome guide…
@pemjahat I dont know much about this but is extending to uint32 not a good solution for your case?

[pemjahat]

Anything changing the gbuffer packaging is a red flag at this moment.
Currently, this packed ShadingModel and MaterialMask only have room for uint8 in gbuffer.

I am exploring another solution, like Strata too see if it’s feasible.

[shinryuk3n1]

Thanks again for your guide… this is working for me (havent done shadows yet), but I’m a bit stuck on the curve atlas usage. Does the curve atlas require a specific type of compression, such as Alpha or Grayscale? If I dont use one of those, I get a stair-step effect at what seems to be the 0 value, please see attached picture.

Its like, the light response is taking up all of the Curve information.
Here is the curve:

I think there is something to do with ‘Disable Adjustments’ and Compression options within the Curve Atlas settings, but I don’t have enough understanding. All I know is that Disable Adjusments preserves negative values in curves, but not sure how this applies here since my curve goes from 0,0 to 1,1… does it mean that the full range of light needs to be accounted for (including negatives?) Anyway, here are some things I have experimented…

If I ‘Disable Adjustments’ and plug Specular with the same node as Base Color (with atlas compression still Default), I get slightly better result, but don’t fully know why:

If I keep Adjustments disabled, but change compression to Grayscale (or Alpha), it seems to give the proper result, including the light response. I’m guessing that when plugging the same node as Base Color into Specular, I am essentially overwriting the light response to be that of Base Color, which is why it disappears, but this raises a question - If I do want to disable the light response, is that the correct way to do it? Setting roughness to 1 also works, maybe that is the correct way. Or should I be handling that in the Color Curve?. At this stage though, I’m still unsure how the light response and the rest of the shading are encompassed within a single curve like mine:

I guess I’m just more confused than ever Any help to clarify would be greatly appreciated! I know I have more learning to do regarding lighting, shaders etc…

[0ne3y3]

I didn’t touch any settings on my curve atlas, except disabling adjustments. It didn’t change anything with or without.

After checking my curve atlas settings, it set the texture to HDR (RGBA16F) by default.
You definitely need, at least RGBA 8bits no sRGB, so VectorDisplacementMap.
The curve is encoded into only 1 pixel height, so with the texture compression, it will blend between the curve at the bottom and at the top. Which is full white in your case. Also, no sRGB because it’s linear “math” data and will also blend your curves together.

---

On another topic.
I made a small correction, if we don’t use the new gbuffer rendertarget. We need to add the default celshading to HasCustomData() in the HLSL file, else it doesn’t write the curve selection value to GBuffer.CustomData.a .

[ShinraCorp]

Hey this is a good tutorial, still going through it but small typo found.

Our atlas RHI texture, its sampler and a global uint for the texture height (used later to determine which curve to choose).

Now we need to initialize our values in the constructor. It’s MANDATORY because the texture and values won’t be sent at the start of the engine and the Renderthread really doesn’t like nullptr (the engine will just crash).

In the file SceneManagement.h, go to the constructor FViewUniformShaderParameters::FViewUniformShaderParameters() and add at the end

Should be SceneManagement.cpp instead of the header file.

[Gankable]

Thank you for this in-depth tutorial. I really love seeing the deeper dives and concepts on the forums and hope to one day also be able to add my own content to advanced concepts. Thanks again!

[unreal_toon_learner]

Hi, Thank you for the tutorial.
However I get the following error in UE5 Material Editor after going through step 8 B Celshading calculation:

“[SM5] /Engine/Private/ShadingModelsMaterial.ush:211:35: error: use of undeclared identifier ‘GetCelShadingSelection0’
GBuffer.CustomData.a = saturate(GetCelShadingSelection0(MaterialParameters)/View_CelshadingTextureHeight);
^”

Am I missing something?
Thanks

[0ne3y3]

I can see 2 things.

1. You didn’t added the Celshading node in your material graph

2. The name of the function isn’t good in the celshading node class

virtual FString GetFunctionName() const override { return TEXT("GetCelShadingSelection"); }

[unreal_toon_learner]

Thank you very much! it works:

I can now make the cel-shading as shown above.

However the shadow part is seems still smooth:

I have already followed “Celshading shadow cast by other object” section but the shadow part is still seems smooth.

Am I still missing something in the shadow part?
Thanks!

[0ne3y3]

The chair’s shadow on the ground is because your ground isn’t using the celshading model.

The “smooth shadow” on the object itself is Lumen GI / ambient occlusion kicking in

[unreal_toon_learner]

Thanks. After I disable the Lumen GI, I get below:

Which is not a smooth shadow finally.

So does it means that if I want non-smooth shadow in this cel shading model, I cant use Lumen GI?

Another question is that I find the lit part is too bright, comparing to the base color of the texture.

You can see the color (yellow color) comparison below:

How can I control the brightness of the lit part?

Thanks so much again for your guidance and help.

[0ne3y3]

The goal of global illumination (GI) is to make the light bounce of the surface and illuminate more or less the surface, so yes you will get smooth shade with Lumen GI.

For the color, you can tone down the intensity of your light, use a darker color or instead of using 1 for the celshading curve, use 0.75. It’s up to you in your asset creation pipeline how to tackle this.

[Min1]

Thank you for the tutorial! It’s so detailed.
It works on 5.3. Now I can apply this in my projects. Some problems left… I’d like to use vertex color in my codes (maybe in ShadingModels.ush? I’m not sure).
Any help would be appreciated
Thanks again!

[Min1]

I think I have a new problem… Now color can be only written in Base Color when Blend Mode is Opaque. W hen I change Blend Mode into Translucent, the whole model turn into black.

[ScottDunbar.io]

Hi 0ne3y3. First off. Thank you for this very detailed guide. Very insightful!

I’m trying to go the Non GBuffer route. While I am proficient in C++, this is my first deep dive into messing with unreal ush/usf shaders so I’m trying to put together the inferred pieces, specifically how you detail out how to write the function: “CelshadingPreintegratedSkinBxDF”

Is this what you meant by " I copied PreintegratedSkinBxDF() and celshade the dot(N, L) * .5 + .5 . "

FDirectLighting CelshadingPreintegratedSkinBxDF( FGBufferData GBuffer, half3 N, half3 V, half3 L, float Falloff, half NoL, FAreaLight AreaLight, FShadowTerms Shadow )
{
	FDirectLighting Lighting = DefaultLitBxDF( GBuffer, N, V, L, Falloff, NoL, AreaLight, Shadow );
	
	half3 SubsurfaceColor = ExtractSubsurfaceColor(GBuffer);
	half Opacity = GBuffer.CustomData.a;

	float CurveIndex = 1.0f / View.CelshadingTextureHeight;
	float DotValueCelShaded = Texture2DSampleLevel(View.CelshadingAtlas, View.CelshadingSampler, CurveIndex, 0).b;	
	float OldDotValue = (dot(N, L) * .5 + .5);

	//half3 PreintegratedBRDF = Texture2DSampleLevel(View.PreIntegratedBRDF, View.PreIntegratedBRDFSampler, float2(saturate(OldDotValue), 1 - Opacity), 0).rgb;
	half3 PreintegratedBRDF = Texture2DSampleLevel(View.PreIntegratedBRDF, View.PreIntegratedBRDFSampler, float2(saturate(DotValueCelShaded), 1 - Opacity), 0).rgb;

	Lighting.Transmission = AreaLight.FalloffColor * Falloff * PreintegratedBRDF * SubsurfaceColor;

	return Lighting;
}

Thanks!!!