Creating a Custom Shading Model in Unreal Engine 5

Development Asset Creation
,
1

I have a custom shading in Unreal Engine 4 that I’m trying to make work in Unreal Engine 5.

The model is implemented similarly to these two resources:

The issue I’m having is that in Unreal Engine 5 my material is just black. Looking at the visualization of the shader model ID the system seems to think it’s just unlit material.

On the C-side of the engine things seem ok - I can select my shading model in a material editor and my custom pin names and such all work. But on the shader side the shading model ID doesn’t seem to come through correctly.

If I change the code below to use my shading model it uses my shading math, so that part also works. The issue is just that the shading model ID doesn’t seem to come through correctly.

FDirectLighting IntegrateBxDF( FGBufferData GBuffer, half3 N, half3 V, half3 L, float Falloff, float NoL, FAreaLight AreaLight, FShadowTerms Shadow )
{
return CelShadingBxDF(GBuffer, N, V, L, Falloff, NoL, AreaLight, Shadow);

In EngineTypes.h I have the following:

MSM_Strata					UMETA(DisplayName="Strata", Hidden),


//JAMES SEZ our custom cel shading model
MSM_CelShaded               UMETA(DisplayName = "Cel Shading"),


/** Number of unique shading models. */
MSM_NUM						UMETA(Hidden),

In ShadingCommon.ush I have this:

#define SHADINGMODELID_STRATA 12 // Temporary while we convert everything to Strata
#define SHADINGMODELID_CEL_SHADED 13
#define SHADINGMODELID_NUM 14

Maybe there is some issue with having the custom model after strata?

In BasePassPixelShader if I hardcode the ShadingModelID then objects draw using my shading model math, but the objects actually set to my shading model are still black. So it seems like it’s failing at a very high level.

Any ideas? Maybe having anything after MSM_Strata is bad? (Little scared to change that ordering)

Apologies if this is a very niche topic and not directly relevant to the core UE5 changes.

Update: I tried swapping the ordering of Strata and my custom model and that had no effect.

TL;DR - made a new shading model, working in UE4, in UE5 the shadingmodelID for it is unlit and it seems like BasePassPixelShader isn’t even being used at all for the new model.

7 Likes
2

Just jumping in to say i’m interested as well. You’re not alone @jmargaris !

3

I think I can mostly answer my own question:

There is a new file called ShaderGenerationUtil.cpp that handles packing the GBuffer and some other stuff - that file also requires some changes to work with a new shading model. I made two changes to it - if you look at where the EYE shading model is used in the file you need to add similar stuff for your new shading model. DetermineUsedMaterialSlots and void FShaderCompileUtilities::ApplyFetchEnvironment(FShaderMaterialPropertyDefines& SrcDefines, FShaderCompilerEnvironment& OutEnvironment) need changes.

I also changed SetSlotsForShadingModelType but I’m not sure if that code is ever called (??) or what it does exactly…My new shading model uses CustomData0 and 1, which are normally merged into CustomData slot but can also exist as two separate 8-bit channels? I think I understand the intent of this code but I have no idea how it maps values to the slot definitions and some of the slots never seem to get used anywhere…

So:

  1. Change a couple places in ShaderGenerationUtil
  2. Maybe change a third place if you are adventurous

Other than that just doing what worked in UE4 seems to work. I also had to save some materials with my custom shading model as a different shading model then set it back to my custom one to get changes to take, but that might just be a shader caching issue.

7 Likes
4

Glad you figured it out and thanks for sharing your solution !

5

Thanks for sharing! I’m walking through that same Medium article. This should save me a lot of time when I get to that point. I’ll share more as it unfolds

6

Feel free to ask questions here if you run into problems.

7

Great! Thank you so much.

I first thought I merged my code incorrectly.

Just for completion if someone has the same problem: you also need to add your material shading model to the struct FShaderMaterialPropertyDefines in ShaderMaterials.h in order to use it in ShaderGenerationUtil.cpp

1 Like
8

Thanks - forgot to include that in my post. (Tried to edit it in but can’t for whatever reason) It would be nice for someone to create a more up-to-date start-to-finish guide on custom shading models since the existing ones are a bit outdated now but I don’t have the inclination to do it personally.

3 Likes
9

Followed all the steps but now in 5.5 the shading model is recognized as substrate (id (1,1,0))