Creating a custom Substrate Shader Model/BSDF

DMeville · March 26, 2024, 1:48pm

I’ve been wanting to create a custom shader model for my project for a while now, since before Unreal 5 and resources on it are few and far between. This thread will be a working journal as I (try to) get this figured out, and including all the information so others can follow along. If you have successful done this, or are working on it, please let me know! I’d love to be able to collaborate on this (or at least share info, two heads are better than one and I have no idea what I’m doing!) Follow me on twitter @DMevile if you want to follow the struggles. When I have new info I will collect it here. And also in this github fork: https://github.com/DMeville/UnrealEngine-Custom-Substrate/tree/5.4

This is for substrate only. If you are not using substrate, there ARE some guides you can follow.

The current goal is to create a custom Substrate BSDF that has a few extra custom pins in order to:

Do ramped (toon) diffuse lighting via a “Shadow Sharpness” float pin
Do shadow-masked rimlighting via a “Rimlight Intensity” float pin
Do ramped (toon) specular for hair highlights via a “Specular Sharpness” float pin

This will require using a source code build of Unreal Engine 5 from github (I am using 5.4). I am using Jetbrains Rider as my coding environment.

First step is to download the engine and compile it (takes a while) and get the editor up and running. Instructions are on the UE5 github page. Once you have the editor open, make a new project and create a new scene with some objects that are using a material with the Substrate Slab BSDF directly.

Order of things I’m going to try:

First, how to hard code lighting changes on the default BSDF evaluation. This will show me where the final code changes need to happen.
How to create a custom pin on the default BSDF material node
How to take this custom pin data and use it to drive code lighting changes
Take all these changes and make a custom BSDF node with all these options.

Hard Coding Lighting Change
Once you have a working source build and enable substrate. You can change the lighting calculation for the standard slab BSDF pretty easily. This happens in SubstrateEvaluation.ush, in the SubstrateEvaluateBSDFCommon method, Sample.IntegratedDiffuseValue is the final diffuse lighting. By modifying the AreaLightContext.NoL term we can affect how lighting affects the object, so we can just step this value to get toon-lighting.

Replace that line with this, using a ShadowHardness value of 0 to get fully sharp shadows, or 1.0 to get normal non-toon shadows.
Can also do wrap lighting or whatever other kind of lighting you want. But I am doing it like this so that once we have custom pin data, we can just use that data to drive the shadow hardness directly and give it up to the material to control it.

float ShadowHardness = 0.0f;
float NoLStep = smoothstep(0.0, ShadowHardness , AreaLightContext.NoL);
Sample.IntegratedDiffuseValue += (((ShadowTerms.SurfaceShadow) * (NoLStep) * AreaLightContext.Falloff) * Sample.DiffusePathValue * AreaLightContext.AreaLight.FalloffColor)

We can also add a shadow-masked rimlight here easily using a very sharp fresnel and use the shadowmask to scale it away for parts of the mesh in cast shadow.

float3 rimlight = saturate(pow(((1.0 - saturate(NoV)) + 0.25), 50)) * ShadowTerms.SurfaceShadow;
float3 finalRimlight = rimlight*AreaLightContext.AreaLight.FalloffColor*AreaLightContext.Falloff; //scale the rimlight by the lighting falloff and colour, makes it work with other (non directional) lights too

and then just append this finalRimlight at the end of Sample.IntegratedDiffuseValue line to have

Sample.IntegratedDiffuseValue += (((ShadowTerms.SurfaceShadow) * (NoLStep) * AreaLightContext.Falloff) * Sample.DiffusePathValue * AreaLightContext.AreaLight.FalloffColor) + finalRimlight;

Right click the UE5 project in the engine folder, click “Build Selected projects” and wait until it succeeds. Then right click it again and hit “Debug UE5” and it will launch the engine and the project and you should see all lighting is now rim-lit and with hard shadows like this.

For specular it’s just a bit farther down in the same method I think. Find the line “Sample.IntegratedSpecularValue”. I’m just replacing this line with and it seems to work okay for now, but need to look into it more as.

Sample.IntegratedSpecularValue = smoothstep(0.46, 0.5, Sample.SpecularPathValue);

You could just stop here this looks cool and works across all objects using the basic BSDF. But I want to be able to have more control, able to turn the rimlight on/off for different objects and such.

Custom Material Node Pins
Adding a custom pin on an existing BSDF node is actually easy. Happens in the MaterialExpressionSubstrate.h, just find the UMaterialExpressionSubstrateSlabBSDF and add (after the GlintUV property) this

UPROPERTY()
FExpressionInput Custom0;

Next is a case of following where this code goes, and understanding how substrate stores it and how I can access it over on the lighting calculation area to scale the rimlight etc.

DMeville · March 26, 2024, 4:38pm

Adding in the extra Custom0 parameter to all method signatures that need it (there’s a handful of them that I won’t document right now, I just fixed them all until no errors), like SubstrateSlabBSDF in HLSLMaterialTranslator.h and similar code gen functions. Compiles all good,

When I plug in a float param into the new Custom0 node pin on the BSDF in material graph, and compile. Using Window > Shader Code > HLSL Code to view the compiled code, I can see the value being passed in. So that’s something.

But so far all attempts to grab this value has always returned 0 (even if I change the default to something other than 0). So something is not quite working from grabbing the value from the compiled shader and using it in the lighting calculation shader code section.

As a test, If I instead just use the Anisotropy pin on the BSDF material node, and put my float value in that, and grab the Anisotropy value in the lighting calculation code, it does work! (but also controls anisotropy at the same time) so I’m just doing something wrong with the custom pin data so far.

DMeville · March 27, 2024, 5:01pm

Decided to try and start and extend the Unlit BSDF with a custom pin, as that BSDF has a bit less going on (and free VGPRs channels to use) so it would be easier to understand what is going on. After two days of trying things and failing, Got it working after two days. Wooo

Changes here: https://github.com/EpicGames/UnrealEngine/commit/df0b6c595c1cf1d36cd6f9068ec3da7b174eea8a
What it’s doing is adding a new pin, and passing that value along to be used as a multiplier for the emission.

Next step is to try and do similar to the SubstrateSlabBSDF and figure out how to make extra VGPRs or find how to pass data through.

DMeville · March 27, 2024, 7:23pm

Adding in a custom0 pin on the slab BSDF, and using the VGPRs[0].x to store the value (which is the same register that F0.x uses) and using that to drive the rimlight scale in the lighting function from the first post. This makes it so changing the Custom0 value on the pin also controls F0, but as a proof it does work. Current code that does this below:

https://github.com/EpicGames/UnrealEngine/commit/515979d8846247095936b059f8527ff862432250

Current issue is that since the SlabBSDF doesn’t have any empty VGPR register channels like the Unlit BSDF did, not sure how to get the data through other than increasing the number of channels. Single Layer Water BSDF does have some extra channels that I might be able to use. Hmm…

DMeville · March 29, 2024, 11:16pm

Spend a few days and can’t get this working. Trying to re-use the extra InlineVGPRs from the SLW doesn’t work. Extending the VGPRs to have some extra channels doesn’t work. I feel like there’s something I’m missing when it comes to packing/unpacking the data, but from the code it doesn’t appear that the InlineVGPR’s used in SLW do anything like that at all. So I should just be able to use them Frustrating as it feels like I’m so close to getting this working.

DMeville · March 31, 2024, 1:47am

First working test

https://github.com/EpicGames/UnrealEngine/commit/5c87f6ba45a8bb6eadb6e4933ea3131a23b9d0e9

Explanation coming soon but at least here’s some working code

DMeville · March 31, 2024, 5:29pm

Okay so lets break these changes down in how I understand this entire thing works.

Adding a Material pin
Is straightforward. In MaerialExpressionsSubstrate.h UMaterialExpressionSubstrateSlabBSDF method, add your pin after GlintUV near the bottom

UPROPERTY()
FExpressionInput Custom0;

And then we have add a few more changes in MaterialExpressions.cpp on the method overrides for the UMaterialExpressionSubstrateSlabBSDF::GetInputType method we just add a new branch to the if that corresponds where the pin is on the node. Since we added the pin after GlintUV we add our branch in the same location. This Custom0 pin will be pin 18, so we just gotta make sure it’s type is what we want (float)

}
else if(InputIndex == 18)
{
	return MCT_Float; //Custom 0
}

In the same file, we have to fix up UMaterialExpressionSubstrateSlabBSDF::GetInputType so the pin will have the correct label to whatever we want.

}
else if(InputIndex == 18)
{
	return TEXT("Custom 0");
}

In the same file again. We have to add our material input pin and compile it to an int32 “Code Chunk”. This happens on line 24472ish in SlabBSDF::Compile. Adding it after the GlintUVCodeChunk…

	int32 Custom0CodeChunk				= CompileWithDefaultFloat1(Compiler, Custom0, 0.0f); //Custom 0

We need to pass this into the OutputCodeChunk when we compile the full slab a bit below in the Compiler->SubstrateSlabBSDF() call. So add your Custom0CodeChunk below the GintUVCodeChunk as a parameter there.

Substrate Slab BSDF
Now that we changed the SubstrateSlabBSDF() method signature, we have to go through and change all usages of this to allow for the extra parameter we added. This happens in a few places…

HLSLMaterialTranslator.h line 1233
HLSLMaterialTranslator.cpp line 12697
MaterialCompiler.h line 1257, 1273, and 625ish

SubsrateLegacyConversion.ush line 104ish the return GetSubstrateSlab call needs default values for our new params too. Adding a 0.0f after the glintUV line.

With all those changes we should now have everywhere important so we can write out the substrate material…

This happens in HLSLMaterialTranslator.cpp line 12729ish there’s a big long string in the return AddCodeChunk line. We just need to add another “%s,” and then add a new

*SubstrateGetCastParameterCode(Custom0, MCT_Float),

Line after the GlintUV line again. This is getting the pin data and writing it to the compiled material successfully. Which, when called, will call the GetSubstrateSlabBSDF method we just modified with extra params.

We also have to do the same thing again line 12771ish.

And finally, have to full out some default values in DebugProbes.ush line 121ish, adding in 0.0f after the glint parameter.

Packing, Unpacking
All the substrate stuff happens in Substrate.ush. Here we are gunna create a macro for our data, so we can easily bitpack it, and unpack it. This is just how substrate works.
So first lets create a new float value on our FSubstrateBSDF line 184ish, under haziness

float Custom0;

Line 608ish, after the #endif for the SUBSTRATE_INLINE_SINGLELAYERWATER stuff we add a macro to access this for ease of use (and that’s how all the other packed values do it).

#define SLAB_CUSTOM0(X)                 X.Custom0

Note, that macro does not and should not have a semicolon at the end of it!!

We’re gunna add a new parameter to the method signature of InternalGetSubstateSlabBSDF on line 1346ish of our float Custom0 again always after glints.

Inside this method at line 1483ish, under the SLAB_ROUGHNESS call we are going to add

SLAB_CUSTOM0			(SubstrateData.InlinedBSDF)      =   Custom0;

To take the data from the function call and store it in our new float.

We need to add our float params to the method signature of GetSubstrateSlabBSDF on line 1529 and 1569ish, also, and fill in each return InternalGetSubstrateSlabBSDF call to pass through our added parameter also.

And finally add some default values to the call to GetSubstrateSlabBSDF on line 3710ish, adding 0.0 under the GlintUV param.

Now for packing. This happens in SubstrateExport.ush in the PackSubstrateOut method, On line 527 we are going to after the closing brace for the //Data1 block, we are going to add another code block that will take our float Custom0, pack it into a uint, and store it on the substrate header.
Here I am packing the same value 4 times, to show how you can pack multiple values. But you could just pack the single Custom0 once.

//Data2 (Custom0)
{
	const uint PackedCustom0Data = PackR8(SLAB_CUSTOM0(BSDF));
	const uint PackedCustom1Data = PackR8(SLAB_CUSTOM0(BSDF));
	const uint PackedCustom2Data = PackR8(SLAB_CUSTOM0(BSDF));
	const uint PackedCustom3Data = PackR8(SLAB_CUSTOM0(BSDF));

	const uint ReplicatedData = PackedCustom0Data | (PackedCustom1Data << 8) | (PackedCustom2Data << 16) | (PackedCustom3Data << 24);
	SUBSTRATE_STORE_UINT1(ReplicatedData);
}

Then to unpack it. This happens back in Substrate.ush in the UnpackFastPathSubstrateBSDFIn method. We need to do a few things.
First, add this line under the uint Data1 variable on line 3932. Since we added a new uint1 to our substrate, we need to get it out.

uint Data2 = SubstrateLoadUint1(SubstrateBuffer, SubstrateAddressing); //custom 0 x 4 channels

Then we need to unpack the floats from that buffer. So below the PackedDiffuse20Bits variable on line 3943 add the following line. Again showing how to pack 4 values, but since we packed the same value 4 times all these will have our pin data.

const uint PackedCustomData0_8Bits = 0xFF & (Data2); //custom 0
const uint PackedCustomData1_8Bits = 0xFF & (Data2 >> 8); //custom 1
const uint PackedCustomData2_8Bits = 0xFF & (Data2 >> 16); //custom 2
const uint PackedCustomData3_8Bits = 0xFF & (Data2 >> 24); //custom 3

And then a bit below on line 3953 under the SLAB_ANISOTROPY line, we are going to take our now unpacked data add assign it to our SLAB_CUSTOM0 macro.

SLAB_CUSTOM0(OutBSDF) 		= UnpackR8(PackedCustomData0_8Bits); // Custom0

We can now USE this SLAB_CUSTOM0 value in our actual shader code to do whatever we want with! Huzzah!

Using The Data

Going back to the first post in this thread. We can now just use this macro to control our rimlight or shadow sharpness.

In SubstrateEvaluation.ush line 392ish. Before the Sample.IntegratedDiffuseValue we can grab the lighting data, calculate a rimlight, or do whatever we want here.

Here I have two custom values, SLAB_CUSTOM0, and SLAB_CUSTOM1 which control both effects.

float ShadowHardness = SLAB_CUSTOM1(BSDFContext.BSDF); // value of 0 is hard shadows, 1 is normal lighting
float NoLStep = smoothstep(0.0, ShadowHardness , AreaLightContext.NoL); //calculate the stepped shadow
			
float rimlightScale = 1.0f; //We want to eventually hook this up to a custom pin float
rimlightScale = SLAB_CUSTOM0(BSDFContext.BSDF);
float3 rimlight = saturate(pow(((1.0 - saturate(NoV)) + 0.25), 50)) * ShadowTerms.SurfaceShadow * rimlightScale; //sharp fresnel rimlight, scaled by SurfaceShadow mask
float3 finalRimlight = rimlight*AreaLightContext.AreaLight.FalloffColor*AreaLightContext.Falloff; //scaled by light params so it works with all light types (?)

Sample.IntegratedDiffuseValue += (((ShadowTerms.SurfaceShadow) * (NoLStep) * AreaLightContext.Falloff) * Sample.DiffusePathValue * AreaLightContext.AreaLight.FalloffColor) + finalRimlight;

And vooolia! It works!

Source code is available in the link to the github fork of 5.4 in the original post. Will work to clean it up a little bit and make sure it actually works in all cases. But for now, very happy to have this working and can make some cool lighting tricks for my game, toon, half-lambert, stylized spec, whatever!

DMeville · April 6, 2024, 8:16pm

I have noticed that using any value in anisotropy, fuzz, and probably glints causes this to break. Rimlight disappears, and the shading snaps to hard shadows for some reason. Been looking into why this happens for the last few days but haven’t been able to make sense of it.

VerI0 · April 7, 2024, 2:51pm

nice job

thewhimer · May 25, 2024, 10:37pm

And you left out several optimized subcases for substrate single slab mode.

Search if(OptimisedLegacyMode == SINGLE_OPTLEGACYMODE_CLEARCOAT) in SubstrateExport.ush, to see these subcases. Though they are named with legacy, but actually they are special optimized modes that just happen to corespond to those separate legacy shading models.

For example, if fuzz is enabled and fuzz-roughness is the same as slab roughness, and sub-surface scattering is disabled, then SINGLE_OPTLEGACYMODE_CLOTH is activated. Code as below

if (BSDF_GETHASFUZZ(BSDF) && SLAB_ROUGHNESS(BSDF) == SLAB_FUZZ_ROUGHNESS(BSDF))
{
	OptimisedModeMask |= SingleHasCloth;
}

...

else if ((OptimisedModeMask & SingleHasCloth) == SingleHasCloth && BSDF_GETSSSTYPE(BSDF) == SSS_TYPE_INVALID)
{
	OptimisedLegacyMode = SINGLE_OPTLEGACYMODE_CLOTH;
}

thewhimer · May 25, 2024, 10:38pm

Hi, after two days of reading source code about substrate shaders and materials, I wonder if anyone has already tried to create a custom shading model in substrate. Then google leads me here, nice work @DMeville !

The problem you describe comes from forgetting to update the required byte size that substrate has to allocate for every pixel.

Search SubstrateMaterialRequestedSizeByte += UintByteSize in HLSLMaterialTranslator.cpp, to see how the byte size is computed.

DMeville · May 26, 2024, 5:23am

And you left out several optimized subcases for substrate single slab mode.

Yes. This is not meant to be a comprehensive replacement all ready to go 100% covering/replacing all optimized passes. Most of these subcases are not actually features I’m using, and don’t plan on using honestly for my project. So I didn’t want to spend too much more time stumbling around, when I already felt like I had no idea what I am doing! But the code is on github, so if you feel like it PR’s are open

The problem you describe comes from forgetting to update the required byte size that substrate has to allocate for every pixel.
Search SubstrateMaterialRequestedSizeByte += UintByteSize in HLSLMaterialTranslator.cpp, to see how the byte size is computed.
Which problem exactly? The fuzz/glints not working when adding extra data? If so, that’s awesome I’ll take a look! Good to have another set of eyes looking at this!

thewhimer · May 26, 2024, 3:08pm

Yes, I think both are related to the problem of fuzz/glints not working when used, as explained above.

SubstrateMaterialRequestedSizeByte needs to be updated.
When fuzz is enabled but with no other advanced/opt-in features, packing/unpacking bytes will follow the SINGLE_OPTLEGACYMODE_CLOTH subcase code path, which means that code path should consider the additional Custom0-4 data too. Or you make sure to avoid those subcase code paths when doing feature tests.

Saukkoman · June 8, 2024, 10:40am

Love this system! Have you managed to get this engine working in a build? I tried to package a project but seems like Substrate is making things very difficult.

DMeville · June 12, 2024, 4:56am

Thanks! You know that completely slipped my mind and I haven’t even tried to package a build. I’m still an unreal noob but I’ll take a look!

DMeville · June 15, 2024, 4:56am

Spent some time trying to add a custom input to the SingleLayerWater BSDF, so I can pass in an extra value for sparkly water. Usually this would just be done in emission, but I HATE when the sparkles show up in shadows/darkness (since emission isn’t masked by shadows). So this fixes that. Code updated in Github now that I finally got this working. Will clean up the commit next (the actual water in this example is still super wip and hasn’t got a pretty pass yet, as I JUST got things working)

DMeville · July 16, 2024, 1:13am

There was an issue with packaging a project, with an error about Substrate material failing to compile by the material compiler. After a week of building and testing and trying to narrow down what was causing this, it seems that in Substrate.ush the way I was just adding extra floats for whatever custom data I wanted (eg, float Custom0) worked fine in testing, but when packaging it didn’t like this. No idea WHY adding extra variables is not allowed, but removing those variables made things package properly.

I converted my code to just piggyback on the VGPRs that are defined in Substrate.ush, and adding a few extra registers to use for 8 custom properties on those, and things appear to package and work correctly now.

Github has been updated with these changes.

kvickerz · August 1, 2024, 3:52am

Hey, nice work, somewhat of a side tangent, but what are you doing to mask the sparkles with the shadow? I’m trying to figure out a way to do a similar effect by sampling shadows

DMeville · August 4, 2024, 7:13pm

Thanks! Masking (that custom spec input) by shadows is part of what the custom BSDF (in this case the extended SLW BSDF) does. There’s no way to do it just via material graph, unfortunately, which was always a point of frustration for me. So I wanted to add something that could do something similar.

In this case, it’s just taking that input, and masking it by the lighting. You can see where this happens for SLW in ForwardLightCommon.ush, line 256 (https://github.com/EpicGames/UnrealEngine/commit/df79152eac7bb114009496f8b29931d4833b12c3)

kvickerz · August 5, 2024, 11:38pm

Ah, thanks for the reply, not willing to do a custom engine build for this but hope it becomes possible through easier methods in the future, thanks!