Creating custom shaders and binding them with the material editor

[RyanTorant]

Hy everyone. I’m working on GI inside UE4 (Realtime Dynamic GI + Reflections + AO + Emissive plugin - AHR - Game Development - Epic Developer Community Forums), but I’m stuck now with a problem that I hope someone is able to solve.
I created a bunch of diferent shaders that are used to render the objects, and as I didn’t find documentation, I created them copying the base pass shader classes.
For example, here’s the vertex shader class I use on the voxelization stage

class FAHRVoxelizationVertexShader : public FMeshMaterialShader
{
	DECLARE_SHADER_TYPE(FAHRVoxelizationVertexShader,MeshMaterial);

protected:
	FAHRVoxelizationVertexShader() {}
	FAHRVoxelizationVertexShader(const FMeshMaterialShaderType::CompiledShaderInitializerType& Initializer):
	FMeshMaterialShader(Initializer)
	{
	}

public:
	static bool ShouldCache(EShaderPlatform Platform,const FMaterial* Material,const FVertexFactoryType* VertexFactoryType)
	{
		return IsFeatureLevelSupported(Platform, ERHIFeatureLevel::SM5);
	}

	static void ModifyCompilationEnvironment(EShaderPlatform Platform, const FMaterial* Material, FShaderCompilerEnvironment& OutEnvironment)
	{
		FMeshMaterialShader::ModifyCompilationEnvironment(Platform, Material, OutEnvironment);
	}

	virtual bool Serialize(FArchive& Ar)
	{
		bool bShaderHasOutdatedParameters = FMeshMaterialShader::Serialize(Ar);
		return bShaderHasOutdatedParameters;
	}

	void SetParameters(
		FRHICommandList& RHICmdList, 
		const FMaterialRenderProxy* MaterialRenderProxy,
		const FVertexFactory* VertexFactory,
		const FMaterial& InMaterialResource,
		const FSceneView& View
		)
	{
		FMeshMaterialShader::SetParameters(RHICmdList, GetVertexShader(),MaterialRenderProxy,InMaterialResource,View, ESceneRenderTargetsMode::DontSet);
	}

	void SetMesh(FRHICommandList& RHICmdList, const FVertexFactory* VertexFactory,const FSceneView& View,const FPrimitiveSceneProxy* Proxy,const FMeshBatchElement& BatchElement)
	{
		FMeshMaterialShader::SetMesh(RHICmdList, GetVertexShader(),VertexFactory,View,Proxy,BatchElement);
	}

private:
};

and the shader

// @
#include "AHRCommon.usf"
#include "AHRVoxelizationCommon.usf"

void Main(
	FVertexFactoryInput Input,
	OPTIONAL_VertexID
	out FAHRVoxelizationVSOut Output
	)
{
	FVertexFactoryIntermediates VFIntermediates = GetVertexFactoryIntermediates(Input);
	float4 WorldPositionExcludingWPO = VertexFactoryGetWorldPosition(Input, VFIntermediates);

	float3x3 TangentToLocal = VertexFactoryGetTangentToLocal(Input, VFIntermediates);	
	FMaterialVertexParameters VertexParameters = GetMaterialVertexParameters(Input, VFIntermediates, WorldPositionExcludingWPO.xyz, TangentToLocal);

	Output.FactoryInterpolants = VertexFactoryGetInterpolantsVSToPS(Input, VFIntermediates, VertexParameters);
	Output.Position = WorldPositionExcludingWPO;

	// add the material offset
	ISOLATE
	{
		Output.Position.xyz += GetMaterialWorldPositionOffset(VertexParameters);
	}
}

In this case, GetMaterialWorldPositionOffset(VertexParameters) returns 0 always, no mater what do you create on the material editor.
My question is what is wrong or what do I need to do to get that information inside my shader, so that I can drive the shader parameters trough the material editor. Things like world offset, or emissive color.
It would be awesome if someone can explain a bit how the stage from the material editor to the shader works, so for example, what things I need to set up to get the base color from the material editor, or the emissive color.
I tried looking at the shaders, and from what I can see, the material compiler replaces the %s inside the shader with code generated from the material ( at least I assume it does that), but an explanation will help a lot
Hope I was clear enough, but feel free to ask anything.

EDIT:
Here’s the code for AHRVoxelizationCommon.usf

// @

#include "Common.usf"
#include "BasePassCommon.usf"
#include "Material.usf"
#include "VertexFactory.usf"

struct FAHRVoxelizationVSOut
{
	FVertexFactoryInterpolantsVSToPS FactoryInterpolants;
	float4 Position : TEXCOORD7;
};

struct FAHRVoxelizationGSOut
{
	FVertexFactoryInterpolantsVSToPS FactoryInterpolants;
	float4 Position : SV_POSITION;
	float3 wPos : TEXCOORD7;
};

[anonymous_user_f2b8b290]

DECLARE_SHADER_TYPE(FAHRVoxelizationVertexShader,MeshMaterial);

When you make a new MeshMaterial shader, you also have to make a IMPLEMENT_MATERIAL_SHADER_TYPE somewhere in a cpp and then make a drawing policy that actually uses the shader for rendering.

I assume you want to compute material color for voxels in your volume texture. One way to do this is to rasterize each mesh from the 3 axis directions, building a linked list per voxel of the intersecting fragments. Then in a second pass you average and normalize the diffuse and emissive color of the voxel. This is very similar to what LPV does so you can use that code as guidance.

[RyanTorant]

@DanielW;212018

When you make a new MeshMaterial shader, you also have to make a IMPLEMENT_MATERIAL_SHADER_TYPE somewhere in a cpp and then make a drawing policy that actually uses the shader for rendering.

I assume you want to compute material color for voxels in your volume texture. One way to do this is to rasterize each mesh from the 3 axis directions, building a linked list per voxel of the intersecting fragments. Then in a second pass you average and normalize the diffuse and emissive color of the voxel. This is very similar to what LPV does so you can use that code as guidance.

I am calling that on a cpp:

IMPLEMENT_MATERIAL_SHADER_TYPE(,FAHRVoxelizationVertexShader,TEXT("AHRVoxelizationVS"),TEXT("Main"),SF_Vertex);
IMPLEMENT_MATERIAL_SHADER_TYPE(,FAHRVoxelizationGeometryShader,TEXT("AHRVoxelizationGS"),TEXT("Main"),SF_Geometry);
IMPLEMENT_MATERIAL_SHADER_TYPE(,FAHRVoxelizationPixelShader,TEXT("AHRVoxelizationPS"),TEXT("Main"),SF_Pixel);

and I’m using the shader, as the voxelization part is working correctly, but, for example, if I create a material with world offset, the voxel pass is unaware of the world offset. Also, I’m using as a placeholder a emissive color per material, I would like to use the emissive color output of the node editor.
That’s what I’m asking, but I’ll look at that code, see if I can see how it works.
Also, I would like to write a custom RSM rendering pass, as my needs are different from the ones of LPV, and to do that I would like to know how to get acess to the base color outputted from the node editor

EDIT: For the voxelization part I’m actually using the openGL insights method, where you use a geometry shader to select the dominant axis and then write to a UAV on the ps
EDIT 2: Here’s the code for the drawing policy, just in case

/**
* drawing policy to voxelize a mesh
*/
class FAHRVoxelizerDrawingPolicy : public FMeshDrawingPolicy
{
public:
	/** The data the drawing policy uses for each mesh element. */
	class ElementDataType
	{
	public:
		/** Default constructor. */
		ElementDataType()
		{}
	};

	FAHRVoxelizerDrawingPolicy( const FVertexFactory* InVertexFactory,
							    const FMaterialRenderProxy* InMaterialRenderProxy,
							    const FMaterial& InMaterialResource,
								ERHIFeatureLevel::Type InFeatureLevel,
								FAHRVoxelizerDrawingPolicyFactory::ContextType* _context) : FMeshDrawingPolicy(InVertexFactory,InMaterialRenderProxy,InMaterialResource,bOverrideWithShaderComplexity)
	{
		context = _context;

		// Get the shaders
		VertexShader = InMaterialResource.GetShader<FAHRVoxelizationVertexShader>(InVertexFactory->GetType());
		GeometryShader = InMaterialResource.GetShader<FAHRVoxelizationGeometryShader>(InVertexFactory->GetType());
		PixelShader = InMaterialResource.GetShader<FAHRVoxelizationPixelShader>(InVertexFactory->GetType());
	}

	// FMeshDrawingPolicy interface.
	bool Matches(const FAHRVoxelizerDrawingPolicy& Other) const
	{
		return FMeshDrawingPolicy::Matches(Other) &&
			VertexShader == Other.VertexShader &&
			PixelShader == Other.PixelShader &&
			GeometryShader == Other.GeometryShader;
	}

	void SetSharedState(FRHICommandList& RHICmdList, const FViewInfo* View, const ContextDataType PolicyContext) const
	{
		// Set the shaders parameters
		VertexShader->SetParameters(RHICmdList, MaterialRenderProxy, VertexFactory, *MaterialResource, *View);
		GeometryShader->SetParameters(RHICmdList, MaterialRenderProxy, VertexFactory, *MaterialResource, *View);
		PixelShader->SetParameters(RHICmdList, MaterialRenderProxy,*MaterialResource,View);
	}

	FBoundShaderStateInput GetBoundShaderStateInput(ERHIFeatureLevel::Type InFeatureLevel)
	{
		return FBoundShaderStateInput(
			FMeshDrawingPolicy::GetVertexDeclaration(), 
			VertexShader->GetVertexShader(),
			FHullShaderRHIParamRef(), 
			FDomainShaderRHIParamRef(), 
			PixelShader->GetPixelShader(),
			GeometryShader->GetGeometryShader());
	}

	void SetMeshRenderState(
		FRHICommandList& RHICmdList, 
		const FViewInfo& View,
		const FPrimitiveSceneProxy* PrimitiveSceneProxy,
		const FMeshBatch& Mesh,
		int32 BatchElementIndex,
		bool bBackFace,
		const ElementDataType& ElementData,
		const ContextDataType PolicyContext
		) const
	{
		const FMeshBatchElement& BatchElement = Mesh.Elements[BatchElementIndex];

		VertexShader->SetMesh(RHICmdList, VertexFactory,View,PrimitiveSceneProxy,BatchElement);
		GeometryShader->SetMesh(RHICmdList, VertexFactory,View,PrimitiveSceneProxy,BatchElement);
		PixelShader->SetMesh(RHICmdList, VertexFactory,View,PrimitiveSceneProxy,BatchElement);

		context->RHICmdList->SetDepthStencilState(TStaticDepthStencilState<false, CF_Always>::GetRHI());
		context->RHICmdList->SetRasterizerState(TStaticRasterizerState<FM_Solid,CM_None,false,false>::GetRHI());
		auto gridCFG = AHREngine.GetGridSettings();
		auto imax = ](uint32 a, uint32 b){ uint32 tmp = a > b; return a*tmp + (1 - tmp)*b; };
		uint32 gres = imax(gridCFG.SliceSize.X,imax(gridCFG.SliceSize.Y,gridCFG.SliceSize.Z));
		context->RHICmdList->SetViewport(0,0,0,gres*2,gres*2,1);
		Mesh.VertexFactory->Set(*context->RHICmdList);
	
		// Bind the voxels UAV and bind a null depth-stencil buffer
		FUnorderedAccessViewRHIParamRef uavs] = { AHREngine.GetSceneVolumeUAV(),AHREngine.GetEmissiveVolumeUAV() };
		context->RHICmdList->SetRenderTargets(0,nullptr,nullptr,2,uavs);

		//FMeshDrawingPolicy::SetMeshRenderState(RHICmdList, View,PrimitiveSceneProxy,Mesh,BatchElementIndex,bBackFace,FMeshDrawingPolicy::ElementDataType(),PolicyContext);
	}

	friend int32 CompareDrawingPolicy(const FAHRVoxelizerDrawingPolicy& A,const FAHRVoxelizerDrawingPolicy& B)
	{
		COMPAREDRAWINGPOLICYMEMBERS(VertexShader);
		COMPAREDRAWINGPOLICYMEMBERS(PixelShader);
		COMPAREDRAWINGPOLICYMEMBERS(GeometryShader);

		return 0;
	}
private:
	FAHRVoxelizerDrawingPolicyFactory::ContextType* context;
	FAHRVoxelizationVertexShader* VertexShader;
	FAHRVoxelizationGeometryShader* GeometryShader;
	FAHRVoxelizationPixelShader* PixelShader;
};

[RyanTorant]

Well, I gave the code you said a look, but I’m still stuck with this. Anyone can help?

[RyanTorant]

Well, I still haven’t been able to fix this.
I know devs are busy, but I really think that for a project like the Unreal Engine, documentation about thinks like that should be provided for paying customers.
I’m not asking for people to solve my problems, I’m asking for documentation on a feature that is basic for anyone working on the source.
I hope to get an answer soon, this is really stopping me, and I hate wasting my time trying to find how things are done when they could have been either documented or solved in forum thread

[RCaloca]

Your changes involve one of the most non-straightforward aspects of the renderer, hence the complexity/lack of documentation. We really want to document all this, but also want to fix crashes & bugs… So it gets bumped in priority.

If you want to add a new pass, you can take a look at VelocityRendering.h/.cpp; search the codebase for FVelocityDrawingPolicy and FVelocityDrawingPolicyFactory; track down who calls FVelocityDrawingPolicyFactory::AddStaticMesh() and FVelocityDrawingPolicyFactory:: DrawDynamicMesh() and see where and when they are used.

You’ll also need to add something like TStaticMeshDrawList<FVelocityDrawingPolicy> VelocityDrawList in FScene.

Good luck!

[RyanTorant]

@RCaloca;214218

Your changes involve one of the most non-straightforward aspects of the renderer, hence the complexity/lack of documentation. We really want to document all this, but also want to fix crashes & bugs… So it gets bumped in priority.

If you want to add a new pass, you can take a look at VelocityRendering.h/.cpp; search the codebase for FVelocityDrawingPolicy and FVelocityDrawingPolicyFactory; track down who calls FVelocityDrawingPolicyFactory::AddStaticMesh() and FVelocityDrawingPolicyFactory:: DrawDynamicMesh() and see where and when they are used.

You’ll also need to add something like TStaticMeshDrawList<FVelocityDrawingPolicy> VelocityDrawList in FScene.

Good luck!

That’s not what I want to do.
I just want to know how to get a custom material shader working with the material editor.
Let’s think on a hypothetical case. Let’s say I want to render the base color of an object to a target. I have everything working to the point that I can get my objects to render a color to the target. Now, to get the base color I should call GetMaterialBaseColor( MaterialParameters ). What things do I need to set up on my shader so that GetMaterialBaseColor returns the color that the material editor says that it should return?
Hope it’s clear what I’m asking now. I have the pass working, I have the object selection working, everything is working except the things that are driven trough the material editor. A more real example, is that the world offset defined on the material editor has no effect on the voxelization shader. Everything gets voxelized as if there where no offset, even when there is in the material editor.

[anonymous_user_52a14607]

Seems like you’re doing everything correctly, from looking at your code. It might be something small that’s hidden somewhere else in the source which you might’ve missed.
In situations like this, I usually have to resort to a brute-force method of recreating an existing effect (maybe something like PostProcessMaterial.cpp as it is simple), i.e. duplicating all of that code as PostProcessMaterial2.cpp and its associated files and seeing if it still works. Then compare that code to see what differences there are with my own effect. This is how I’ve always debugged these sorts of problems when there is no documentation available.

[RyanTorant]

@gboxentertainment;214328

Seems like you’re doing everything correctly, from looking at your code. It might be something small that’s hidden somewhere else in the source which you might’ve missed.
In situations like this, I usually have to resort to a brute-force method of recreating an existing effect (maybe something like PostProcessMaterial.cpp as it is simple), i.e. duplicating all of that code as PostProcessMaterial2.cpp and its associated files and seeing if it still works. Then compare that code to see what differences there are with my own effect. This is how I’ve always debugged these sorts of problems when there is no documentation available.

That was exactly what I wanted to avoid, but if I don’t get an answer anytime soon I’ll have to do it, as I’m about to ran out of things to do that don’t require the material thing

[RyanTorant]

Well, most of it it’s working now, the only part that’s still not working is world offset.
The code I posted earlier works, and if you need to access some mat thing you need to call first

FMaterialPixelParameters MaterialParameters = GetMaterialPixelParameters(input.FactoryInterpolants, input.Position);
CalcMaterialParameters(MaterialParameters,true, input.Position);

and then you can use things like GetMaterialEmissive(MaterialParameters).
Any idea why world offset would behave different?
EDIT: A quick test shows that materials driven trough the time parameter are not working, it’s fixed at 0

[anonymous_user_52a14607]

@;214710

Any idea why world offset would behave different?
EDIT: A quick test shows that materials driven trough the time parameter are not working, it’s fixed at 0

Have you checked if it has anything to do with this?:
Under void CalcMaterialParameters() in MaterialTemplate.usf, there is a line saying //If the material uses any non-offset world position expressions, calculate those parameters. If not, the variables will have been initialised to 0 earlier.

Also, with the time parameter, would it have anything to do with the static voxelization? I’m assuming that all other material expressions (i.e. add, subtract, multiply, etc.) are all working?

[Venom89]

Hey @ , how do you assign this FMeshMaterial to your static mesh?

[ZkarmaKun]

Hey @ I know this is one real old, but I am facing a similar problem than you, except that in mine the WorldPosition is never there, for the offset I did find this:

OutEnvironment.SetDefine(TEXT(“HAS_SCREEN_POSITION”), (bool)Parameters.MaterialParameters.bHasVertexPositionOffsetConnected);

It looks like you can force some functionality with Defines inside the ModifyCompilationEnvironment static function, unfortunately non I used works for me

[rgreene_kint]

@ I know this is old, but did you ever find a good example or guide for how the shader binding system works? I’m in a similar situation where I’m trying to make sense of how it actually works (currently I’m learning via breakpoints throughout the code and trying to piece together how all the datastructures and tasks work together).

I also very much wish this was documented, as the code is really hard to follow since so much works via #define magic. Like, could we just get a non-optimized minimal code example that demonstrates how things actually go through the pipe from culling, draw call building, binding on the RHI?

My situation is a little different to what spawned this post: I’m trying to use a material shader on a fullscreen pass with some additional shader parameters: Implementing new render pass questions

I’ll take a peek at at the velocity rendering to see how that works.