[TUTORIAL] Edge Detection Post Process Effect using World Normals

Hello fellow Unreal Engine users!

This is not really a tutorial in that I guide you through every single step. Instead I’m going to throw my Material asset (including comments) at you for everyone to use, adjust and play around with.

Note that the following images were captured using the Vehicle Game Sample/Template/Demo which is made by Epic Games. I’ve only created the post process material and applied it to their content/game.

So before I’ll give you the material setup, you can see the post process effect in action if you like.

Or just enjoy these postcards, err… I mean screenshots.

So here is the actual material I created:

As you can see it does not need many nodes. This is because most stuff is hidden in the Custom Node.
This node uses three inputs which you’ll have to set up. “offset”, “UV” and “scale”. The Output Type is CMOT Float 1. The following is the code used for that node:


//kernel size
int sizeX = 5;
int sizeY = 5;

//offset added to all uvs to get from the center to the corner
MaterialFloat2 baseOffset = MaterialFloat2(-sizeX/2.0f, -sizeY/2.0f) * offset.rg;
//sample the world normal from the GBuffer
float4 baseNormal = SceneTextureLookup(UV, 8);
//dot product for two normalized vectors ranges between -1.0f to 1.0f
//we're going to search the smallest value so initialize to something larger
float minDot = 10.0f;

for (int i = 0; i < sizeX; ++i)
{
	for (int j = 0; j < sizeY; ++j)
	{
		//calucalte clamped uvs
		float2 uvs = min(max(UV + baseOffset * scale + i * offset.r * scale.r * float2(1.0, 0.0f) + j * offset.g * scale.g * float2(0.0, 1.0f), float2(0.0f, 0.0f)), float2(1.0f, 1.0f));
		//sample world normal from the GBuffer at the new position
		float newDot = dot(baseNormal, SceneTextureLookup(uvs, 8));
		//if smaller than our current value
		if (newDot < minDot)
		{
			//use the new value
			minDot = newDot;
		}
	}
}

//return smallest dot product in this region
return minDot;

There are a few more things you’ll need to set up before you’re ready to go. The first is to set the Blenable Location for the main material node to “Before Tonemapping”. The second step is to create a Material Instance from this Material (you can right-click it and choose “Create Material Instance” in the Content Browser) which allows you to control the edge thickness as well as how fast an edge is detected. Finally you just have to make use of the Material Instance for example by adding it to the Blendables of a PostProcessVolume.

I did not spend alot of time improving it, so it has a few issues. If used with TXAA some edges flicker. On the other hand some edges look worse without TXAA.

It’s been over a year since I initially created this with 4.2 so some things may be different and require some adjustments in other versions.

What exactly is that first element, the TexCoord? Sorry, I am new to all of this.

All it does is call your UVs as a float2. You can manipulate your UVs using that node.

Not working in 4.12: “Error [SM5] Material.usf(1257,21): error X3013: ‘SceneTextureLookup’: function does not take 2 parameters”.

you need to replace the “SceneTextureLookup(UV, 8);” in the custom node with “SceneTextureLookup(UV, 8,true);” (or use a false if you don’t want filtering (got this from the HLSL output of another shader))
It would be better to provide the scene lookups as input parameters, but the second one may not be easy.
Does anybody know that the node version of the lookup is?

I made a slight improvement, I added the unroll specifier to the loops:


//kernel size
int sizeX = 5;
int sizeY = 5;

//offset added to all uvs to get from the center to the corner
MaterialFloat2 baseOffset = MaterialFloat2(-sizeX/2.0f, -sizeY/2.0f) * offset.rg;
//sample the world normal from the GBuffer
float4 baseNormal = SceneTextureLookup(UV, 8,true);
//dot product for two normalized vectors ranges between -1.0f to 1.0f
//we're going to search the smallest value so initialize to something larger
float minDot = 10.0f;

[unroll]
for (int i = 0; i < sizeX; ++i)
{
	[unroll]
	for (int j = 0; j < sizeY; ++j)
	{
		//calucalte clamped uvs
		float2 uvs = min(max(UV + baseOffset * scale + i * offset.r * scale.r * float2(1.0, 0.0f) + j * offset.g * scale.g * float2(0.0, 1.0f), float2(0.0f, 0.0f)), float2(1.0f, 1.0f));
		//sample world normal from the GBuffer at the new position
		float newDot = dot(baseNormal, SceneTextureLookup(uvs, 8,true));
		//if smaller than our current value
		if (newDot < minDot)
		{
			//use the new value
			minDot = newDot;
		}
	}
}

//return smallest dot product in this region
return minDot;

Hey there, this is great! Easy-to-follow tutorial and I got it working in a few mins :smiley:
I’m wondering how I would use this method with scene colour as well and not just world normal, but I don’t have a single clue how HLSL code works…
and scene depth, that would be great too. I’m assuming its something embedded in that custom node, but again no clue.

Hey everyone, sorry it took me so long to reply.

Thank you for fixing the error and the [unroll] optimization.

The node version should just be SceneTexture:WorldNormal like the one used in the screenshot of the material above (the color pin).

You may be able to provide the scene lookups as input parameters yes, and yes it would probably also be better to do it that way.
Since the intention of the function was to only be used with the world normal, I hardcoded the scene texture index that represents the world normal: 8. You could probably make another input parameter that replaces this and specifies which scene texture index you want to use. The problem with that is you won’t know what that value means and where it comes from unless you dig it up or have it commented/documented somewhere. But using the dot product on those values probably has a different meaning than what was intended.

Thanks for the kind words!
I currently do not have the time to write a follow-up about how to do edge detection on scene color or scene depth. But this https://udn.epicgames.com/Three/DevelopmentKitGemsSobelEdgeDetection.html may get you started with a scene depth edge detection material. Do note that the link is on the UDN and for the UDK but most of the nodes should be very similar (does not require any HLSL but you could condense it a lot with it). You could also just go ahead and replace the “8” in the HLSL code with some other number. But be warned, you may get really strange results since it really is only meant to be used with a value of “8” :slight_smile:

Hey

Great tutorial on normal edge detection! Plus the postcards look fantastic.

I’m currently trying to implement it myself and my edges aren’t being rendered smoothly at all like in your video. I have attached my post process blueprint below. I am using PostProcessInput0 as I get the following error with SceneTexture:
Error [SM5] (Node SceneTexture) SceneColor lookups are only available when MaterialDomain = Surface. PostProcessMaterials should use the SceneTexture PostProcessInput0.

I know this post is dead but any and all help is appreciated!

de24a51a0aadbccc38ae03fb062acf5ee5aa7620.jpeg

all you need to do for that is change the scene texture option in the menu to PostProcessInput0 instead of SceneColor

How exactly do I implement the code for the custom node? I keep getting errors

Hello,

I copied the code but the custom node isn’t working in 4.20.

在4.25.1上搞成了。ue4真坑
I make it work in 4.25.1. Hope can help somebody


//kernel size
int sizeX = 5;
int sizeY = 5;
MaterialFloat2 UV=GetDefaultSceneTextureUV(Parameters, 8);
//offset added to all uvs to get from the center to the corner
MaterialFloat2 baseOffset = MaterialFloat2(-sizeX/2.0f, -sizeY/2.0f) * offset.rg;
//sample the world normal from the GBuffer
float4 baseNormal = SceneTextureLookup(UV, 8,false);

float minDot = 10.0f;

for (int i = 0; i < sizeX; ++i)
{
for (int j = 0; j < sizeY; ++j)
{
//calucalte clamped uvs
float2 uvs = min(max(UV + baseOffset * scale + i * offset.r * scale.r * float2(1.0, 0.0f) + j * offset.g * scale.g * float2(0.0, 1.0f), float2(0.0f, 0.0f)), float2(1.0f, 1.0f));
//sample world normal from the GBuffer at the new position
float newDot = dot(baseNormal, SceneTextureLookup(uvs, 8,false));
//if smaller than our current value
if (newDot < minDot)
{
//use the new value
minDot = newDot;
}
}
}


return minDot;

Thanks for everyone!!
It worked easy here in UE4.25.4.

What I did:

  • Followed the image above;
  • Copied the code above and pasted it inside the Custom node.

Here is a short video selecting all nodes to see the details: (83) UE4.25.4 Edge Detection Post Process Effect using World Normals - YouTube](https://www.youtube.com/watch?v=SzqCrlJqpYg&feature=youtu.be)

1 Like

Thanks for the reference! By the way, I’m curious why you chose to use the “Preintegrated Skin” shading model?


Hi, thanks for sharing this, your results look very nice.
I’m getting a large offset between the outlines and my scene geometry.
I tried both versions of your code.
Using 5.4, could something have been changed in the newest UE5 to cause this?

What filter was used? I’d like to know what algorithm-based shader formula it applys.