Trouble with Oil Painting Post Process Effect

J.J.Franzen · June 24, 2016, 2:09am

Hello folks. I’m trying to implement an algorithm I found for creating an oil panted effect over a given image:

but I can’t seem to get it to work. I’m using a custom node in a post process material to handle the nested FOR loops needed to compare the pixels surrounding the pixel being rendered, and everything seems to compile fine, but when I apply the effect in my scene, all I get is some weird noise.

So, obviously something is failing. I’m assuming it’s the SceneTextureLookup call I am making within the custom node, but I can’t find any decent docs or examples of how to use this function to be sure if that’s the cause or not. If anyone is able/willing to take a look at my material & code and point out what idiot thing I am doing, I would be very grateful.

Here’s the material:

And here’s the code in the OilPaint node:

float4 outputBuffer = float4(1,0,1,1);
float4 curPixel = float4(1,0,1,1);
float nSumR[50];
float nSumG[50];
float nSumB[50];
float intensityCount[50];

for(int bob = 0;bob < 50;bob++)
{
nSumR[bob] = 0.0;
nSumG[bob] = 0.0;
nSumB[bob] = 0.0;
intensityCount[bob] = 0.0;
}

for( float yRadInc = (curY - effectRadius); yRadInc <= (curY + effectRadius); yRadInc++ )
{
if(!( (yRadInc < 0.0) || (yRadInc > height)))
{
for( float xRadInc = (curX - effectRadius); xRadInc < (curX + effectRadius); xRadInc++ )
{
if(!( (xRadInc < 0.0) || (xRadInc > width)))
{
curPixel = SceneTextureLookup(float2(xRadInc,yRadInc), 14, false);
float nR = curPixel.r;
float nG = curPixel.g;
float nB = curPixel.b;
			// Find intensity of RGB value and apply intensity level.
			float curIntensity =  ( ( ( nR + nG + nB ) / 3.0 ) * effectIntensity );
			if( curIntensity &gt; 1.0 )
				curIntensity = 1.0;
				int i = (int)curIntensity;
				intensityCount*++;
			
			nSumR* = nSumR* + nR;
			nSumG* = nSumG* + nG;
			nSumB* = nSumB* + nB;
		}
	}
}
}

float nCurMax = 1.0;
int nMaxIndex = 0;
for( int nI = 0; nI < (effectRadius * effectRadius); nI++ )
{
if( intensityCount[nI] > nCurMax )
{
nCurMax = intensityCount[nI];
nMaxIndex = nI;
}
}

outputBuffer = float4((nSumR[nMaxIndex] / nCurMax),(nSumG[nMaxIndex] / nCurMax),(nSumB[nMaxIndex] / nCurMax),1.0);

return outputBuffer;

As you can see, I’ve done my best to duplicate the algorithm from the link at the top, but it’s just not working. Hopefully it’s some simple thing I’m over looking and not some intrinsic limitation to the post process material that is preventing this from working as I’d think it would look pretty cool…

Thanks in advance for any help. And yes, this is a repeat of a post on the Answerhub, but most people don’t check there too often, so I’m hedging my bets. Cheers,

J^2

Construc · June 24, 2016, 5:01am

have you looked at the errors you may be getting? try making a material instance of the post processing effect and open it up, sometimes the errors only show up there.
I imagine it might have something to do with the effect outputing a 4-vector, try breaking that down to 3?

RyanB · June 24, 2016, 5:08pm

looks to me like you are incrementing your XY values by whole integers but you really need to increment them by pixelsize which is 1/screenresolution

In the link you sent they handled converting that into UV space inside of the lookup function itself but there are many ways you could write that.

Check out the ue4 custom node example from the docs, it has a nested loop like that in UV space.

J.J.Franzen · June 24, 2016, 6:57pm

Oh. Right. The UV that the function is expecting is in the 0.0 to 1.0 range. Since all UVs are in the 0.0-1.0 range. Duh. OK. That helps. Thank you!

BTW, is there a way to add debug print output to an HLSL shader to check a given var’s value?

J^2

RyanB · June 24, 2016, 7:09pm

Not directly but you can early out your function at any time and return a debug value and then display it on the mesh by using “Debug Scalar Values” or Float2-3 whatever. I end up doing that all the time.

J.J.Franzen · June 24, 2016, 9:06pm

Well, this is a post process material, which makes it a bit harder to do that. Does it make sense to build the material as a normal mesh material first and then swap in the sceneTexture to replace the regular 2D texture? Alas, there’s not a ton of really good reference materials/examples for best practices when developing post processes in UE4… Cheers,

J^2

J.J.Franzen · June 24, 2016, 9:50pm

OK, I’ve updated my code with using normalized pixels in mind, and while the code compiles when I hit Apply without any errors, when I hit save, the editor locks up. Is saving doing something more then applying does? Don’t they both compile the shader?

Here’s my updated code, for reference:

float4 outputBuffer = float4(1,0,1,1);
float4 curPixel = float4(0,1,1,1);
float nSumR[50];
float nSumG[50];
float nSumB[50];
float intensityCount[50];
float normalizedPixelSizeX = 1 / width;
float normalizedPixelSizeY = 1 / height;

for(int bob = 0;bob < 50;bob++)
{
nSumR[bob] = 0.0;
nSumG[bob] = 0.0;
nSumB[bob] = 0.0;
intensityCount[bob] = 0.0;
}

for( float yRadInc = (curY - (effectRadius * normalizedPixelSizeY)); yRadInc <= (curY + (effectRadius * normalizedPixelSizeY)); yRadInc += normalizedPixelSizeY )
{
if(!( (yRadInc < 0.0) || (yRadInc > 1.0)))
{
for( float xRadInc = (curX - (effectRadius * normalizedPixelSizeX)); xRadInc < (curX + (effectRadius * normalizedPixelSizeX)); xRadInc += normalizedPixelSizeX )
{
if(!( (xRadInc < 0.0) || (xRadInc > 1.0)))
{
curPixel = SceneTextureLookup(float2(xRadInc,yRadInc), 14, false);
float curR = curPixel.r;
float curG = curPixel.g;
float curB = curPixel.b;
			// Find intensity of RGB value and apply intensity level.
			float curIntensity =  ( ( ( curR + curG + curB ) / 3.0 ) * effectIntensity );
			if( curIntensity &gt; 1.0 )
				curIntensity = 1.0;
				int i = (int)curIntensity;
				intensityCount*++;
			
			nSumR* = nSumR* + curR;
			nSumG* = nSumG* + curG;
			nSumB* = nSumB* + curB;
		}
	}
}
}

float nCurMax = 0.0;
int nMaxIndex = 0;
for( int nI = 0; nI < (effectRadius * effectRadius); nI++ )
{
if( (intensityCount[nI] > nCurMax) && (nI < 50) )
{
nCurMax = intensityCount[nI];
nMaxIndex = nI;
}
}

outputBuffer = float4((nSumR[nMaxIndex] / nCurMax),(nSumG[nMaxIndex] / nCurMax),(nSumB[nMaxIndex] / nCurMax),1.0);

return outputBuffer;

Cheers,

J^2

Update:

Watching the process viewer as I hit apply shows a few ShaderCompiler threads being launched that seem to crunch for a very long time but eventually finishes and the editor becomes responsive again. Not sure why changing the pixel values to normalized is causing this behavior. Very strange. Once the compilers are done, I get the output:

Error [SM5] Common.usf(236,2): warning X3570: gradient instruction used in a loop with varying iteration, forcing loop to unroll
Error [SM5] Material.usf(1277,8): error X3511: unable to unroll loop, loop does not appear to terminate in a timely manner (202 iterations), use the [unroll(n)] attribute to force an exact higher number
Error [SM5] Material.usf(1273,6): error X3511: forced to unroll loop, but unrolling failed.

So something in the for loops is invalid. I am being reminded why coding was such a pain in the ***…

RyanB · June 25, 2016, 1:47am

hmm I am not sure why SceneTextureLookup would be using a gradient but that is why it would be easier to test using a regular texture.

Add input “Tex” to your function and then sample like:

Tex.SampleLevel(TexSampler, UV, 0); where 0 is mip level 0. You probably don’t need to worry about mips for this at least initially.

J.J.Franzen · June 25, 2016, 2:44am

Hmm. Trying it as a material gives me the same error. There must be something wrong with my math that’s causing the for loop to go to infinity. Will have to bang on it some more. Thanks for all the help Ryan.

J^2

RyanB · June 25, 2016, 2:57am

My first guess would be something with these lines:

for( float yRadInc = (curY - (effectRadius * normalizedPixelSizeY)); yRadInc <= (curY + (effectRadius * normalizedPixelSizeY));

I’d probably keep them as pure integers using a search radius and then convert back into UV space inside of the loop. Much easier to debug at least.

Also this shader might be slow due to the large number of temporary values, all the ones with [50].

J.J.Franzen · June 26, 2016, 7:12pm

Interesting. I’m used to having to initialize my vars before I use them. I know Python doesn’t have that req, is that true of HLSL as well?

J^2

DieByZer0 · June 26, 2016, 9:05pm

nSumR* = nSumR* + curR;
nSumG* = nSumG* + curG;
nSumB* = nSumB* + curB;

I would first advise you rewrite the above to something like:
nSumR* += curR;
nSumG* += curG;
nSumB* += curB;

That means exactly what u originally had but cleaner.

I see syntax errors above this looks like CG shader and not HLSL. Correct me if im wrong.

J.J.Franzen · June 27, 2016, 9:36pm

Heya Ryan. I reworked the for loops so they used integers, and I’m getting a different error now:

Error [SM5] Material.usf(1277,14): error X4014: cannot have gradient operations inside loops with divergent flow control

I’m not sure why but it appears the Texture2DSample function does some sort of gradient operation? I’ve verified by commenting out that one line and everything compiles fine.

What exactly does it mean by “divergent flow control”? Is that in reference to the if statement just before that function is called?

Does this mean what I am trying to do just is not possible using the custom node and I’ll have to write a full custom shader in c++? That would make me a sad, sad panda…

Here’s the current code and graph for reference:

float3 outputBuffer = float3(1,0,1);
float4 curPixel = float4(0,1,1,1);
float nSumR[50];
float nSumG[50];
float nSumB[50];
float intensityCount[50];
float normalizedPixelSizeX = 1 / width;
float normalizedPixelSizeY = 1 / height;
float curPixelU = inputX - (effectRadius * normalizedPixelSizeX);
float curPixelV = inputY - (effectRadius * normalizedPixelSizeY);

for(int bob = 0;bob < 50;bob++)
{
nSumR[bob] = 0.0;
nSumG[bob] = 0.0;
nSumB[bob] = 0.0;
intensityCount[bob] = 0.0;
}

for( int incY = 0; incY < (int)(effectRadius * 2); incY++)
{
curPixelV += normalizedPixelSizeY;
if((curPixelV >= 0.0) && (curPixelV <= 1.0))
{
for( int incX = 0; incX < (int)(effectRadius * 2); incX++)
{
curPixelU += normalizedPixelSizeX;
if((curPixelV >= 0.0) && (curPixelV <= 1.0))
{
curPixel = Texture2DSample(Scene, Material.Texture2D_0Sampler, float2(curPixelU,curPixelV));
float curR = curPixel.r;
float curG = curPixel.g;
float curB = curPixel.b;
			// Find intensity of RGB value and apply intensity level.
			float curIntensity =  ( ( ( curR + curG + curB ) / 3.0 ) * effectIntensity );

			if( curIntensity &gt; 1.0 )
			{
				curIntensity = 1.0;
			}
			
			int i = (int)curIntensity;
			intensityCount*++;
			nSumR* += curR;
			nSumG* += curG;
			nSumB* += curB;
		}
	}
}
}

float nCurMax = 0.0;
int nMaxIndex = 0;
for( int nI = 0; nI < (int)(effectRadius * effectRadius); nI++ )
{
if( (intensityCount[nI] > nCurMax) && (nI < 50) )
{
nCurMax = intensityCount[nI];
nMaxIndex = nI;
}
}

outputBuffer = float3((nSumR[nMaxIndex] / nCurMax),(nSumG[nMaxIndex] / nCurMax),(nSumB[nMaxIndex] / nCurMax));

return outputBuffer;

Thanks in advance for any advice you can give. Cheers,

J^2

J.J.Franzen · June 27, 2016, 10:24pm

Hmm… Re-reading this thread, I saw Ryan’s suggestion using the Tex object and its Tex.SampleLevel() function. I swapped out the Texture2DSample function with that and now I am able to compile! Yay. So, my next question is, why does Texture2DSample use a gradient function but SampleLevel not? And is there an equivalent to the SampleLevel function that I can call for the GBuffer?

Cheers,

J^2

J.J.Franzen · June 27, 2016, 10:47pm

So, I have it working now as a mesh material. Using a screen cap of a regular scene, it seems to be close to what I was aiming for. Now I just need to figure out two things.

It seems to be sampling outside the 0.0-1.0 UV range, which I had thought the if statements that check to make sure that both U and V are within the proper range before sampling would prevent. So, how do I deal with the edges? Decrease the “safe” range in my if statements to something like 0.1 through 0.9?
How do I convert this to a post effect if I can’t use the regular SceneTextureLookup function since that has some kind of gradient function within it which HLSL seems to strongly disapprove of? Is there any other way to access the gbuffer?

BTW, here’s what it looks like:

Pre:

Post:

Cheers,

J^2

Edit: Saw why the edges were bleeding over. Was checking against the V value for both loops. Doh! Fixed that and all good. Now I just need to figure out how to convert this to a PPE…

Edit of my edit: Just to confirm, tried out the code using a SceneTexture node and same error as with the Texture2D:

Error [SM5] Material.usf(1283,14): error X4014: cannot have gradient operations inside loops with divergent flow control

J.J.Franzen · June 28, 2016, 7:05pm

Today’s update. Been playing around with different outputs from the SceneTexture node, namely the BaseColor or Specular (As stored in GBuffer), and the shader compiles without any gradient complaints but as soon as I try to use it in a scene, Editor crashes. Hard. I thought perhaps I was using the wrong index for that pass, so I tried a lower index (i.e. 24 (custom stencil) instead of 25 (basecolor) or 26 (Specular)) and that doesn’t crash anything, but all I get is a black screen. Which makes sense as the custom stencil buffer pass is unused and hence would be nothing but 0… So, something about trying to access anything stored in the Gbuffer (indices 25 & 26) as well as other indices such as 1,2,3,4,etc. just straight up crashes, trying to use the PostProcessInput passes (indices 14-20) give the Gradient compile error (only when saving, not when applying which is odd), but using index 0, the initial basecolor pass (which in the tool tips says NOT to use for a post process) seems to work. Kinda.

I can see why you wouldn’t want to use that pass since you can see there are lots of artifacts in the final image which I assume stem from sampling before other passes have had a chance to do their job. For example:

Ideally I should be able to apply this effect to the final image in the buffer as one of the last steps before displaying it, preferably before the AA pass so it can smooth out the effect. I would assume that would be the base color as stored in GBuffer, yet that goes kaboom when I try to use it. In fact, most of the passes I can sample from SceneTexture seem to compile without complaint when I hit apply, but either throw the Gradient compile error when saving or crash the editor when the post effect is made active. BaseColor (index 0) is the only one that seems to work, but is non optimal for obvious reasons. Any idea why the initial base color pass would work when the basecolor pass stored in the Gbuffer would not?

J^2

P. S. Installed 4.12.4 hotfix and seeing the same behavior so it feels like I’ve hit a wall. Any advice or suggestions would be very much appreciated…

ex3me · April 19, 2018, 12:34pm

I know this is bit too late, but leave this for anyone who will come across this and want to create their own effect. I’ve managed to make Oil Painting effect work with UE 4.18. via shader, I believe that an issue with this code was use of if statements around for loops. As a matter of fact, this is currently not necessary (nor is use of separate parameters for width and height of the texture), as UV textures just wrap around (SceneTextureLookup function works with this premise).

I based my code off Gaussian Filter effect (Gaussian Blur Example - Community Content, Tools and Tutorials - Unreal Engine Forums), which works on similar principle. It too needs to gather information about the neighborhood of the pixel. Hope it’s gonna help someone.