Trouble with an Oil Painted Post Effect

Hello folks. I’m trying to implement an algorithm I found for creating an oil panted effect over a given image:

but I can’t seem to get the lil ■■■■■■ to work. I’m using a custom node in a post process material to handle the nested FOR loops needed to compare the pixels surrounding the pixel being rendered, and everything seems to compile fine, but when I apply the effect in my scene, all I get is some weird noise.

So, obviously something is failing. I’m assuming it’s the SceneTextureLookup call I am making within the custom node, but I can’t find any decent docs or examples of how to use this function to be sure if that’s the cause or not. If anyone is able/willing, could you please take a look at my material & code and point out what idiot thing I am doing, I would be very grateful.

Here’s the material:

And here’s the code in the OilPaint node:

float4 outputBuffer = float4(1,0,1,1);
float4 curPixel = float4(1,0,1,1);
float nSumR[50];
float nSumG[50];
float nSumB[50];
float intensityCount[50];

for(int bob = 0;bob < 50;bob++)
	nSumR[bob] = 0.0;
	nSumG[bob] = 0.0;
	nSumB[bob] = 0.0;
	intensityCount[bob] = 0.0;

for( float yRadInc = (curY - effectRadius); yRadInc <= (curY + effectRadius); yRadInc++ )
	if(!( (yRadInc < 0.0) || (yRadInc > height)))
		for( float xRadInc = (curX - effectRadius); xRadInc < (curX + effectRadius); xRadInc++ )
			if(!( (xRadInc < 0.0) || (xRadInc > width)))
				curPixel = SceneTextureLookup(float2(xRadInc,yRadInc), 14, false);
				float nR = curPixel.r;
				float nG = curPixel.g;
				float nB = curPixel.b;
				// Find intensity of RGB value and apply intensity level.
				float curIntensity =  ( ( ( nR + nG + nB ) / 3.0 ) * effectIntensity );
				if( curIntensity > 1.0 )
					curIntensity = 1.0;
					int i = (int)curIntensity;
				nSumR[i] = nSumR[i] + nR;
				nSumG[i] = nSumG[i] + nG;
				nSumB[i] = nSumB[i] + nB;

float nCurMax = 1.0;
int nMaxIndex = 0;
for( int nI = 0; nI < (effectRadius * effectRadius); nI++ )
	if( intensityCount[nI] > nCurMax )
		nCurMax = intensityCount[nI];
		nMaxIndex = nI;

outputBuffer = float4((nSumR[nMaxIndex] / nCurMax),(nSumG[nMaxIndex] / nCurMax),(nSumB[nMaxIndex] / nCurMax),1.0);

return outputBuffer;

As you can see, I’ve done my best to duplicate the algorithm from the link at the top, but it’s just not working. Hopefully it’s some simple thing I’m over looking and not some intrinsic limitation to the post process material that is preventing this from working as I’d think it would look pretty cool…

Thanks in advance for any help.



To update this within info gleaned from posting in the forums, I have gotten the effect to work as a basic material after fixing some issues with the code, such as not using normalized UV coors, etc. but have hit a roadblock in that the SceneTextureLookup function refuses to run within nested loops. The compiler gives the error of:

Error [SM5] Material.usf(1283,14): error X4014: cannot have gradient operations inside loops with divergent flow control

If anyone knows of a way to work around this, I’d love to hear about it. It feels like I’m really close, but this seems to be an internal engine thing. Be a real bummer if I couldn’t get this working as a PPE…

Here’s the updated (working) graph and code for the material version:

[link text][2]

And 2000 words worth of proof:



Error [SM5] Material.usf(1283,14): error X4014: cannot have gradient operations inside loops with divergent flow control

Hey this is a few months late, I’m just posting this for the next guy. If you get the above error in a For or If statement, change the way you’re grabbing texture pixels. For example:

[loop]  // This forces it to loop the For loop. To unravel it, you could use [unroll(8)]
for (int i = StartIndex; i < EndIndex; i++)
    float V = floor(i / Width);
    float U = (i - (V * Width)) / Width;
    V = V / Width;
    float2 UV = { U,V };

    //BAD; This samples multiple MIP levels, which isn't allowed different length For loops
    //sum = sum + Texture2DSample(TEX, Material.Texture2D_0Sampler, UV);

    //GOOD; This only samples one MIP level, and thus loops fine
    sum = sum + Texture2DSampleLevel(TEX, Material.Texture2D_0Sampler, UV,MaterialFloat(0.00000000));

I got the texture sample code by going to >Material Editor top pane menus >Window >HLSL Code >Copy. And I got it to sample 1 MIP level by setting a Texture Sample Parameter node to MIP level 0, then finding it near the bottom of the output code.

I don’t fully understand why you can’t MIP texture samplers inside flow control (like IF or FOR statements), but apparently it’s a fundamental feature of HLSL. You can find more info here hlsl - DirectX texture sampling function SampleCmpLevelZero returns blank intermittently? - Stack Overflow

Apologies if this doesn’t apply to your post, but you’re the one and only hit on google for >“ue4” HLSL for “cannot have gradient operations”

I would love to be able to verify but I haven’t been able to get Unreal Editor to run under OS X for months now. Kinda given up on the whole thing.