Hello folks. I’m trying to implement an algorithm I found for creating an oil panted effect over a given image:
but I can’t seem to get it to work. I’m using a custom node in a post process material to handle the nested FOR loops needed to compare the pixels surrounding the pixel being rendered, and everything seems to compile fine, but when I apply the effect in my scene, all I get is some weird noise.
So, obviously something is failing. I’m assuming it’s the SceneTextureLookup call I am making within the custom node, but I can’t find any decent docs or examples of how to use this function to be sure if that’s the cause or not. If anyone is able/willing to take a look at my material & code and point out what idiot thing I am doing, I would be very grateful.
As you can see, I’ve done my best to duplicate the algorithm from the link at the top, but it’s just not working. Hopefully it’s some simple thing I’m over looking and not some intrinsic limitation to the post process material that is preventing this from working as I’d think it would look pretty cool…
Thanks in advance for any help. And yes, this is a repeat of a post on the Answerhub, but most people don’t check there too often, so I’m hedging my bets. Cheers,
have you looked at the errors you may be getting? try making a material instance of the post processing effect and open it up, sometimes the errors only show up there.
I imagine it might have something to do with the effect outputing a 4-vector, try breaking that down to 3?
Not directly but you can early out your function at any time and return a debug value and then display it on the mesh by using “Debug Scalar Values” or Float2-3 whatever. I end up doing that all the time.
Well, this is a post process material, which makes it a bit harder to do that. Does it make sense to build the material as a normal mesh material first and then swap in the sceneTexture to replace the regular 2D texture? Alas, there’s not a ton of really good reference materials/examples for best practices when developing post processes in UE4… Cheers,
OK, I’ve updated my code with using normalized pixels in mind, and while the code compiles when I hit Apply without any errors, when I hit save, the editor locks up. Is saving doing something more then applying does? Don’t they both compile the shader?
Here’s my updated code, for reference:
Watching the process viewer as I hit apply shows a few ShaderCompiler threads being launched that seem to crunch for a very long time but eventually finishes and the editor becomes responsive again. Not sure why changing the pixel values to normalized is causing this behavior. Very strange. Once the compilers are done, I get the output:
Error [SM5] Common.usf(236,2): warning X3570: gradient instruction used in a loop with varying iteration, forcing loop to unroll
Error [SM5] Material.usf(1277,8): error X3511: unable to unroll loop, loop does not appear to terminate in a timely manner (202 iterations), use the [unroll(n)] attribute to force an exact higher number
Error [SM5] Material.usf(1273,6): error X3511: forced to unroll loop, but unrolling failed.
So something in the for loops is invalid. I am being reminded why coding was such a pain in the ***…
Hmm. Trying it as a material gives me the same error. There must be something wrong with my math that’s causing the for loop to go to infinity. Will have to bang on it some more. Thanks for all the help Ryan.
Hmm… Re-reading this thread, I saw Ryan’s suggestion using the Tex object and its Tex.SampleLevel() function. I swapped out the Texture2DSample function with that and now I am able to compile! Yay. So, my next question is, why does Texture2DSample use a gradient function but SampleLevel not? And is there an equivalent to the SampleLevel function that I can call for the GBuffer?
So, I have it working now as a mesh material. Using a screen cap of a regular scene, it seems to be close to what I was aiming for. Now I just need to figure out two things.
It seems to be sampling outside the 0.0-1.0 UV range, which I had thought the if statements that check to make sure that both U and V are within the proper range before sampling would prevent. So, how do I deal with the edges? Decrease the “safe” range in my if statements to something like 0.1 through 0.9?
How do I convert this to a post effect if I can’t use the regular SceneTextureLookup function since that has some kind of gradient function within it which HLSL seems to strongly disapprove of? Is there any other way to access the gbuffer?
Today’s update. Been playing around with different outputs from the SceneTexture node, namely the BaseColor or Specular (As stored in GBuffer), and the shader compiles without any gradient complaints but as soon as I try to use it in a scene, Editor crashes. Hard. I thought perhaps I was using the wrong index for that pass, so I tried a lower index (i.e. 24 (custom stencil) instead of 25 (basecolor) or 26 (Specular)) and that doesn’t crash anything, but all I get is a black screen. Which makes sense as the custom stencil buffer pass is unused and hence would be nothing but 0… So, something about trying to access anything stored in the Gbuffer (indices 25 & 26) as well as other indices such as 1,2,3,4,etc. just straight up crashes, trying to use the PostProcessInput passes (indices 14-20) give the Gradient compile error (only when saving, not when applying which is odd), but using index 0, the initial basecolor pass (which in the tool tips says NOT to use for a post process) seems to work. Kinda.
I can see why you wouldn’t want to use that pass since you can see there are lots of artifacts in the final image which I assume stem from sampling before other passes have had a chance to do their job. For example:
Ideally I should be able to apply this effect to the final image in the buffer as one of the last steps before displaying it, preferably before the AA pass so it can smooth out the effect. I would assume that would be the base color as stored in GBuffer, yet that goes kaboom when I try to use it. In fact, most of the passes I can sample from SceneTexture seem to compile without complaint when I hit apply, but either throw the Gradient compile error when saving or crash the editor when the post effect is made active. BaseColor (index 0) is the only one that seems to work, but is non optimal for obvious reasons. Any idea why the initial base color pass would work when the basecolor pass stored in the Gbuffer would not?
P. S. Installed 4.12.4 hotfix and seeing the same behavior so it feels like I’ve hit a wall. Any advice or suggestions would be very much appreciated…
I know this is bit too late, but leave this for anyone who will come across this and want to create their own effect. I’ve managed to make Oil Painting effect work with UE 4.18. via shader, I believe that an issue with this code was use of if statements around for loops. As a matter of fact, this is currently not necessary (nor is use of separate parameters for width and height of the texture), as UV textures just wrap around (SceneTextureLookup function works with this premise).