Temporal AA sharpening

[hallatore]

@Loginer

I didn’t take any screenshots with sharpening on, but in my opinion, the difference between blur reduction on/off is somehow more pronounced when combined with a bit of sharpening.

Sharpening definitely has a more dramatic impact than blur reduction. You can instantly tell if it’s on or not, while blur reduction is something that’s harder to notice without flipping it on/off in realtime. But they both measurably improve VR image quality.

Are these without the dynamic sharpen. Or without the tonemap sharpen?
And, what settings are you finding working best for you?

Will do some testing with my Rift later today.

[Loginer]

Sharpen was off on the screenshots; I made sure to set DynamicSharpen to 0. Personally, I think sharpening looks good between 0.5 and 0.7(!), with the Konflict blur reduction at 1.0. It gives the image a grungier look, but it also makes it look very crisp without supersampling, which is a big deal in VR.

Oculus and Valve both recommend not using Temporal AA, but our game has too much shader aliasing for MSAA to be effective. I really appreciate the work you guys are doing on improving TAA.

[hallatore]

@Loginer

Sharpen was off on the screenshots; I made sure to set DynamicSharpen to 0. Personally, I think sharpening looks good between 0.5 and 0.7(!), with the Konflict blur reduction at 1.0. It gives the image a grungier look, but it also makes it look very crisp without supersampling, which is a big deal in VR.

Oculus and Valve both recommend not using Temporal AA, but our game has too much shader aliasing for MSAA to be effective. I really appreciate the work you guys are doing on improving TAA.

DynamicSharpen is 0.25 by default since that is closest to the same texture sharpness as without TAA. I made it a console variable so you get artistic control.
0.5 to 0.7 is strong, but because of how it works it handles extreme values much better than tonemap.sharpen.

You can also try using a tiny bit of tonemap.sharpen in combination with the others if you want a slightly sharper result. For example dynamicSharpen set to 0.25 and tonemap.Sharpen set to 0.2. They do slightly different things, so combining them is a valid option.

[hallatore]

Here is a quick explanation on which order stuff is run in.

• BasePass

• Temporal AA

  • BlendFinal tweak

• Dynamic Sharpen

• Other post process stuff… (motion blur, dof, etc)

• Tonemap (Sharpen)

When the camera stands perfectly still the TAA creates a nice but slightly blurry result.
When the camera moves TAA produces much more blur. The BlendFinal code counters this by adjusting it’s strength based on how fast you move. It basically blends less of the previous frames the faster you move.

DynamicSharpen fixes the result from TAA before it gets passed on. But it’s not meant to fix the blur caused by movement, so that has to be fixed within the TAA itself. (Which is what BlendFinal tries to do.)

You can still sharpen the overall picture with TonemapSharpen. But it’s now more of an artistic choice, and less a method of fixing TAA blur.
TonemapSharpen is also a flawed method of fixing TAA since it can’t sharpen blurry textures without also sharpening edges that was just smoothed by TAA.
This results in TonemapSharpen undoing some of the work TAA just did, and the end result is a unnatural sharp image as you can see here: Imgsli

The reason you might want to use TonemapSharpen is that DynamicSharpen will never sharpen edges. So if you want a slightly overall sharper picture a combination is probably ideal.

[hallatore]

It’s interesting to see that 200 screen percentage (Imgsli) is almost the same as dynamic sharpening (Imgsli).
There isn’t that much difference even though one of these has to render 4 times as many pixels. Imgsli

[hallatore]

Noticed it didn’t work on Shader Model 4 since the post process was only loading on SM5. Updated the code to work on SM4 now.
Just changed one line: https://github.com/hallatore/UnrealEngine/commit/a267de955ed6274159ade57a6007f43d99c08be1

[mittense]

Are the TAA_Sharpen and TAA_MovementBlurReduction branches mutually exclusive, or just have yet to be combined?

[hallatore]

@mittense

Are the TAA_Sharpen and TAA_MovementBlurReduction branches mutually exclusive, or just have yet to be combined?

There is a different movement blur reduction in TAA_Sharpen.
TAA_MovementBlurReduction is just Konflict’s original code with a console variable.

[anonymous_user_bfb9c84a1 (anonymous_user_bfb9c84a)]

wow good job

[hallatore]

The master branch just got updated with some movement blur reduction code, so I updated the main branch and pull request.
4.15 and 4.17 branch still contain the movement blur reduction since it was just added to the engine 20 hours ago.

[hallatore]

Just a comparison of the default materials in the starter map.

Default vs. Dynamic: Imgsli
150% vs. Dynamic: Imgsli

[hallatore]

Updated the formula a bit and it should now be even better!

• Default vs: Imgsli (Mask: Imgsli)
• AA Off vs: Imgsli
• 200% vs: Imgsli
• Imgsli (mask: Imgsli) - With heavy foliage scenes it only touches textures like the bark on the trees.
• Imgsli (mask: Imgsli)
• Imgsli (mask: Imgsli)
• Imgsli (mask: Imgsli)

[Chosker]

is there a way to tweak the mask? I tried it at work and it made no difference until I zoomed in. the game is a topdown game where everything is very far from the camera so I believe the mask is suffering from the typical edge-detection “loss of strength” over high distances

[Konflict]

@Chosker

everything is very far from the camera

The mask is intended to protect the content from loosing the antialiased edges, and as such by reducing the mask it would result in running jaggies and unwelcomed artifacts. You can try to tweak it anyways. If you have very small things on the screen you maybe should consider an SMAA prepass and only then reduce the temporal antialiasing to avoid such incidents.

[hallatore]

@Chosker

is there a way to tweak the mask? I tried it at work and it made no difference until I zoomed in. the game is a topdown game where everything is very far from the camera so I believe the mask is suffering from the typical edge-detection “loss of strength” over high distances

Do you have a screenshot of your scene?

You can try tweaking r.TemporalAADynamicSharpen. But it also might be good to combine it with r.Tonemap.Sharpen since that sharpens the whole image.

[hallatore]

Edit

Got some great feedback on the pull request and their reasons for closing it.
[HR][/HR]Looks like Epic don’t want the pull request because they say you can already use r.tonemap.sharpen to do the same thing… (https://imgsli.com/i/3bbf9372-8ce8-4…6ca7f44488.jpg)
Not exactly encouraging when it feels like they spent 5 minutes not understanding what it does…

https://github.com/EpicGames/UnrealEngine/pull/3924

[HiIAmMoot]

Maybe make a high resolution comparison video to convince them and the rest of the community?

[hallatore]

Was just the first comment that was a bit harsh. Got some great feedback later and some issues that make it might not work well in every scenario. On of them was flickering caused by the scene depth jittering between frames. It was only slightly noticable, but still there. I have greatly reduced this locally and it improved it a lot.

Here is a breakdown of how it currently works. Some might find it interesting.

First I get the color contrast.

https://imgsli.com/i/49e8389a-6a98-4fd9-b248-67997131faae.jpg

Then I get the depth difference.

https://imgsli.com/i/59793933-f5ab-482a-b7d7-044f041347fd.jpg

Combined I get this mask.

https://imgsli.com/i/6f9ff546-d271-4356-9a08-44924be0d916.jpg

And then I sharpen the image based on this mask. (High value so it’s easier to see)

https://imgsli.com/i/8a788b6e-d6fe-4df0-b2c8-8132f152a74d.jpg

And finally here is a before/after of the scene: Imgsli

[Chosker]

@hallatore

Do you have a screenshot of your scene?

You can try tweaking r.TemporalAADynamicSharpen. But it also might be good to combine it with r.Tonemap.Sharpen since that sharpens the whole image.

can’t really provide a screenshot (don’t think I’m allowed to) but you can see some of the official alpha screenshots on the steam page
you can see there’s a huge distance between the camera and the objects, which results in your sharpening getting lost unless I zoom in quite a lot
tweaking temporalAADynamicSharpen did next to nothing in my case, and the tonemap sharpen just sharpens the whole image, but I’d be interested in not sharpening the edges (which is what you’re doing). I suspect that such great distances just don’t play well with the edge detection

the upside is that as my scenario is quite different from yours, if you can put together something similar and make your shader work across the board you’ll get closer to a more mainstream solution (and so higher chance to get the PR accepted)

[hallatore]

@Chosker

can’t really provide a screenshot (don’t think I’m allowed to) but you can see some of the official alpha screenshots on the steam page
you can see there’s a huge distance between the camera and the objects, which results in your sharpening getting lost unless I zoom in quite a lot
tweaking temporalAADynamicSharpen did next to nothing in my case, and the tonemap sharpen just sharpens the whole image, but I’d be interested in not sharpening the edges (which is what you’re doing). I suspect that such great distances just don’t play well with the edge detection

the upside is that as my scenario is quite different from yours, if you can put together something similar and make your shader work across the board you’ll get closer to a more mainstream solution (and so higher chance to get the PR accepted)

Since you have so much tiny objects I think the depth mask will be pretty much black. You can try doing only the LuminanceContrast and remove the DepthFactor. It might be better than tonemapper.sharpen.