TAA - Tweaking motion quality

Development Rendering
1

In the TAA implementation there is some code that ignores the history color at certain times. Like when the old position was off screen or behind an object.

Here you can see it visualized when I rotate quite fast: Imgsli & Imgsli
Notice where the mask is applied only the current frame is used which causes aliased pixels.

By blurring the masked area slightly I get a result that is more in line with the rest of the screen:
- Imgsli
- Imgsli

How to use:

  1. Open …\UE_4.20\Engine\Shaders\Private\PostProcessTemporalAA.usf
  2. Goto line 2212 (or around there) and look for OutputPayload.Color = TransformBackToRawLinearSceneColor(OutputPayload.Color);
  3. Add the following code **above **that line
  4. If you are running the editor you can use CTRL + SHIFT + . to compile the shaders

    // BLEND FILTERED COLOR WITH NEIGHBORS WHEN HISTORY IS IGNORED
    // -----------------------------------------------------------
    if (IgnoreHistory)
    {    
        half3 Neighbor1 = TransformCurrentFrameSceneColor(PostprocessInput0.SampleLevel(PostprocessInput0Sampler, InputParams.NearestBufferUV, 0, int2( 0, -1))).rgb;
        half3 Neighbor3 = TransformCurrentFrameSceneColor(PostprocessInput0.SampleLevel(PostprocessInput0Sampler, InputParams.NearestBufferUV, 0, int2(-1,  0))).rgb;
        half3 Neighbor5 = TransformCurrentFrameSceneColor(PostprocessInput0.SampleLevel(PostprocessInput0Sampler, InputParams.NearestBufferUV, 0, int2( 1,  0))).rgb;
        half3 Neighbor7 = TransformCurrentFrameSceneColor(PostprocessInput0.SampleLevel(PostprocessInput0Sampler, InputParams.NearestBufferUV, 0, int2( 0,  1))).rgb;
        float NeighborWeight = lerp(10.0, 4.0, saturate(Velocity / 40));
        OutputPayload.Color.rgb = ((OutputPayload.Color.rgb * NeighborWeight) + Neighbor1 + Neighbor3 + Neighbor5 + Neighbor7) / (NeighborWeight + 4.0);
        //OutputPayload.Color.rgb = float3(2,0,0);        
    }

Pull request: https://github.com/EpicGames/UnrealEngine/pull/4933

More examples
Camera standing still: Imgsli
Camera also rotating: Imgsli & Imgsli

2

Here is another test with a worst case (impossible) noise material. Imgsli

I rotate the character 360 every 2 seconds.
Also remember to use the zoom button to see the details. :slight_smile:

3

Love your work man :smiley:

4

looks good and I really appreciate the effort
however I wonder why not go the opposite: sharpen everything else instead of blurring the affected area. your previous efforts are about sharpening which were great in a static camera but didn’t hook into the temporal reprojection stuff, and for me the biggest issue at this point with TAA is how small camera/object movements make things blurry compared to a static camera/object :slight_smile:

5

We were using MSAA to run away from the blurriness of TAA because our game uses small details and objects on screen. But when we couldn’t use some of the more advanced skin/hair and other shaders in MSAA we have now switched to experimenting with TAA. Still trying to get an acceptable sharp consistent look… It is frustrating sometimes not to be able to have the best of both worlds in this day and age, especially something related to AA.

If you had told me that AA is going to be the single biggest issue to worry about from a visual standpoint a few years back, I would not have believed you.

I’ve seen your other work, but I never got around testing it yet, I hope it would work well with fast animated objects without causing artifacts, any hints will be appreciated.

1 Like
6

It is a noisy (aliased) area with flickering pixels. Have to be smoothed. TAA works best with motion blur applied, but if you turn it off, you will have to take care the artifacts TAA left behind.

7

The effect this fixes only happens when you move the camera rather fast. So the blur is not an issue.
Without this you can see something that looks like ghosting, but is in reality the area after the character having no TAA applied.

It’s a small fix for a small part of the TAA. :slight_smile:

PS: The code isn’t quite done. I have a slightly better version at home.

8

for me motion blur usually makes TAA worse but I haven’t tested this recently (and I know TAA improved a bit in 4.19).
still the point is that the aliased pixels are already sharp on their own right, and despite them being noisy and aliased for me it’s not so bad because it’s only for pixels that are in fast motion so its harder to spot. for me the issue is again the blurry pixels that are left behind by TAA on not-so-fast motions

9

I noticed that the aliased anti-ghosting pixels ghost into the next frame.

Here is an example with a noisy background (extreme case): Imgsli
And here is the scene: Imgsli

Remember to use the zoom button to look at the details! Like the Green pistol label, grass to the left of the character etc.

10

And here is a test combining Blur and the sharpening post process material I made.

Notice the tree trunks and the rocks: https://imgsli.com/MTg0Ng

11

Hi Hallotore, I’m not sure I understand from what I can tell visually the blur+Sharpness looks blurrier than the default one, this is prominent on the text? How is it better?

12

It’s looks blurrier because some of the scene is aliased in the default one. Zoom in and look at the branch to the left of the characters head.
The overall image is a bit blurry because the camera is moving really fast.

13

I think I can see it now,

The above code you applied is fixing the aliased for taa during motion what did you apply for the sharpness? Do you have a recommendation or step by step guide that may come with a scene perhaps. Any other hints as how to make the image even sharper? Maybe by applying a custom subtle post effect?

Thanks for your effort.

14

Here is the sharpness material I’m talking about. https://forums.unrealengine.com/development-discussion/rendering/1449167-sharper-image-without-the-edge-artifacts

15

I’ve tweaked the code a bit to change based on velocity. This helps when IgnoreHistory is used on movable objects while the camera is standing still.

Here are some awesome before/after! :smiley:

Camera standing still: Imgsli
Camera also rotating: Imgsli & Imgsli

PS: Here is a video of my test setup: https://www.youtube.com/watch?v=RpSl2lryVvo

16

Thanks will give these a try.

17

Hi @hallatore , I hope you are well and can answer my question, integrate the code but when I start the ue4 I get a lot of yellow stripes and other colors with the blur ghost, I wonder if you can do a tutorial with images or in a video to apply that code, it’s important to me, thanks

18

I would like to make use of this tweak, as the ghosting is very visible in my environment, and TAA is vital for my foliage to look its best.

But it doesn’t appear like “PostProcessInput0” is a valid variable/reference anymore and it fails to compile, what would I replace it with?

19

With InputSceneColor and InputSceneColorSampler respectively.

20

Thanks! Now that I’ve gotten to test this in practice, it’s exactly what I hoped, the ghosting of my character/other objects moving quickly is practically gone and no longer visible!