Gen 5 Temporal Anti-Aliasing

General Feedback & Requests
1

tested the experimental aa algorithm a bit…

all my tests were done in vr.

I believe that it is selectively smoothing edges- putting the aa where it needs it most, leaving a sharper image where no aa is needed…
in theory this is great.

it is very expensive. at least for vr. cooking may help…

still images look great. things that move get artifacts- mostly specular… maybe dither that?.

zero ghosting!

to sum it up… i get better sharper smoother using standard taa and turning up the pixel density. i mean… its only ghosting…
fix the specular jaggies and the milliseconds… you would have something.

Great work epic!

2

Hi,

Thanks for taking time to trying it out and reporting feedback. :slight_smile:

Excellent, I have not tested VR yet, but I did test split screen so in theory you shouldn’t run into surprises here. Happy to hear you didn’t run into issues!

Not quite. There was community efforts notably by hallatore (https://forums.unrealengine.com/deve…-aa-sharpening and https://forums.unrealengine.com/deve…edge-artifacts among many other forum threads).

But here the quality improvement of Gen5 TAA is in the way the history rejection works, and notably this history rejection mechanism no longer loose detail that was the source cause of bluryness before.

I here you on the performance. :wink: It is still noted experimental work in 4.26 for quality and perf. Although it will receive some improvement in 4.26 branch even after the initial release for the benefit our next gen users that will be shipping on 4.26, but also benefiting the all users too.

It has been optimised on PS5 and XSX, and we should soon be able to have this optimizations enabled on D3D11, D3D12 and Vulkan. One is already available on PC with r.TemporalAA.R11G11B10History=1.

having said that I think I didn’t do a got job on the 4.26 preview release notes: It is more expensive, because it is at its core designed to be a temporal upscaler (TAAU with r.TemporalAA.Upsampling=1) more than it it is a TAA pass, to maintain quality when upscaling.

https://docs.unrealengine.com/Images/Engine/Rendering/ScreenPercentage/SpatialAndTemporalUpsample.webp

However as oposed to Gen4’s TAAU (r.TemporalAA.Algorithm=0 r.TemporalAA.Algorithm=1), it’s performance cost scale not only based on output resolution, but also it’s input resolution.

Therefore I want to amphasize that you be able to take fully advantage of it with:



r.TemporalAA.Algorithm=1 to switch to gen 5 TAA
r.TemporalAA.Upsampling=1 to have TAA do the upscaling and allow output TAAU resolution different from input resolution
r.ScreenPercentage below 100 to start lowering input resolution and therefore TAA cost, but also your overall frame's GPU time

Maintaining the quality with lower input resolution is what truly differentiate Gen5 TAA to Gen4’s, and we find that the savings by lowering the inputs resolution quickly outweigh the performance overhead it has while maintaining better output quality with the console optimization. Already we have cases where Gen5 TAAU, despite configured with a lower screen percentage, looks better than Gen4’s. But where the performance and quality benefit comes in, is when you can lower even more the screen percentage: not only to compensate for the higher upscaling cost, but also to free up some ms on the rest of the frame you can reinvest to increase quality or even turn-ON some other rendering features that ultimately also contribute to a higher output pixel quality.

Sorry my English is failing me to understand what you meant here. Would you mind explain a little bit more what you are witnessing?

You gonna have more information about temporal upsampling, and for sspecial case for VR on here: https://docs.unrealengine.com/en-US/…age/index.html

https://docs.unrealengine.com/Images/Engine/Rendering/ScreenPercentage/VRPixelDensity.webp

3 Likes
3

thank you Guillaume.Abadie for your work on this and the follow up…

It is as you say, the image quality is truly amazing with taau, gen5 taa, screen%<100. the more you cut the screen%, the more the taa enhances. screen% 30 is totally viable… and perhaps the only solution available atm to drop below the 50% mark.

there are trade offs though… stuff in motion gets a bit jagged. -somehow the ghosting disappears though.

a definite improvement in most cases.

i can see many uses for this…

i will continue to test this as higher frame rate = more eye candy!

perhaps if this plays well with dlss and dfr :wink:

is there any other recommended settings? frameweight, qualty, etc.

thanks again

4

Out of curiosity, will the new TAA upscaling behave well with depth of field? Currently when you set r.TemporalAA.Upsampling=1 , most of the DOF just disappears.

5

Well I’m glad it works on your content, because I’ve not be particularly happy with ScreenPercentage<50 on the content I’ve been testing with. There are some known shader permutation I need to implement when ScreenPercentage<50, but it hasn’t been much of focus, lately mostely because on shading change like blinking light the fundamental low input res can get really rough. So my focus has been more on making it work on next gen consoles, and improve the known quality problem and stability in the ScreenPercentage [50;100]

While I think I have an idea of aliasing artifact you are talking about under motion that is on my list, I’m having trouble to be exactly sure what you mean by Jagged. Could you illustrate with a screen shot?

Not under all conditions though, there is cases where it stills ghost. Watch out objects going out of frame. :rolleyes:

DLSS is temporal upscaler that takes advantage of Nvidia hardware, so it would be replacing TAA. The way we ship it in Fortnite is thanks to 4.26’s ITemporalUpscaler interface that can be overridden.

We never change that on any of the TAA samples or frame weight cvars on out project. We find the default settings are the most content universal settings. As the ghosting and stability improves on Gen5’s, I hope to reach a day no one would ever need to touch this settings at all. A tech that just works.

So when r.TemporalAA.Upsampling=1, it basically forces r.DOF.Recombine.Quality=0 that looses the slight DOF convolution, and that is due to DiaphragmDOF.cpp’s bSupportsSlightOutOfFocus. There needs to have some changes in the handling of the slight out of convolution (about 5pixels and below) when doing temporal upsampling that I didn’t have time to come down to. And we were only using temporal upsampling on current gen consoles. Wasn’t a big deal back then because if your frame would need to be temporally upsampled, that probably meant you didn’t have the performance to run DOF’s slight out of focus… However we exactly ran into this issue for our Lumen in the Land of Nanite demo running on PS5, but it is still prototype and I’m not sure whether I’m gonna have this finished by 4.26’s release. But yeah given how temporal upsampling is going to become important, it’s definitely something to fix very high on the priority list. :slight_smile:

6

Amazing work! Looking forward to see this improve.

I’ve been doing some testing in VR (Oculus Rift S). Ghosting is a lot less noticable compared to current TAA, especially on small particles like sparks. It’s a great improvement in that regard. The performance cost on is quite steep so I had to put the screenpercentage at around 50-60 for it to be impactfull. ( around 1ms frame reduction). However the visual difference becomes quite noticable under 75 screenpercentage, there is quite some jittering in screenspace effects that rely on TAA(ssr, ao). Also noticing some increased shimmering on deferred decals.

Looks we can’t rely on it yet for our project since the visual impact outweights the performance boost but hoping there will be some nice optimizations later on next year.

Everything below 50% seems to introduce some weird noise.

noise.PNG.jpg
noise.PNG.jpg732×444 118 KB

7

Hi Nepfish,

Sorry for the delay, I had to take a break for personnal reasons. :stuck_out_tongue:

I wonder if that could be due to r.UsePreExposure. Your project requires this to be enabled.

I had left this ensure to try warn users if they had not this enabled, but maybe this isn’t discoverable enough?

I don’t think I have seen this artifact before, would you mind sharing a simple repro project?

8

Heads up that quality issues due to the use of DitherTemporalAA is known, I have not tested that node yet and the noise pattern is certainely causing interferences pattern within Gen5 TAA’s history rejection mechanism

9

I’ve played around with it a bit and it looks fantastic, especially good at preserving detail on thin lines and I’ve yet to see much ghosting. It might make one of our concept vine characters be viable for production now, as TXAA gen 4 made him look too soft.
As a technical artist, I do hope that DitherTemporalAA will be supported and will be considered a top priority, as most of our opacity and dissolve tricks use DitherTemporalAA, and it’s an invaluable tool in a tech artist’s tool bag. It’d really suck to lose it, considering how many different workflows and effects depend on it.

Cheers, stay healthy, and keep on doing amazing work!

10

I have also tested the Gen5 TAA now, and it definitely looks way superior to the previous one!

It is significantly more expensive, thanks very much @Guillaume.Abadie for mentioning the “r.TemporalAA.R11G11B10History=1” optimization, that does help a bit with performance. Are there any further optimizations coming to 4.26 for this?

But for some reason, with Gen5 TAA, I see way, way more ghosting on WPO foliage, and extreme ghosting on some translucent particles. I have “Output velocities during base pass” enabled in the project settings, and I think foliage correctly uses the velocities with Gen4 TAA, as I do not see any significant amount of ghosting on WPO foliage with Gen4 TAA. Gen5 TAA looks significantly sharper on non-moving foliage, but when there’s a bit of wind, Gen5 TAA looks quite bad on the foliage due to the ghosting. I have also tried enabling r.BasePassForceOutputsVelocity=1, but that does not help with the foliage WPO ghosting.

I’ve also compared it with DLSS, and DLSS does not have the ghosting on WPO foliage and translucent particles either. Apart from DLSS not having those issues, DLSS 2.1 and Gen5 TAA actually look very similar, with DLSS being more expensive (on my Turing GPU), while having slightly better results.

11

I’m using these settings with ‘Output velocities during base pass’ in the project settings and it seems to have improved the ghosting:

r.TemporalAA.Algorithm=1
r.BasePassOutputsVelocity=1
r.BasePassForceOutputsVelocity=1

1 Like
12

That are the settings I have tested with, but Gen5 TAA has the strong ghosting on WPO foliage with those settings, and the very strong ghosting on translucent particles.

13

re: weird noise

I’m seeing the same thing, but only with r.TemporalAA.R11G11B10History=1; with r.TemporalAA.R11G11B10History=0 the noise goes away. So it may be something with floating point precision? screenpercentage 50 doesn’t seem to be a hard cutoff for it, it gets worse as it goes lower. I can see it gradually accumulate in over many frames and get wiped out only if the area goes completely off screen and then back in.

With r.TemporalAA.R11G11B10History=0 I can go down to screenpercentage 25 with many less problems (there seems to be some bad fireflies after motion blur from turning fast or zooming, but they gradually fade out instead of indefinitely accumulating).

The noise largely has a blueish tint so it could be dengenerating fastest due to the 10 bit history for blue.

14

r.BasePassForceOutputsVelocity=1 breaks reflections on single layer water.

15

Hi Guys,

sorry for the delay…

Gen5 TAA is very unforgiving if the motion vectors are incorrect. On a WPO animation, this can happen when for instance animating with a material parameter without a PreviousFrameSwitch node to also feed what was the material parameter at the previous frame. More information here: Utility Material Expressions in Unreal Engine | Unreal Engine 5.3 Documentation

Best tool to verify the motion vector of your world position animations is the motion blur visualizer show flag

Untitled.jpg
Untitled.jpg576×1373 187 KB

I have made some improvements there in UE5 to be a bit more forgiving, but I won’t be able to cherry pick back this in UE4 due to how the code base has diverged a bit.

Yes, in fact I’ve made a change in UE5 so that r.BasePassForceOutputsVelocity=1 becomes automatic when r.TemporalAA.Algorithm=1 and r.BasePassOutputsVelocity=1

Don’t forget the r.TemporalAA.Upsamling=1 and change screen percentage, you can compensate the TAA by upscaling and go beyond and save frame performance by lowering r.ScreenPercentage without loosing to much quality in the upscaling. Gen5TAA and its costs are primarily intended for upscaling in term of quality but also performance. Many of it’s pass’s cost will scale with r.ScreenPercentage too.

I use r.ProfileGPU.Root=TAA and r.ProfileGPU.ShowUI=0 to quickly profile TAA in the editor’s console.

r.TemporalAA.Algorithm=1 r.TemporalAA.Upsamling=1 r.ScreenPercentage=50


LogRHI: 23.2% 1.29ms TAAU 2420x1221 -> 2420x1221 7 dispatches
LogRHI: 1.7% 0.09ms TAA ClearPrevTextures 2420x1221 1 dispatch 303x153 groups
LogRHI: 3.9% 0.22ms TAA DilateVelocity 2420x1221 1 dispatch 303x153 groups
LogRHI: 2.4% 0.14ms TAA DecimateHistory 2420x1221 1 dispatch 303x153 groups
LogRHI: 5.3% 0.29ms TAA FilterFrequencies 2420x1221 1 dispatch 303x153 groups
LogRHI: 2.3% 0.13ms TAA CompareHistory 2420x1221 1 dispatch 303x153 groups
LogRHI: 1.1% 0.06ms TAA DilateRejection 1210x611 1 dispatch 152x77 groups
LogRHI: 6.6% 0.37ms TAA UpdateHistory R11G11B10 2420x1221 1 dispatch 303x153 groups

r.TemporalAA.Algorithm=1 r.TemporalAA.Upsamling=1 r.ScreenPercentage=60


LogRHI: 17.3% 0.61ms TAAU 1210x611 -> 2420x1221 7 dispatches
LogRHI: 0.8% 0.03ms TAA ClearPrevTextures 1210x611 1 dispatch 152x77 groups
LogRHI: 1.6% 0.06ms TAA DilateVelocity 1210x611 1 dispatch 152x77 groups
LogRHI: 1.6% 0.06ms TAA DecimateHistory 1210x611 1 dispatch 152x77 groups
LogRHI: 2.1% 0.07ms TAA FilterFrequencies 1210x611 1 dispatch 152x77 groups
LogRHI: 1.1% 0.04ms TAA CompareHistory 1210x611 1 dispatch 152x77 groups
LogRHI: 0.6% 0.02ms TAA DilateRejection 605x306 1 dispatch 76x39 groups
LogRHI: 9.5% 0.34ms TAA UpdateHistory R11G11B10 2420x1221 1 dispatch 303x153 groups

Yeah I ran into similar issue on Lumen in land of Nanite demo. Fixed in UE5 and haven’t ran into other issues since to a point I was able to change the default of that cvar. I cherrypicked the simple fix from UE5 to 4.26.2 for you. :slight_smile:

Not as many as I had hoped, there has been some issues with devkit deliveries to test my changes, so we ended up needing to prioritize UE4 to remain stable given this is what Fortnite is shipping with on PS5 and XSX. Also there the source code has diverged quite a bit in UE5 that makes cherry picking some of the improvements back into 4.26 dificult. :frowning:

Stay safe! I’ll try to be a bit more responsive. Sorry about this :o

16

Couldn’t get that fix in 4.26.2 because invalidating too many shaders for a minor release, but you can cherrypick that change: https://github.com/EpicGames/UnrealEngine/commit/8631cb027cef93f1e59d0ea11a6fdcce041f3aca

17

I found that the issues I saw with ghosting on foliage was caused by the Speedtree Wind which was applied as WPO. If I replaced the speedtree wind with some custom wind calculation in WPO, there no longer was any ghosting with Gen5 TAA. It seems the Speedtree Wind somehow does not work with Gen5 TAA. It works nicely with Gen4 TAA, but with Gen5 TAA, it always resulted in that very strong ghosting. It’s no longer an issue for me now though, as I just stopped using Speedtree Wind and replaced it with custom wind, as that also has other benefits.

I also managed to fix the ghosting on lit translucent particles by enabling the “Responsive AA” option for them. For some reason Gen4 TAA does not need that on them, but with Gen5 TAA, it’s absolutely required to not have very strong ghosting.

Ah, too bad! Would love to have those improvements in 4.26, as I’m not sure if I’ll ever be able to update to UE5. Having a well working Gen5 TAA would be a huge visual improvement for me.

I have basically done all my tests with a screen percentage of 50, comparing Gen4 TAA and Gen5 TAA. In my tests Gen5 TAA is usually roughly as expensive on 50% SS as Gen4 TAA is at 70% SS. So I’ve spent quite some time comparing which of those options looks better overall.

The main issue I noticed with the Gen5 TAA now after more testing, and after I fixed the foliage and particle ghosting issue, is that Gen5 TAA looks bad when the player moves (walks) around. The image with Gen4 TAA always looks pretty much the same, no matter if the camera is static, or moving. Gen5 TAA looks really good when the camera is static, or only rotating. But when the camera moves, the image quality starts to fall apart, and there are a lot of areas in the image which look as if there would be no AA at all, so very pixelated. What’s somewhat weird is that it’s not happening on the whole image uniformly, instead there are “blotches of pixelated areas” on the image when the player moves. These blotches seem to be somewhat local to specific screenspace areas. Does that make sense? Why does the image quality not uniformly become worse? Why does rotating the camera around (Yaw/Roll/Pitch) look perfect, but moving it (X/Y/Z) look bad?

Due to this issue, Gen4 TAA overall unfortunately looks better as it has more consistency in the image. Gen5 TAA is superior when the player stands still and only looks around, Gen4 TAA is superior when the player moves. And the fact that there is no significant difference in image quality between static/moving camera with Gen4 TAA makes looking at that more pleasing, while the constantly changing quality of the Gen5 TAA feels bad.

Is that deterioration of the image specifically while the camera moves expected with the Gen5 TAA?

If there are improvements that could be cherry picked, but as just too “dangerous” to be added to the 4.26 branch, would it maybe be possible for you to still make those accessible to be cherry picked by people who want to experiment with trying them out, maybe by just creating a PR to 4.26 for each of those improvements? I’d love to try out any such changes if they might improve the image quality of performance of the Gen5 TAA in 4.26. It really feels like not much is missing for this to be an amazing feature, if I could somehow get the image quality with a moving camera more similar to a static camera (or at least as good as with Gen4 TAA), and maybe slightly improve the performance, I could make Gen5 TAA the default option in my game, and that would hugely improve the visual quality.

Thanks very much for your great work on this!

2 Likes
18

Hi John,

based on what you describe, the core issue is that the SpeedTree’s calculation for the motion vector is wrong that is causing Gen5 TAA to ghost. We haven’t investigated that issue. :frowning:

So here Gen5 TAA have quite concerning ghosting issue when the particle becomes too small on screen. This is high on my list to tackle, but I still don’t have a solution that doesn’t kill performance yet. :frowning: When the responsive AA is enabled, Gen5 TAA actually completly behave identically to Gen4 TAA with its own trade off.

Another alternative has been to enable this translucent particle to draw velocities in the material settings and relis on Gen5’s TAA motion vector based parrallax disocclusion heuristic. However we have found that the motion vector for niagara particle where not as accurate as we would have liked, and we have been investigating what solution we will have to have accurate motion vector. Stay tunned.

Oops that is interesting, are you targetting 120hz? What GPU you have? Have a ProfileGPU log output to share?

I’ve never seen that artifact before. I wrote Gen5 TAA initially using Lumen in the Land of Nanite demo that have a character walking too before cherry picking this work in 4.26 for Fortnite on next gen consoles. Fortnite uses characters too and isn’t running into such issue as well.

To me this sounds like another motion vector problem in the skeletal mesh code path you are specifically using. The visualize motion vector I posted earlier might help you indentify if this is the issue. Feels free to post screenshots you feel confortable sharing. Maybe I’m wrong and there is a bigger problem for the particular content you are trying to build. :wink:

I preffer to stay honest and transparent and please apologises for any disapointment: I don’t know if I’ll have the bandwidth myself to maintain a 4.26 pull request with all the lattest of Gen5TAA in it. The challenge of TAA is that all the pixel of renderer goes through it, and this implies pretty high standard on quality and performance needs for Gen5 TAA to successfully ship in production. Due to the amount of work generated by this goal, we indeed have a higher priority for me to focus on improving quality and performance. Moreover we are also dog fooding it on internal project too where I also need to help when something goes wrong. :o

Having said that, we find important and I personally like taking the time to throroughly help you guys out on this forum thread with detailed response to as many question you all may have, because this is centralized knowledge any user of current and futur version of Gen5 TAA might also be please to find to understand and fix the issues they might be running into that ultimately is also part of my job to make the transition to newer TAA as smooth as possible for everyone. This also helps me making sure feedback about this functionality remains positive and there isn’t any huge problem I could have missed. :rolleyes:

While some UE5 improvement to Gen5 TAA are still not publicly available on github at this day, I would personnaly love to comment further details than I already have, but sorry I don’t think I can just yet. :rolleyes:

One thing I can comment though is it is still under heavy developement, and given the variety amount of content rendering through TAA, a regression on specific content case can happen, and might not be ideal as an user experienced. TAA is a delicate hand crafted heuristic to decide whether to use previously rendered frame or not. Gen5 TAA is just a steroid boosted (and slower :p) crafted heuristic. Hand crafted because this is the only thing we can do to work on maximum number of high end platforms that may not have expensive machine learning hardware. However not every user have the time or the knowledge to go tune an heuristic as complicated as Gen5 TAA for his content. So my goal is to make a TAA that just works (assuming there isn’t yet again wrong motion vector rendered somewhere for some pixel :p). The problem of hand crafting means decision needs to be made over what artifact should be prioritised over others. And this heavy development may also need to have some tradeoff between some artifact versus the others to also be reevaluated. And I know this as a user might be very frustrating because it used to work well before, but not anymore in his unfortunate very specific case. All that to say, what we have release in 4.26 might not be the lattest sadly, but it might not be too bad. :cool: Despite 4.26’s is shipping at production scale on Fortnite, it is still an experimental tech that may change.

19

Thanks for the info!

Yes, I am targeting and have tested it with 120 hz. Tested with both a RTX 2070 Super and a RTX 3090, the ratio of how much slower Gen5 is compared to Gen4 seems to be roughly the same. I till now mostly only compared total GPU frametime, not specific TAA time, so I don’t have exact profiling for those. I don’t want to spend too much time with profiling it now though before I found some way to fix the pixelation/aliasing issue, because before that’s fixed, the performance doesn’t actually matter too much as it’s not really usable due to visuals anyways.

The issue can’t be related to any skeletal meshes, I am not using any skeletal meshes. The player camera is just a floating camera, my player neither is a ACharacter (it’s only a APawn), nor has a mesh.

There are very clearly visible blotches of aliasing/pixelation traveling across the image when the camera is moved. Not when the camera is rotated or static. Only when the camera is moved. And Gen4 TAA does not have that issue.

I can’t post a screenshot publicly, but I’ve sent you a short youtube video of a few seconds of screen recording in a PM where you can clearly see those blotches of pixelation traveling across the screen when the camera is moved. The issue is same no matter if R11G11B10History is enabled or disabled.

That sounds like it might be possible to tweak some parameters to more ideally match the TAA performance to our own content though, if we wanted to spend some hours with fine tuning parameters for our specific content. I’d definitely be willing to do that if it means getting a better result at the end. Having something that works well out of the box is cool, but fine-tuning parameters also is not too big of a deal. Would it maybe be possible, for those people who want to do it, to allow fine-tuning such parameters with some console variables, or maybe even just shader constants that can be adjusted in the TAA .usf or .ush files? If changing those does not trigger a whole engine shader recompile, that sounds easy to adjust and tweak.

20

How much stuff is hardcoded to the view matrix jitter being within a single input pixel’s span? I’d ultimately like to use temporal upsampling in combination with variable rate shading, for VR, and that would require jittering within the span of the largest VRS block of pixels (probably 4x4).

I tried just modifying the jitter span from -0.5/+0.5 to -4.0/+4.0 to see and it seems ok for gen4 and gen5.

I didn’t try mixing in VRS yet and that would also need other changes to make the reconstruction aware of it. I think the Halton sequence may also need to be changed to be multi-octave as well.