TSR feedback thread

Guillaume.Abadie · April 10, 2023, 6:25pm

Hi,

My name is Guillaume Abadie, principal graphics programmer at Epic Games, author of Temporal Super Resolution in UE5.

The goal of the TSR is to be a temporal upscaler that answer the anti-aliasing need of displaying unprecedented Nanite amount of details on screen, while accommodating enough GPU performance to entire renderer by rendering lower resolution than we were before, to in turn enable as many rendering feature as possible to produce the best image as possible on all the platform we support (such as Lumen, VSM, Volumetric clouds and many more). But the cherry on the cake for TSR being such a technical challenge is the fact that absolutely all 3D rendered pixels are going through it, transforming the problem to solve into a very delicate balance between needs of pretty much all the different rendering use cases each art department may envision in their own project. As a result the development of TSR has been a humbling 3 years ride of exploratory research and experimentations, many more failures than successes to learn from, and major changes of technical directions upon each incremental technical discovery.

It started with very early experimental prototype after our experience of Lumen in the land of Nanite demo and how such technology was required to hope to ship at 60hz on now current gen consoles PS5 and XSX. While born in UE5, it was “hidden” in 4.26 (r.TemporalAA.Algorithm=1) because we wanted to experiment how much work it would be to ship in Fortnite that was still on UE4 at this early time. Suffice to say I wouldn’t recommend a ship to project with it, and it became a very different beast in 5.0 from our learning of shipping the Matrix Awakens demo at 30hz on console.

It then substantially changed again for the shipping at a production scale of Fortnite Chapter 4 at 60hz on console: needed to shrinking its runtime cost from 3.1ms to 1.5ms on console 1080p → 4k; and yet improve on the compatibility with the gigantic amount of content that had already been authored in previous seasons of Fortnite without adding migrating cost on the art team.

Release schedule 5.1 compared to Chapter 4 and all the changes needed in TSR made it such that is was still not stable enough which is not great considering absolutely all 3D rendered pixels are going through it. This is where the upcoming 5.2 release is in fact a big deal for TSR being the first release both production proven state, stable and with more debuging tools.

For instance 5.2 adds a VisualizeTemporalUpscaler show flag designed to quickly diagnose common ghosting problem (wrong velocity vector of object, translucency ghosting, overdark pre-exposure of our internal buffers causing TSR to be mislead of the actual exposure of pixels to the user, or more generally any artifacts actually produced by other rendering algorithms).

There’s been quite some hype and noise about upscalers recently and sometimes technical information is mixed up with marketing messages, and that could lead to misunderstandings: for instance there was some confusion around its role to fit the Matrix Awakens demo in GPU budget on consoles ( The Matrix Awakens Tech Analysis + PS5 vs Xbox Series S/X Performance Analysis - YouTube ). In this instance TSR got a little bit victim of its own success hiding too well how low the rendering resolution was on this demo and changed/eliminated previous paradigm used by news channel to know what is in fact the rendering resolution ( https://twitter.com/Dachsjaeger/status/1469927361365880835 ). Also shortly after that time there were UE 4.2x games that had exposed the older prototype as TSR to the player, and that caused even more confusion as to what TSR was actually able to deliver.

TSR 5.0 Low (r.AntiAliasingMethod=4, sg.AntiAliasingQuality=0), ScreenPercentage=50, 8x zoom in on camera cut (r.Test.SecondaryUpscaleOverride=8, r.Test.CameraCut=1), without the spatial anti-aliaser where it is easy to count rendering resolution thanks to the aliasing on geometric edges.

TSR 5.0 Medium (r.AntiAliasingMethod=4, sg.AntiAliasingQuality=1), ScreenPercentage=50, 8x zoom in on camera cut (r.Test.SecondaryUpscaleOverride=8, r.Test.CameraCut=1) with the spatial anti-aliaser that introduced between 5.0 Early Access and The Matrix Awakens for 5.0 where it is no longer that easy to count rendering resolution on geometric edges.

Therefore we have been very careful projecting a fair image of the tech to our players and developers: Showing just frame rate increases as the rendering resolution lowers is glancing over an important compromise any temporal upscaler is subject to: discarding previously accumulated details for various reasons brings on a fundamental problem that is is impossible to know how a new details should ultimately look on screen until enough information is accumulated.

(CitySample editor, r.AntiAliasingMethod 4, r.Test.SecondaryUpscaleOverride 4, ScreenPercentage=61)

(CitySample editor, r.AntiAliasingMethod 0, r.Test.SecondaryUpscaleOverride 4, ScreenPercentage=61)

Knowing this information is in fact directly tied to knowing more information about the details that is limited by how fast detailed information is accumulated. Therefore 5.2 added a new stat tsr command that shows how two very important metrics the TSR output quality is directly dependent:

TSR feed: how many pixels per second is being rendered that represent total amount of information TSR can work with to converge to the image. This metric has the property of communicating how a final image can look at macroscopic level at distance of the monitor, regardless of the displayed resolution.
TSR 1spp: the convergence speed of details per pixel displayed. The lower the resolution gets (either r.ScreenPercentage or r.DynamicRes.*) the more frames are needed to have 1 rendered pixel per displayed pixel. How fast a frame is accumulated being directly tied to frame rate that not only depends on GPU performance, but also CPU performance and VSync, this metric is shown in milliseconds to display a better understanding how all of this is tied together and directly affects TSR’s microscopic details accumulation.

(stat tsr command completing stat unit command in UE 5.2)

I personally hope you will like TSR whether it is in UE 5.2, Fortnite Chapter 4 season 2 or Fortnite Creative 2.0. I also hope that these few words would convince any of our licensees shipping a title on 5.0 or 5.1 to consider merging 5.2’s TSR and its tools back. But also hope this catalyzes healthy feedback on this thread about what you think of TSR, and most importantly where are the weaknesses that affects your project with concrete illustration, and would like to see it improve in future release of Unreal Engine.

Looking forward to hearing from you!

TemporalConsistencyTSR
(giff of TSR 5.2 Epic scalability at ScreenPercentage=50)

TheKJ · April 12, 2023, 9:04am

Thank you for creating this thread and bringing these new features to unreal 5.2.

To see Epic implementing these tools and fixes in response the community feedback on TSR is very commendable.

I think TSR has a lot of potential, beating DLSS and Epics TAA in the ghosting department. But it seems inconsistent at times.

Take the City Sample for example. It’s a great UE5 benchmark, and without AA it’s unacceptable at 1080 with no AA (Imo, 4k doesn’t need any AA unless certain features are dependent on a temporal filter). FXAA doesn’t fix aliasing. And TAA is horrible and insulting to UE5 features like nanite because of the temporal blurring and ghosting.

TSR upscaling almost accommodates for the temporal blurring at least for more close objects. A custom depth-sharpen post process filter+ a camera blueprint that changes the sharpening values based on camera velocity can fix the controversial blurring effect. But here are some issues that a dynamic sharpen BP idea can’t fix.

Compared to TAA, TSR’s ghosting is almost nonexistent, but:

As you can see in the video that light pole smears on the sky. This is only an example of a strange smearing bug in TSR that happens on any object. As mentioned in this post

Hopefully this a bug being worked on in 5.2.
Or at least fixed with the new debug tools
TSR does indeed have better image stability and has way less ghosting, making it the goto for most UE5 devs. But there are still problems that will hold back UE5 from its fullest visual potential.

Like for instance when the camera is rotating around an object. The road smears on the characters face. NOT acceptable. I’m not dissing TSR or the people who made it, but these issues that must not be ignored for the sake of UE5, current gen games, and healthy maturity of TSR:

This happens on all characters even from a standard third person distance.

Here’s a character rotating in 5.2 preview 2. It’s the same problem (I tested with same meta human head but spent all day working on this presentation/recording).
I bring this up for a concern for fighting and action games.

But there are some issue’s that I find very worry some. Hopefully with the final UE5.2 release these odd ghosting artifacts will be solved. Game like 505’s next game other studios will have vast open worlds.
The Epic team said fast creation of open worlds is the goal but TSR is going to going to cause some problems.
Far away moving objects from a stationary camera, ghosting. This next video below shows this problem and needs work as well.

Listen, I personally hate temporal stuff because the overall blur hurts my eyes, but TSR has a lot of potential and I want to stay positive by giving my feedback on it.
I’m no expert in graphic programing but I think I have done as much digging on UE5 Anti-Aliasing as any normal person could.

I watched the 5.2 demo and was very disappointed to hear it ran on a 4090 and still looked blurry when editing the level(it was beyond the lack of motion vector creation when new assets are copied or moved into scene for the first time).

Another thing is the confusion around how certain games on PS5 produced by Epic Games or looking less artifact’s on RNDA2 console architecture.
Maybe a presentation on how to optimize TSR would help devs become better at making crisper and less smudgy looking games on PC architecture if that’s currently possible?
Bad or good enough AA isn’t going to be good for UE5. It needs to be amazing, on par with its other features like Lumen and Nanite. Or else, it’s just a step backwards.
I hope to find UE5 looking great and clear by the time I move to digital game production.

Thank you for reading. Typos (If I missed any) will be fixed but I’ve been working on this post for several hours, recording, properly testing, and writing this.

Guillaume.Abadie · April 12, 2023, 6:20pm

Hi!

Thank you for reading. Typos (If I missed any) will be fixed but I’ve been working on this post for several hours, recording, properly testing, and writing this.

Welcome and thanks for the genuine and extensive feedback, spending this time is well apreciated! Will do my best to be concise on verything said but please to not hesitate to bring up anything again I could have missed in my futur repplies.

(and no worries about typo and stuf, my english isn’t my native language and so it is far from being sheakspear quality , raw content I only what I personnaly care ).

TSR upscaling almost accommodates for the temporal blurring at least for more close objects. A custom depth-sharpen post process filter+ a camera blueprint that changes the sharpening values based on camera velocity can fix the controversial blurring effect.

So TSR developements in 5.1 was really focused on both PS5, XSX and XSS consoles and sadly shipped with an unfortunate overbluring issue in 5.1 especially on epic scalability settings.

That is because in few time dedicated for PC, an interesting discovery was made arround reprojecting frame N-2 into N to know how much it has been overblured when reprojecting into frame N-1 and then N. This is r.TSR.History.GrandReprojection. This new technic eliminated other technics use to counteract overblur at expense of image stability. But in attempts to optimising its runtime performance, it ended up loosing bit of its own quality. Good news in 5.2 is that it has been replaced with r.TSR.History.ScreenPercentage=200 while more time is being invested on this ( https://github.com/EpicGames/UnrealEngine/commit/9ccd56dfcc06852b5b89a81411972f81e3ac31e3 ) on epic scability settings that previously only on cinematic scalability settings. It’s already used in Fortnite Chapter 4 Season 2 and I’m really eager on community feedback on this change on 5.2. Sadly I’m failing to understand wheather your video are on 5.2 or 5.1.

This TSR change also exposed couples of new cvars to expose trade off between image blur or stability in reprojection notably with r.TSR.History.SampleCount, r.TSR.ShadingRejection.SampleCount, r.TSR.Velocity.WeightClampingSampleCount. As an example, Fortnite Chapter 4 Season 2 being competitive with a lot of shooting, is shipping with r.TSR.History.SampleCount=8.0 and r.TSR.Velocity.WeightClampingSampleCount=2 in the /Config/DefaultEngine.ini. If these looks intimidating to change TSR settings like this, I’ve added in 5.2 more extensive explanation of each TSR cvars, among with flush caveats and trade off each have in this change which I hope will make customizing TSR settings a lot more intuitive: https://github.com/EpicGames/UnrealEngine/commit/2d3e9b6e1b9e66dbe2807c9c80ee64987bc708f2 . Planning to have all these cvars and their detailed explanation exposed in TSR online documentation for 5.2 release or slightly after.

Naturally how much the image look statically can be hit different taste, whether it’s player test with setting often directly controlable on TV or monitor or content creator in their project. For instance Fortnite adds r.Tonemapper.Sharpen=0.5 on all platform using TAA, TSR or third party upscaler. Now interestingly with that r.TSR.History.ScreenPercentage=200 is could expose even better look sharpening directly in the TSRResolveHistory.usf that does the downsample from history resolution to display resolution using a Mitchell-Netraval spatial kernel. Right now the parameters are hard coded in the shader but we could envisions a futur where sharpening instead happens here.

As you can see in the video that light pole smears on the sky. This is only an example of a strange smearing bug in TSR that happens on any object.

That building disocclusion happening is very interesting case where there can be some improvements: TSR knew how the building look before the pole moved over. Problem currently is when TSR discard the history, it is forever because it only keeps its history for frame N-1. It also keep N-2 for r.TSR.History.GrandReprojection, but depending of the thickness of pole, this might still not enough: maybe TSR discarded the details of that building at N-6. I’ve been thinking a lot how mechanism for doing more general purpose history resurection (for the lack of a better name) but there is couples of performance problems to tackle:

detecting when this can be done cheaply;
selectively reproject that holder N-X data into N which this experimental r.TSR.History.GrandReprojection brought up a lot of new interesting possibility especially with the use of D3D Texture2DArray;
TSR history memory footprint: Keeping say the last X frame in memory to possibly reproject doesn’t scale well as your pole thinkness on screen could be.

Sadly how much smarter TSR becomes of that “history resurection” on disocclusion, it is still important to keep in mind there will still be disoccluded areas on screen that will be discovered the first time and have the quality tied there only to how fast details are accumulating at image macroscopic and microscopic details.

This is where I hope the new in 5.2’s stat tsr command will bring more understanding that there will always be a limit how much lower the resolution can be, how much ScreenPercentage can be lowered based of the display resolution and is the reason for instance the screen percentage on a XSS in Fortnite chapter 4 presenting at 1080p is higher than on PS5 and XSX presenting at 4k.

That is also why on Fortnite on PC I added that default Recommended setting in the Temporal Super Resolution configuration that suggest a screen percentage based of displayed resolution using FStaticResolutionFractionHeuristic.

	FStaticResolutionFractionHeuristic StaticHeuristic;

	StaticHeuristic.Settings.PullRunTimeRenderingSettings();
	StaticHeuristic.Settings.Mode = EScreenPercentageMode::BasedOnDisplayResolution;
	StaticHeuristic.TotalDisplayedPixelCount = ScreenWidth * ScreenHeight;

	float RecommendedResolutionFraction = StaticHeuristic.ResolveResolutionFraction();

Hoping to bring that successful new Recommended setting by default in Lyra for future release of the engine.

Hopefully this a bug being worked on in 5.2.

Please be aware 5.2 already branched of UE5/Main and is being stabilized for release and when that happen it means most often, current ongoing developement is future release for stability reason.

Like for instance when the camera is rotating around an object. The road smears on the characters face. NOT acceptable. I’m not dissing TSR or the people who made it, but these issues that must not be ignored for the sake of UE5, current gen games, and healthy maturity of TSR

Yeah I see what you mean, this has been known to happen. But my experience this has been mostly noticeable at low frame rate or resolution that is very low TSR feed (MP/s).

However through your video, not sure to understand your resolution or frame rate due to the various limitation of video and its encoding. This is my falt I should have said on my first message what was the extent of information I was meaning when saying “illustration”. Screenshot, videos are best way to show the visual artifact, but it is always important to add some additional information, such as display resolution, rendering resolution (which the VisualizeTemporalUpscaler show flag have in 5.2 conveniently), and frame rate. The stat tsr in 5.2 will be ideal to record and share video of TSR failure too because displaying all this information directly on the video.

Would you mind sharing these further details you recorded your video with?

Far away moving objects from a stationary camera, ghosting. This next video below shows this problem and needs work as well.

We ran into this issue many times during the developement of the Matrix Awakens: the way the cars are drawn in the distance is not the same as when they are closer. And that code path drawing the geometry that also draws motion vector can regress for various and sadly to many reason. Whether this is engine regression (which it looks like in 5.2 because I swear this was fine in the demo released on console) or project settings or material settings or missing PreviousFrameSwitch to feed previous frame’s value of animated material parameters in some WPO animations for instance.

Because we have been very tied to runtime performance, no time was spent in TSR to do optical flow analysis to try to ease developement burden of motion vector break for various reason. So this is where in the time we had in 5.2 was first spent on tools to more quickly diagnose this sort of problem. Just open up that show VisualizeTemporalUpscaler or show VisualizeMotionBlur or show VisualizeReprojection and this should show immediatly how the motion vector look. This is where the upcoming documation of TSR is also being actively worked on to better explains how to diagnose the core reason of this kind of artifact.

Doing optical flow in TSR we are already investing research on, for better reprojection of translucencies, reflections, fast moving shadow. It’s just a matter of finding a solution that works with largest amount of content variety and in performant maneur universally on existing plaftorms we support in the end!

Listen, I personally hate temporal stuff because the overall blur hurts my eyes, but TSR has a lot of potential and I want to stay positive by giving my feedback on it.
I’m no expert but I think I have done as much digging on UE5 Anti-Aliasing as any normal person could. The devs who worked on this are brilliant and are on a whole other level than I am.
I watched the 5.2 demo and was very disappointed to hear it ran on a 4090 and still looked blurry when editing the level(it was beyond the lack of motion vector creation when new assets are copied or moved into scene for the first time).

Thanks a lot for your time, it is very well apreciated!

Another thing is the confusion around how certain games on PS5 produced by Epic Games or say a certain fighting game by ramdi camco looking less artifact’Ish on RNDA2 console architecture.
Maybe a presentation on how to optimize TSR would help devs become better at making crisper and less smudgy looking games on PC architecture (If that’s currently possible?).
Bad or good enough AA isn’t going to be good for UE5. It needs to be amazing, on par with its other features like lumen and nanite. Or else, it’s just a step backwards.
I hope to find UE5 looking great and clear by the time I move to digital game production.

Or better documentation? You are definitly right and it’s entirely my bad here! TSR in 5.0 had many problems and documenting these problems was a lot of time not spent on significantly improving the tech instead that was dearly needed. A lot of these improvements in 5.1 and rest in 5.2 and this is where the focus has been on tools and documentation for 5.2 release. This is also the goal of that public thread between community and Epic, where questions can be asked and accurate technical response can be shared to everyone immediatly while we learn what is missing and need to update our online our doc.

TheKJ · April 12, 2023, 10:25pm

Thanks for the long thorough reply. Very much appreciated.

Good news in 5.2 is that it has been replaced with r.TSR.History.ScreenPercentage=200 while more time is being invested on this ( https://github.com/EpicGames/UnrealEngine/commit/9ccd56dfcc06852b5b89a81411972f81e3ac31e3 ) on epic scability settings that previously only on cinematic scalability settings. It’s already used in Fortnite Chapter 4 Season 2 and I’m really eager on community feedback on this change on 5.2. Sadly I’m failing to understand wheather your video are on 5.2 or 5.1.

For the City Sample videos, I recorded at epic aa settings, 1080p, 100% ScreenPercentage on 5.1.

The only video I recorded 5.2 in was the one with the starter mannequin and that was also at epic, 1080p, 100% ScreenPercentage.
I try to set shadows, GI, and reflections on low settings in most of the test I do with TSR to rule out any lumen noise.

I did more test this morning In 5.2 preview 2 at 1080p, 100% ScreenPercentage. I test every scalability (Low, Epic etc) option for AA in the starting content and TSR. With 100fps, I made the mannequin run back and forth, made the mannequin rotate and rotated the camera around the mannequin.

On every setting including cinematic TSR caused the background to smear on the mannequin’s body. The character looked fuzzy and sharpened 0.3 seconds later.
In the video, it’s 5.2 epic AA setting, same results with >100 or 100 screen percentage at 1080p. I am lowering video resolution so I can attach it to post. But its the same problem and its clear enough. Same results with 30fps cap

The stat tsr in 5.2 will be ideal to record and share video of TSR failure too because displaying all this information directly on the video.

You’re getting me excited here lol.

I haven’t tested Fortnite on PC but I should probably test out your TSR implementation beyond videos on YT.

At a caped 30fps those smearing artifacts(like the lamp pole) on the sky and the taller gray wall behind
I replaced mannequin, this is 5.2, 1080p, 100% ScreenPercentage, cinematic(same with low-epic)

Now maybe that doesn’t seem to concerning but this isn’t really much of a game? Take a game like Tekken 8?
You can check the videos out at regular speed and then slow-mo on ty.

That game is very fast paced and has quick movements.
Ever since it’s TGA trailer, gameplay reveals trailers look fuzzy and at first the brain doesn’t realize what’s makes the visual so odd and unpleasant, until you slow it down.
There’s you can find number of inconsistent pixels left behind from characters movements. In regular speed, these artifacts accumulated into a fuzz, like in the rotation video I included using the UE5 mannequin

It’s a bit worrisome to see triple A studio be totally fine with this. It makes me wonder about other games coming from UE5 I don’t want action-oriented games to have a similar fuzziness that fighting game looks. Is it the devs or TSR?

For me, I want the action in my game to look crisp, so far, I’m just worried about the AA options in UE5. And like I said before, TSR hits the other current options out of the park. Fast Motion seems to be its biggest weakness.

I’ll be posting the 60fps scenario soon of that video above me (Same settings but that fighting game really shows off that background smudgy at 60fps very well)

Guillaume.Abadie · April 14, 2023, 4:13pm

Thanks again for taking the time!

I try to set shadows, GI, and reflections on low settings in most of the test I do with TSR to rule out any lumen noise.

This is good idea. Another reason is that TSR isn’t the only temporal accumulation of the renderer. Reflections and GI is also only practicable in a performance maneur with temporal accumulation too, but they are as much also subject to ghosting for instance due to some specific on how these temporal accumulation works. this is where the VisualizeTemporalUpscaler show flag in 5.2 should help better diagnose if it is indeed TSR or something else that ghost just by looking wether the ghosting is solely in TSR.Output or also in its inputs!

On every setting including cinematic TSR caused the background to smear on the mannequin’s body. The character looked fuzzy and sharpened 0.3 seconds later.
In the video, it’s 5.2 epic AA setting, same results with >100 or 100 screen percentage at 1080p. I am lowering video resolution so I can attach it to post. But its the same problem and its clear enough. Same results with 30fps cap

Beware of the Edit > Edtior Preferences > Performance > Override game screen percentage settings with editor settings in PIE (or r.Editor.Viewport.OverridePIEScreenPercentage). I’ve had many that got hit with this caveats.

Always worth checking the rendering resolution and there is different way to checks. First, TSR draw events displays rendering resolutions so they are visible in ProfileGPU or DumpGPU or any other third party GPU profiler:

ProfileGPU’s UI:

ProfileGPU log:

DumpGPU:

Also even more convenient in 5.2 the VisualizeTemporalUpscaler shows input and output resolutions.

Now, on the reason TAA or TSR look fuzy even at ScreenPercentage 100 under large movement is: Each frame the grid of pixel being rendered is jittered. That is why in vis SceneColor for instance it is very unstable.

But this means for new areas on screen previously not visibles, the only informations that is the rendered pixel are for the same reason slightly shifted from the output.

Moreover for the upscaling case, different input resolution from output resolution means different distance between input and output pixels that necessarily means there is always be a rendered output pixels where the rendered pixels are not aligned.

This is resampling of the rendering resolution into output resolution that cause this blurs and is a limitation of many (if not all) temporal upscaler. Then adds TSR’s spatial anti-aliaser on top trying to clean the image a bit also adds a bit of blur, as much r.AntiAliasingMethod=1 (FXAA) is a bit blurier than r.AntiAliasingMethod=0 (No AA).

The reason we stick with such anti-aliasing technic is because it is self contained in the renderer. MSAA we support on desktop and mobile renderer doesn’t suffer this issue, but on the otherhand it is straight up impossible to maintain resonably for the deferred renderer due to how invasive MSAA is.

Different games have different needs in terms of anti-aliasing, and there is many games that do indeed prefer alternative anti-aliasing solution like MSAA for image clarity in very fast paced movement throughout environement, and that is ok. But it just come at an engineering and runtime price.

It could be possible for TSR to try to mitigate that pixel realignment blur, but for performance reason, it’s GPU pass doing upscaling, update of history and output is running at display resolution = often the most expensive. Being tied to one pass means only one layer of convolution so a sharpening needs to happen in a pass after. This is where currently tonemapper is great fit, and for the case of r.TSR.History.ScreenPercentage=200 there is that additional TSR resolve pass that does downsscaling from history resolution to output resolution, but that r.TSR.History.ScreenPercentage=200 is expensive too, probably not use on a currernt gen console. A very particular interesting case here however being integrated in TSR that that resolve pass could adaptively sharpen based on much details is accumulated. I’ve filed Unreal Engine Issues and Bug Tracker (UE-183345) (which should become public soon) for you to be able to track progress on that cool idea.

Thanks a lot for your continued feedback!

YohanR · April 17, 2023, 1:28pm

The Temporal Upscaler Visualization does not seem to be present in 5.2 Preview2, does it need a command to show up or config change?

Guillaume.Abadie · April 17, 2023, 5:35pm

Hello!

The Temporal Upscaler Visualization does not seem to be present in 5.2 Preview2, does it need a command to show up or config change?

Sadly we dicided to integrate from UE5/Main to 5.2 right after 5.2 Preview 2 that is why it is missing. But you can find the change on github in 5.2 right here: https://github.com/EpicGames/UnrealEngine/commit/2c492529cc0137023c2fced237f7225086b0c985

TheKJ · April 20, 2023, 5:35am

Deleted.

Guillaume.Abadie · April 20, 2023, 8:19pm

Hi, lot to unpack there, thanks for taking the time!

I’ll start off with the character’s motion(I think her name is Echo?). When she walks, runs, rotates her body, starts flying: I NEVER see ghosting.

Yes that is right, we shipped this demo on TAA even thaw it was eating away lot of Nanite amount of details. We got a little bit away with murder there thanks to public seeing this amount of geometric details in real time only for the first time anyway. This is this very demo that motivated the start of development of our new anti-aliasing tech. The reason the character mostly doesn’t ghost in thanks to TemporalAA.usf’s AA_DYNAMIC_ANTIGHOST which reject the history when a dynamic pixel (that draws velocity like animated skeletal mesh) no longer draw velocity (like the grand majority of this environement that are just static nanite mesh).

Matrix awakens, now I know this used TSR.

Yeap. 5.0’s TSR.

After the City Sample released, then it was on. A lot of people including myself are very confused as to why we devs can’t achieve that level of clarity in these two presentations from Epic.
Everything seems so blurry/fuzzy unless the camera and content are slow and boring. I haven’t been able to test it on hardware higher than my 3060 but UE5 looks blurry and fuzzy on every project like the third person starter content, City Sample, etc.

I even tested the City Sample at 4k with my 3060 at 40fps. TSR didn’t look sharp like that matrix demo. This is painful. I tested with and without motion blur, lumen and still got disappointing results compared to those Unreal demos.

Me and some others lost hope in UE5’s ability to actually achieve this desired clearity from those demos. I’m not the only one who has spent hours and hours of research, testing this. For months now, I and others I’ve met online have been trying to achieve that simple clearity in those Unreal demo videos.

I can’t put words in the name of all the users we have, but very often I’ve seen that happening when supporting our licensees, that is for instance CitySample in 5.0 is very CPU intensive with the entire simulation of city that is very demanding, tanking frame rate and as result the amount of MP/s feed into TSR which is very important.

But what I’ve learnt through out awswering our users is that connection between CPU performance and the consequence it can have on the look of TSR was really non intuitive and unexpected to the users. So this is why in 5.2 that stat tsr not only show the microscopic and macroscopic metrics that directly display how fast TSR can accumulate details in the history, but also all of the stat unit too. So in one command you can identify TSR is being pushed beyond it’s limits due to very low MP/s, which might be due not because of rendering resolution, but the frame rate, which the stat unit also shows game thread, render thread and rhi thread timings too. So in one command it becomes a lot more intuitive that the image quality can in fact end up consequence of something completly unrelated to TSR bottlenecking the entire frame rate.

TSR in fortnite looked okay at 4k and 1080p.
I didn’t find a lot of ghosting even with the shader compiling stutter poking at the temporal frame rate stability.
Clarity didn’t look too fuzzy, if not fuzzy at all when sprinting(I tested high and epic TSR at native resolution).

I apreciate the compliment but that is not entirely representative too. For instance water waves could ghost and just been recently fixed in 5.3 ( https://github.com/EpicGames/UnrealEngine/commit/7c20c06df00b9a8f8fc5641c468800dd6f7c0199 ). There is also still quality problem on the grass always moving due to the tricky challenge of upscaling a velocity buffer for better accuracy in history reprojection.

But there was one major problem I found: Characters eyelids have major ghosting. It looks like the eyes are glitching and it looks painfully bad.

Yeap this a challenging case where there is clear motion vectors of the eye lid, but because the eyelid are stylized in such way it is so thin, it makes it impossible to detect the eye lid actually occluding and disoccluding the eye reliably with motion vectors. The eye lid once fully reopened also completly disapear into the head, not drawing velocity at all making it impossible to see the difference. We didn’t run into this issue in the Matrix Awakens because the eye lid are a lot more thicker than in fortnite, and a lot more closer to the camera.

The eye lid once up also completly disapear into the head, not drawing velocity at all so. So there it’s only with difference in shading between the eye and the eye lid we can work with but challenge start to arise to make the difference between a eye lid that close and open in very few frames versus a moire pattern between strcutured geometry and structured pixel grid. So until a better and cheap solution is found, had to come up with compromise. The problem of this compromise here is that one issue only happens on small part of the screen versus another can happen much more widely in an environement. Except we are human, and our brain is trained at looking at other human for biology and social reasons and which make this very few pixels of the eye matter a lot more.

Second: Gameplay footage from new Tekken 8 trailers after that first reveal looks ridiculously fuzzy.

I find it hard to comment on the quality of the anti-aliasing here, mostly because in 4k the video in gameplay looks heavily compressed with all sort of tiled wavelet compression artifacts that have quite an impact on the overall image quality. I understand the role of anti-aliasing tech is to deliver image quality, but blaming the anti-aliasing for every reasons to why an image quality can be compromised seams a little bit unfair, don’t you think?

Also one last thing. It would be great if we could switch the unreal engine Anti-Aliasing pipeline to tensor cores, Xe cores, AMD matrix, for performance reasons. (A console command would allow devs to let players to decide to tap into unused GPU components if DLSS,XeSS, etc is not available).
TSR could run on tensor cores instead of the main graphical computing units providing the games main render frames.

Would be great! But we are limited to the APIs exposed to us: for instance DirectML that requires roundtrip to main memory between a programmed shader versus matrix multiplications which is not in the interest of performance. This is where each HIV have a lot less constrains compared to us because knows exactly what and how their hardware can do and exploit many advantages that are not standardized like avoiding roundtrip to main memory to squeeze as much as possible runtime GPU performance to invest even more on quality.

What we can do specifically on PS5 and XSX is that conveniently most of their AMD GPU is public ( https://www.amd.com/system/files/TechDocs/rdna2-shader-instruction-set-architecture.pdf ) , so we can go really deep into hardware details and imagine and experiments crazy ideas. And this is what TSR did in 5.1, it exploit performance characterists of RDNA’s 16bit instructions heavily which can have huge performance benefits. In UE5/Main for 5.3, added shader permutation ( https://github.com/EpicGames/UnrealEngine/commit/c83036de30e8ffb03abe9f9040fed899ecc94422 ) to finaly tap on these instructions exposed in standard HLSL in Shader model 6.2 ( 16 Bit Scalar Types · microsoft/DirectXShaderCompiler Wiki · GitHub ) and for instance on an AMD 5700 XT, the performance savings in TSR are identical to how much these consoles are optimised too:

LogRHI: 12.5% 2.77ms TemporalSuperResolution() 1857x721 -> 3039x1179 8 dispatches
LogRHI: 0.2% 0.04ms TSR ClearPrevTextures 1857x721 1 dispatch 117x46 groups
LogRHI: 0.1% 0.03ms TSR ForwardScatterDepth 1857x721 1 dispatch 233x91 groups
LogRHI: 0.7% 0.16ms TSR DilateVelocity(MotionBlurDirections=0 OutputIsMoving SubpixelDepth) 1857x721 1 dispatch 233x91 groups
LogRHI: 0.6% 0.13ms TSR DecimateHistory(ReprojectMoire) 1857x721 1 dispatch 233x91 groups
LogRHI: 8.4% 1.88ms TSR RejectShading(WaveSize=32 FlickeringFramePeriod=2.276193 ComposeTranslucency) 1857x721 1 dispatch 155x61 groups
LogRHI: 0.3% 0.07ms TSR SpatialAntiAliasing(Quality=1) 1857x721 1 dispatch 233x91 groups
LogRHI: 0.1% 0.03ms TSR FilterAntiAliasing 1857x721 1 dispatch 233x91 groups
LogRHI: 2.0% 0.44ms TSR UpdateHistory(Quality=High R11G11B10 OutputMip1) 3039x1179 1 dispatch 380x148 groups
AMD 5700 XT 16bit ops:

LogRHI: 6.5% 1.33ms TemporalSuperResolution() 1857x721 -> 3039x1179 8 dispatches
LogRHI: 0.2% 0.03ms TSR ClearPrevTextures 1857x721 1 dispatch 117x46 groups
LogRHI: 0.1% 0.03ms TSR ForwardScatterDepth 1857x721 1 dispatch 233x91 groups
LogRHI: 0.7% 0.15ms TSR DilateVelocity(MotionBlurDirections=0 OutputIsMoving SubpixelDepth) 1857x721 1 dispatch 233x91 groups
LogRHI: 0.7% 0.14ms TSR DecimateHistory(ReprojectMoire) 1857x721 1 dispatch 233x91 groups
LogRHI: 2.5% 0.51ms TSR RejectShading(WaveSize=32 FlickeringFramePeriod=2.425195 16bit ComposeTranslucency) 1857x721 1 dispatch 155x61 groups
LogRHI: 0.3% 0.07ms TSR SpatialAntiAliasing(Quality=1) 1857x721 1 dispatch 233x91 groups
LogRHI: 0.1% 0.03ms TSR FilterAntiAliasing 1857x721 1 dispatch 233x91 groups
LogRHI: 1.8% 0.37ms TSR UpdateHistory(Quality=High 16bit R11G11B10 OutputMip1) 3039x1179 1 dispatch 380x148 groups

What makes TSR in 5.1+ so different from 5.0 is in how many more convolutions it does using this exposed hardware capabilities. That RejectShading is doing like 15 convolutions in 5.1+, each 3x3 on a current hardware compared to 3 in 5.0, which allows to make TSR substentially smarter thanks to very neat discovered properties chaining some very particular convolutions do. And while this number of convolutions massively increased by a factor of 5, the runtime cost of this part of TSR didn’t change, and yet this gain in smartness of the algorithm allowed to cut significant amount of other costs that was no longer required in the rest of the TSR algorithm which is core reason behind this performance saving from 3.1ms to 1.5ms on these console. Sadly this expose hardware capabilities in standard HLSL are not benefiting all GPUs equally because how they decided to architure their hardware too.

We can’t do miracle using specificaly marketed hardware feature in most efficient maneur with what is only a subset of these exposed capability to us at the moment. But we can still do some surprising stuf on existing hardware wrongly assumed by the publicly uncapable of doing some particular things. And we are able to do it just with standard features, and better understanding of how the GPU works thanks for instance to that AMD pdf I linked above.

Even when we adventure ourselves into new creative way to use the GPU, we can hit all sorts of roadblocks, for instance shader compiler too. Some of these optimisations in TSR are pushing the shader compiler in such unexpected ways that they must be disabled on SPIRV platforms (for instance Metal and Vulkan) due to the shader compiler used there blocking the editor start up for tens of minutes ( https://github.com/EpicGames/UnrealEngine/commit/e123c0e4c1c428312550060961559eb77c147291 ).

Thanks a again for your continued feedback!

Guillaume.Abadie · April 21, 2023, 6:06pm

I’m really impressed to hear it used a TAA implementation. It looked amazing and I already mentioned I didn’t even like temporal AA!
Does the Valley of the Ancient have the exact Console Command Cvars used in that showcase from Epic? If so, I’ll take the time to download the 100GB monster demo lol just to get the temporal parameters.
"Lumen in the Land of Nanite” Release Request - #42 by Bladerskb ------

If Valley of the Ancient lacks that steller implementation used in the first demo (LLN) and if LLN If it can’t be released, I would appreciate a release of an in-depth documentation of all the project’s Temporal Console Cvars variables used. As well as the usf parameters?

The Lumen in the land of Nanite just used vanilla TAAU, the exact same implementation as in 5.0 Early-Access on its default settings.

Only r.Tonemapper.Sharpen=0.5 was set to mitigate the amount of nanite details loss we were getting due to how TAA works.

I don’t know much about the .usf’s, is there anything I would need to activate in that file of my engine or was this automatically turned on in 5.2 ? Niagara rocks and animals looked great in Land of Lumen(LLN). Even though it cut down on the Nanite detail it’s really great because it pretty much eliminated everything controversial about TAA.
It would be extremely great to provide that quality TAA in our projects.

AA_DYNAMIC_ANTIGHOST has been enabled by default in TAA since 2016 in UE 4.15.

Can you explain what low MP/s are and how it relates to TSR’s quality? I don’t remember coming across that in the documentations.

So this what that stat tsr coming in 5.2 and accompaniying doc will describe, but I atteched you here a PDF of the draft of the new 5.2 documetnation section that explains this part.

stat tsr doc draft.pdf (2.8 MB)

2: The person who was incharge of recording and uploading the Tekken 8 reveal trailer to Youtube was VASTLY better than the people recording and uploading the the recent/post TGA Tekken 8 trailers ever since. Maybe the first uploading used a video encoding less susceptible to Youtube compression?

But would reason 2 explain why the action looks so fuzzy? 4K on youtube is supposed to reduce that artifacting but it still looks bad?
4K on the Tekken 8 reveal trailer had almost zero youtube artifacting?

I truly appreciate your enthusiasm about image quality and I relate very much as it is literally my bread and butter. However I believe that speculating on why image compression may or may not show up on a online video isn’t the goal of this thread.

I would like to take this opportunity to make sure the goal of this forum thread remains as intended about healthy and respectful feedback on what you think of TSR, and most importantly where are the weaknesses that affect your project with concrete illustration to improve in future release of Unreal Engine.

TheKJ · April 22, 2023, 5:40am

The Lumen in the land of Nanite just used vanilla TAAU, the exact same implementation as in 5.0 Early-Access on its default settings.

Thank you for letting me know, I appreciate it very much. I will be working on downloading 5.0 Early-Access to obtain that implementation.

Only r.Tonemapper.Sharpen=0.5 was set to mitigate the amount of nanite details loss we were getting due to how TAA works.

It has been my experience that tone mapper works very well with temporal Anti-Aliasing Methods.

AA_DYNAMIC_ANTIGHOST has been enabled by default in TAA since 2016 in UE 4.15.

Thank you for letting me know, I wasn’t sure.

1080p, 100% ScreenPercentage, Epic TAA, 35-50fps, all other settings are low to rule out lumen noise (But even at 60fps this ghosting happens) 5.0 and 5.1 City Sample version.

This is why I find the Lumen in the Land of Nanite TAA implementation so shocking and why I wanted it. The screenshots don’t even show the insane amount of ghosting I get from TAA because of the screenshot delay. And this also happens in Valley of the Ancient with TAA.

This is why TSR’s quality is so important, because TSR doesn’t have this on my end. Only **that rotation smearing and issues I stated earlier **.
I can’t show my project for reasons I’m not allowed to say but I’m using City Sample because my project uses metahumans, has fast paced third person action, rotating camera angles, in our own city much like this example. The TAA and TSR issues are always present in every project on my end.

So this what that stat tsr coming in 5.2 and accompanying doc will describe, but I attached you here a PDF of the draft of the new 5.2 documentation section that explains this part.

Heck yeah man this was a great. Thank you for attaching that! I think this doc really shows how important dynamic resolution is was for the console ports TSR. Since frame rates’ can effects it so much.

-However I believe that speculating on why image compression may or may not show up on a online video isn’t the goal of this thread.

Yeah, I understand your point and your concern. I only brought up that because it might be related to AA. And the reason why I think that, is because those recent trailers look exactly like the AA quality that I get in-editor and compiled projects look like on my end.

-I would like to take this opportunity to make sure the goal of this forum thread remains as intended about healthy and respectful feedback on what you think of TSR, and most importantly where are the weaknesses that affect your project with concrete illustration to improve in future release of Unreal Engine.

Absolutely, and trying my best here to get down to the issues I’m encountering with UE5.
I cannot get those quality results from the three examples I gave which. Whether or not it’s the third person template, City Sample, or our own project.
Thank you for every reply you have given so far, our conversation will also help other people who will come across this thread.
I will definitely try to stay on topic.
If we can fix these issues in the already open content we have here like City Sample then we will have TSR improved. Unless I’m missing something?

JanHxH_de · April 24, 2023, 8:54am

Hi @Guillaume.Abadie,
this thread is a really good idea!
I am currently struggling in our production(non gaming) how to reproduce a nice anti aliasing in viewport.
When I use Movie Render Queue and TSR I get a nice AA with setting spatial and temporal sampling to a certain number. Try and Error.
But in the viewport or during play, my object, a car, is flickering all the time. So AA is not near as good as when using MRQ. We are shooting screenshots.

Is there a way how I can some brute force the AA like in MRQ? Or is MRQ using TSR in a different way like in viewport/play mode?

Tech Specs: Unreal 5.1, Lumen, currently no Nanite

My cvars are pretty overblown I think:
r.AntiAliasingMethod 4
r.PostProcessAAQuality 6
r.TemporalAACurrentFrameWeight 0.04
r.Tonemapper.Sharpen 0.2
r.TemporalAASamples 64
r.TSR.History.UpdateQuality 5
r.TSR.History.ScreenPercentage 300
r.TSR.RejectionAntiAliasingQuality 4
r.TSR.ShadingRejection.SpatialFilter 2
r.TSR.ShadingRejection.Flickering 1
r.TSR.Velocity.Extrapolation 1
r.TSR.Velocity.HoleFill 1
r.TSR.RejectionAntiAliasingQuality 2
r.TemporalAAPauseCorrect 0
r.TemporalAACatmullRom 1
r.Upscale.Quality 5
Cmd: r.ScreenPercentage 200

Guillaume.Abadie · April 24, 2023, 11:56pm

Thanks for your continued time on this thread, this is very well apreciated!

Thank you for letting me know, I appreciate it very much. I will be working on downloading 5.0 Early-Access to obtain that implementation.

The implementation of TAA/TAAU between 5.0 Early-Access and 5.2 has barely moved. Targetting last gen consoles, we are very much tied on how much we can improve due to its current performance on these platforms. Onlychanges in TAA are mostly performance optimisation related to mobile renderer, and the specifics of mobile hardware like prefering pixel shader than compute shader for Delta Color Compression that can possibly be unsupported with RHI’s Unordered Access View.

This is why I find the Lumen in the Land of Nanite TAA implementation so shocking and why I wanted it. The screenshots don’t even show the insane amount of ghosting I get from TAA because of the screenshot delay. And this also happens in Valley of the Ancient with TAA.

TAA is very limiting in therms of what content you can author because of the fundamental limitation it has on its runtime performance. So the content has to be build with the constrains of TAA in mind from the ground up to avoid the issues otherwise this can be very high amount of cost of changing existing content.

The two major goals should have in mind when authoring content for TAA are: make sure static objects are not drawing velocity as much as possible with the VisualizeMotionBlur and now the VisualizeTemporalUpscaler show flags, minimal amount noise in the rendered scene color with vis SceneColor command.

A nice read to have better understanding of how TAA function and therefor its limitation is the SIGGRAPH talk from original author: Advances in Real-Time Rendering in 3D Graphics and Games - SIGGRAPH 2014 (almost 9years old!)

If we can fix these issues in the already open content we have here like City Sample then we will have TSR improved. Unless I’m missing something?

There is other limitations that doesn’t necessarily show up on CitySample but that can show up in other content and this is what I’m really eager to hear from the community to make sure fixes and improvements are apropriately sorted and addressed!

Is there a way how I can some brute force the AA like in MRQ? Or is MRQ using TSR in a different way like in viewport/play mode?

Thanks for joining in @JanHxH_de ! I’ll share couples of caveats on the console variable you have below. But I want to first and foremost underline the most important caveat for “brute forcing” (or super-sampling to be exact like we do in MRQ): r.ScreenPercentage doesn’t apply to editor viewports!

Editor viewports’s screen percentage is instead instead configured in top left corner menu of the editor viewport. You can change the default for your projects in Edit > Project Settings > Editor > Performance.

Note that not everyone working on a project necessarily have the same GPU and might prefere different screen percentage settings in editor viewport, so you can also override the project settings in your editor preferences

More conveniently in //UE5/Main for 5.3, I changed the editor viewport screen percentage to be hopefully more intuitive to understand how it works:

Now more specifically about the rest of your cvars.

r.TemporalAACurrentFrameWeight 0.04
r.TemporalAASamples 64
r.TemporalAAPauseCorrect 0
r.TemporalAACatmullRom 1

These only apply to TAA (r.AntiAliasingMethod 2) and have no effects on TSR.

r.TSR.History.UpdateQuality 5

3 is the maximum.

r.TSR.History.ScreenPercentage 300

200 is the maximum . Beware there was a bug where some VFX looking fine at 100 could start to ghost at 200. This fix ( https://github.com/EpicGames/UnrealEngine/commit/9ccd56dfcc06852b5b89a81411972f81e3ac31e3 ) in 5.2 really hunted down all sort of issues could happen with >100.

r.TSR.RejectionAntiAliasingQuality 4

Beware this will be below set to 2, which is the maximum supported. I’ve not spent much time on the TSR’s built in spatial anti-aliaser since it’s for only one frame althrough there is certainely some improvements to have.

Also note that simply putting sg.AntiAliasingQuality 4 for cinematic quality by default maxes out many TSR cvars ( https://github.com/EpicGames/UnrealEngine/blob/ue5-main/Engine/Config/BaseScalability.ini#L87 )

Ligazetom · May 3, 2023, 1:54pm

I’m using 5.2 (few days old now) source build. I’ve migrated from 4.27 because I needed to add custom mesh passes and didn’t feel like converting all the 32bit stuff to 64 and someone already needed it so yay.

I’m really interested what has changed so much that TAA, which I’ve used previously is suddenly unusable. It’s not that there are some little issues, but it’s glitching all around the screen, at least the default values are making everything hideous.

Anyway since I was forced to use TSR I wanted to ask if there is any way of ignoring the upscaling penalty, or at least ask what is the major performance eater because the same scene in UE4 at 1440p with TAA was rendering at 120fps no prob and now TSR with backbuffer resolution set to 1440p is taking 9ms of frame time on 1080Ti. I can obviously make it a bit faster by changing history res and screen percentage, but isn’t there a way of using TSR as TAA without the upsampling etc.?

Just handle the jittering, velocity and all the TAA stuff and don’t bother with scaling because input and output resolution is the same.

What is the “default” approach for this? It’s just really a shame to optimize as much as I could just to get hit by TSR eating everything anyway and obviously it’s resolution bound because it changes drastically depending on resolution.

I’ve digged around the codebase and shaders a bit and I have very faint idea what happens how/where, but I don’t see why it should be such a performance hit if there is no upscaling going on.

After all it’s doing the same stuff that TAA does (all that comes with jittered sampling) with something extra right?

Guillaume.Abadie · May 5, 2023, 3:53pm

Hi @Ligazetom and welcome on this thread!

Jumping from 5.2 and 4.27 si quite a jump!

TAA between 4.27 and 5.2 has only receive very minor changes. As proof of how little has changed on the TAA shader itself, wanted to share with you a diff of the file. Sadly looks like it is not possible two files on github with a link, so instead please find here a diff of the shader file.

TAAShader4.27vs5.2.txt (6.9 KB)

You will see there it’s only minor addition like a pixel shader for mobile, some file struct changes.

Only major thing that changed is in the cvar control the quality. In UE4, sg.AntiAliasingQuality used to control r.PostProcessAAQuality which was 0 aand 1 was FXAA where as 2 and 3 where two quality of TAA. But now in UE5 this sg.AntiAliasngQuality instead controls a r.TemporalAA.Quality. If you want further details how to migrate some settings I recommend looking at the commit message of that change: https://github.com/EpicGames/UnrealEngine/commit/eadb45ef3718ce60c640d915ea1261c30dc6abce

Otherwise something worth noting is that TAA is now capable r.TemporalAA.Quality 0 and 1 to have a R11G11B10 history that saves memory bandwidth thanks to a tricked discovered in TSR development that is enabled by default with r.TemporalAA.R11G11B10History.

So what could actually have changed so much between 4.27 and 5.2 that disrupt your project? My though are on motion vectors that are important consequence on the quality of TAA and more specifically the setting that has changed with this change https://github.com/EpicGames/UnrealEngine/commit/e9b8605b3b2c9235072f67b27eb6c6ca68e31382 replacing the r.BasePassOutputsVelocity and r.DepthPassMergedWithVelocity with a single r.VelocityOutputPass.

The 1080 TI was a great GPU! And is not that far off to current gen consoles! However TSR’s goal was to accomodate performance for other rendering feature of UE5 by allowing low rendering resolution, so it is designed and optimised from the ground up for the rendering resolution to be lower than the display resolution. It can run at whatever rendering and display resolution. Thing is to allow rendering a much lower resolution, it meant investing runtime performance in its quality to hide better the lower rendering resolution than TAA. So while both are Temporal Upscalers, TSR is doing a looooot more than TAA morivated for quality reason.

Interestingly the cost of TSR scales with the rendering resolution unlike TAAU. So as the rendering resolution lowers, it means more work for TSR to maintain image quality, but interestingly it’s runtime costs become cheapers per frames. We explain this a little bit better in upcoming 5.2 TSR documentation.

TSR cost scaling.pdf (302.8 KB)

Now TSR goal was to targetting current gen console but also make sure it future proof for possibly more powerful hardware too. So for instance Fortnite using TSR on PS5 and XSX is using the Medium anti-aliasing scalability (sg.AntiAliasingQuality=1) while XSS uses sg.AntiAliasingQuality=0. In 5.3, I’ve added shader permutation to also uses 16bit instruction in D3D12 for Windows like we do on console ( https://github.com/EpicGames/UnrealEngine/commit/c83036de30e8ffb03abe9f9040fed899ecc94422 ), and while your 1080 TI is comparable to a console, it sadly doesn’t support 16bit VALU instructions which TSR heavily makes uses of to optimise its performance on current gen consoles.
So this is why sg.AntiAliasingQuality=0 might actually be most fitted for your GPU. Here is how to configure it in editor:

One way to find out what goes into the cost of TSR is using ProfileGPU in the console with r.ProfileGPU.Root TemporalSuperResolution and here you can see the cost of each TSR passes, for instanec

Epic Anti-Aliasing Quality:

LogRHI:        9.5% 0.64ms   TemporalSuperResolution() 1739x748 -> 2786x1198 9 dispatches
LogRHI:           0.1% 0.01ms   TSR ClearPrevTextures 1739x748 1 dispatch 109x47 groups
LogRHI:           0.2% 0.01ms   TSR ForwardScatterDepth 1739x748 1 dispatch 218x94 groups
LogRHI:           0.5% 0.03ms   TSR DilateVelocity(MotionBlurDirections=0 OutputIsMoving SubpixelDepth) 1739x748 1 dispatch 218x94 groups
LogRHI:           0.4% 0.03ms   TSR DecimateHistory(ReprojectMoire) 1739x748 1 dispatch 218x94 groups
LogRHI:           1.9% 0.13ms   TSR RejectShading(WaveSize=32 FlickeringFramePeriod=3.000000  ComposeTranslucency) 1739x748 1 dispatch 145x63 groups
LogRHI:           0.4% 0.02ms   TSR SpatialAntiAliasing(Quality=2) 1739x748 1 dispatch 218x94 groups
LogRHI:           0.2% 0.01ms   TSR FilterAntiAliasing 1739x748 1 dispatch 218x94 groups
LogRHI:           4.5% 0.30ms   TSR UpdateHistory(Quality=Epic R11G11B10) 5572x2396 1 dispatch 697x300 groups
LogRHI:           1.2% 0.08ms   TSR ResolveHistory 2786x1198 1 dispatch 349x150 groups

Low Anti-Aliasing Quality:

LogRHI:        3.6% 0.23ms   TemporalSuperResolution() 1739x748 -> 2786x1198 6 dispatches
LogRHI:           0.1% 0.01ms   TSR ClearPrevTextures 1739x748 1 dispatch 109x47 groups
LogRHI:           0.2% 0.01ms   TSR ForwardScatterDepth 1739x748 1 dispatch 218x94 groups
LogRHI:           0.5% 0.03ms   TSR DilateVelocity(MotionBlurDirections=0 SubpixelDepth) 1739x748 1 dispatch 218x94 groups
LogRHI:           0.4% 0.03ms   TSR DecimateHistory() 1739x748 1 dispatch 218x94 groups
LogRHI:           1.3% 0.08ms   TSR RejectShading(WaveSize=32 FlickeringFramePeriod=0.000000  ComposeTranslucency) 1739x748 1 dispatch 145x63 groups
LogRHI:           1.0% 0.06ms   TSR UpdateHistory(Quality=Low R11G11B10 OutputMip1) 2786x1198 1 dispatch 349x150 groups

TheKJ · May 6, 2023, 8:42am

uses 16bit instruction in D3D12 for Windows like we do on console

Like RNDA2 gpus right? If someone wanted UE5 games to run the best on there pc, and thought to equip it with a RDNA2 gpu, would the engine actually check the system and use a more optimized pipeline(The next gen console pipeline) for that RNDA2 gpu?

Also a concern of mine, if this is true. Is there a list of GPUS that have this 16bit instruction? I have a 3060 and fortnite 5.1 runs 42-47fps on native 1080p, low textures, high settings with high TSR. 50-53fps with no AA

Pretty comparable with the City Sample with no AI Tbh. The “less” powerfully RNDA2 AMD cards seem to run it much better as same resolution and settings(maybe even higher settings).

It seems to me that when targeting a card recommended resolution (1080p for A770, 3060) more optimization and cuts need to be made for those brands. If so, is there a “get boolean value” C++/BP function to check if the more optimized pipeline is being utilized so we can turn down more expensive stuff(like material parameters, Cvars for quality) as the game boots up?

Thanks for the attachments and check your private messages on here. I sent you a messages about an odd thing I had with TAA.

Ligazetom · May 10, 2023, 12:03pm

That ProfileGPU command is exactly what I wanted to know about! Thank you so much. I know there isn’t much I can do to change the behavior, but even knowledge of which part of the AA is the heaviest is good thing to know.

I mean I’m developing it on 1080Ti but we have a client with 4090 (laptop) and native 4K is still eating 90% performance in main menu. I understand that TSR is different beast from TAA, so that’s why I asked if there is not some “switch” to focus only on the “temporal” part if the resolution is the same.

In 5.3, I’ve added shader permutation to also uses 16bit instruction in D3D12 for Windows

So the way I understand it atm.: pc version is not using the 16 bit instructions in 5.2 and even that is only supported in DX12 right? I’m asking because we are still on DX11 and SM5. So is DX12 required to get reasonable TSR performance? (the buggy slate api rework really slowed migration a lot so I haven’t had time to migrate stuff that is not throwing errors or deprecations warnings at me :D).

So the valid usage of TSR is to always render at lower res since it should look the same as native, but performance won’t tank? Because that’s what eludes me all this time. If I can render native 4K faster with TAA, why bother with TSR (ignoring all temporal related stuff that has been improved since that could be done for TAA as well), but if it is just as you say about DX12, 16bit stuff etc., then I understand I guess. The thing is I don’t want to upgrade to newest generation gpus so I can optimize it for the lower end first, but if there is “radical” difference and not just “MORE FASTER BIGGER”, then it’s different story.

I’ve read all there is about the TSR I could find (not much), and am a little bit sad there isn’t some technical paper about it, since there are a lot of concepts hidden in passes that even if I see their inputs/outputs with renderdoc I can only imagine what they are actually doing (oh grand reprojection and its mysteries) and even the most basic stuff like how many frames of history are we storing etc. I’m obviously out of my depth here without any graphics programmer background, but I’ve spent last 6 months adjusting the engine so I could render “movies” on image planes without ghosting (custom non-jittered depth bass, custom transparency pass, blending, custom SSR depth for transparent objects, tonemapping, extended gbuffer, customized basspass shader), but boy would it be easier if I knew that editing rejection shading texture was invalid approach, or that first frame of standalone apps is always “FamilyViewTexture” for some reason or how many history frames there are stored so I could play around with ghosts generated because of it, or why tonemapping is actually dependent on the scene texture and generates different LUT accordingly (this is just a guess but there is some trickery :D).

With all that said, I’m basically just asking, if there are plans to write the documentation only about configuration stuff or also a little bit about what is actually happening, because as someone who searched for “UAVs” and got mostly pictures of drones, I’d very much appreciate every bit of info you can spare on “why” opposed to “this is how you do it”.

Anyway, thank you for your answer and looking forward to 5.3 branch on github

Edit: Reading your response made me realize I need to rework part of our custom rendering pipeline to force rendering at higher res because we are not passing everything through the TSR, but using ISpatialUpscaler of ViewFamily. That are some really sad news for me and my next few weeks/months thank you

TheKJ · May 11, 2023, 2:53pm

why bother with TSR (ignoring all temporal related stuff that has been improved since that could be done for TAA as well)

This touches on something I said in my last post. Because Nanite and Lumen is so expensive(referring to Fortnite and matrix city performance) UE5 needed an upscaler to achieve 60+fps on appropriate resolution/hardware .

I’m not a fan of using upscalers to reach target performance unless it’s 4k which seems to be the best use case for TSR or other upscalers(I’m stuck with 1080p-RTX3060 atm).

Hopefully Lumen and Nanite will continue receive huge performance upgrades, that really needs to be the focus now so our 1080p customers don’t have to rely on TSR or other upscalers to get 60fps on high settings since its not sharp at all at those resolutions.

You said you were rendering “Movies” but I’m more speaking about real time gameplay.

EDIT: @Guillaume.Abadie I heard the matrix awakens was locked at 30fps on console, but how was it 4K60fps on the video presentation on youtube?

Guillaume.Abadie · May 11, 2023, 6:29pm

Hello!

Thanks for your continued interest on this thread!

Like RNDA2 gpus right? If someone wanted UE5 games to run the best on there pc, and thought to equip it with a RDNA2 gpu, would the engine actually check the system and use a more optimized pipeline(The next gen console pipeline) for that RNDA2 gpu?

Amount of optimisations we can offer on PC is directly limited to what is exposed to us in D3D specifications and also what the drivers says it can do. For 16bit ops in 4.3’s TSR, it’s directly based of D3D12_FEATURE_DATA_D3D12_OPTIONS4::Native16BitShaderOpsSupported. And the draw events of TSR in ProfileGPU or DumpGPU will says which TSR shader is taking advantage of them:

LogRHI: 8.4% 1.88ms TSR RejectShading(WaveSize=32 FlickeringFramePeriod=2.276193 ComposeTranslucency) 1857x721 1 dispatch 155x61 groups

LogRHI: 2.5% 0.51ms TSR RejectShading(WaveSize=32 FlickeringFramePeriod=2.425195 16bit ComposeTranslucency) 1857x721 1 dispatch 155x61 groups

So much this will saves in the runtime cost is largely if the drivers says it supports, but also how the hardware is capable to take advantage of this optimisation or not. 16bit often saves register pressure that when register bound can saves up to 2x performance improvement, but for instance RDNA GPUs also have the packed instructions like v_pack_mul_f16 capable to two two multiplications of the price of 1 which is another 2x. So that is the use of 16bit instruction on that shader can almost do a x4 perf improvement.

Also a concern of mine, if this is true. Is there a list of GPUS that have this 16bit instruction? I have a 3060 and fortnite 5.1 runs 42-47fps on native 1080p, low textures, high settings with high TSR. 50-53fps with no AA

Frame rate are not a great measurement of the cost of something due to it being inverse propersional to actual runtime costs. Whereas milliseconds are proportional to GPU runtime costs. It’s hard to comment on your frame rate as is, since so many things can be at play when it comes frame rate. This is where stat gpu, ProfileGPU and stat tsr (or stat unit in version before that) will give you further insight into where the GPU runtime costs goes for you scene on your GPU.

It seems to me that when targeting a card recommended resolution (1080p for A770, 3060) more optimization and cuts need to be made for those brands.

That is just not based of runtime cost of TSR, but also with it’s input feed shown in stat tsr in megapixel/s that I explained earlier that is also important metric that directly influence decent displayed image quality.

So the way I understand it atm.: pc version is not using the 16 bit instructions in 5.2 and even that is only supported in DX12 right? I’m asking because we are still on DX11 and SM5. So is DX12 required to get reasonable TSR performance? (the buggy slate api rework really slowed migration a lot so I haven’t had time to migrate stuff that is not throwing errors or deprecations warnings at me :D).

Yes that is right, 16bit ops are in 5.3 and currently only on D3D12. It’s not possible to query the driver in D3D11 to know whether 16bit ops are available. Also to be able to compile HLSL 2021 with Microsoft DXC’s on D3D11, we have to do a crazy shader compiler pipeline that compile TSR shaders as such: TSR shader in HLSL 2021 -(DXC)> Vulkan SPIRV -(SPIRV-Cross)> HLSL 2018 -FXC> D3D11 Bytecode. Problems is DXC’s SPIRV backend currently is hitting multiple minutes long compilation time of TSR shaders due to their complexity. So we had to disable this optimisations where the 16bits are the most important too. Maintenance of TSR on D3D11 is becoming ever increasingly dificult overall and I would not be surprised at some point the support of D3D11 will have top be abandonned for TSR.

So the valid usage of TSR is to always render at lower res since it should look the same as native, but performance won’t tank? Because that’s what eludes me all this time. If I can render native 4K faster with TAA, why bother with TSR (ignoring all temporal related stuff that has been improved since that could be done for TAA as well), but if it is just as you say about DX12, 16bit stuff etc., then I understand I guess. The thing is I don’t want to upgrade to newest generation gpus so I can optimize it for the lower end first, but if there is “radical” difference and not just “MORE FASTER BIGGER”, then it’s different story.

If the quality difference between TSR and TAA are not obvious in your content then maybe perhaps TSR isn’t the right fit for your project if this is only slower. Thaw maybe one of your players will have different opinion too. That is why in Fortnite we expose the many anti-aliasing settings possible.

With all that said, I’m basically just asking, if there are plans to write the documentation only about configuration stuff or also a little bit about what is actually happening, because as someone who searched for “UAVs” and got mostly pictures of drones, I’d very much appreciate every bit of info you can spare on “why” opposed to “this is how you do it”.

I would love to publish more technical details of how TSR works, and tried multiple times! But with all the great rendering tech coming in 5.0, TSR publications had to be abandoned two year in row because there was just to much great tech to talk about already in unreal engine. Adds to that how the pandemic has impacted personal lives and you end up where we are today.

EDIT: @Guillaume.Abadie I heard the matrix awakens was locked at 30fps on console, but how was it 4K60fps on the video presentation on youtube?

Yes The Matrix Awakens was 30hz. I imagine the youtube ended up 60hz because this is the refresh rate of HDMI cable this was recording from regardless of the refresh rate of the actual game, and recording only odd or even or even average of both frame from HDMI cable could lead to issues whether game present odd or even frames based of when a frame was completed a bit late. So in a sence recording at HDMI 60hz refresh rate even on lower frame rate game is the most throughul option to make sure the fidelity of the experience of game fluidity, when frames are display to users identically like they would experience branching their console directly on their TV, and this also include showing in the video when frames were completed bit late by the console and changing from odd to even frame presenting (and vice verso). In a way it is a consequence of us not wanting to cheat the fluidity experience through editing the video after recording.

Please note the updated TSR doc is now live:

TheKJ · May 11, 2023, 10:33pm

Oh my god bro, YES!
Glad to have those finally up!
I cannot wait to read them. Thanks again for reply to my questions!

Frame rate are not a great measurement of the cost of something due to it being inverse propersional to actual runtime costs.

Yeah I heard another person from Epic a year ago talk about that, but if I had the Stat GPU and profiler on fortnite to show the ms, I absolutely would have given that. I’m not sure what’s the lowest hardware the fortnite devs use but I wanted to put that out there.

Thanks again for pointing out the LogRHI:

D3D11 will have top be abandonned for TSR.

Yeah, I feel that coming. UE5 really feels like a push for next gen RT cards which isn’t a bad thing. Nanite is actually insane performance wise given what is does, and iGPUs can’t do what is needs. I think D3D11 won’t be lasting much longer for UE5 at least for mid-intensive graphics. UE5 depending on DX12 seems pretty viable for the future given the amount of sufficient affordable hardware. So if improving any aspect of TSR means bye bye to DX11 then I’m all for it.