I’m developing a game for the Meta Quest and I’m struggling to find a good anti-aliasing solution. I’ve attached a video comparing two methods: TAA (Temporal Anti-Aliasing) on the left and MSAA (4x Multi-Sample Anti-Aliasing) on the right.
Here’s the problem:
TAA:
While it significantly reduces aliasing, the overall image in VR is extremely blurry. The video doesn’t fully capture the severity, but in the headset, it’s very noticeable and uncomfortable.
Also, it causes a significant drop in FPS, which is critical for VR performance.
MSAA (4x):
The FPS is much better and the image appears sharper.
However, aliasing is very pronounced, with severe shimmering and glistening on edges, especially noticeable in VR. Again, the video doesn’t fully show the extent of this issue.
I have attached a link to the project setting file as reference.
In summary: I’m stuck between a blurry image with TAA and a shimmering, aliased image with MSAA. Neither is acceptable for a comfortable VR experience.
My questions are:
Has anyone else encountered similar issues with anti-aliasing on the Meta Quest?
Are there any recommended settings or techniques for achieving a balance between image clarity and performance?
Are there other anti aliasing techniques that are better suited for the meta quest?
Are there any post processing effects that can help remedy these issues?
I’ve spent a lot of time troubleshooting, but I’m at a loss. Any help or suggestions would be greatly appreciated!
The recommendation from most VR platform vendors has AFAIK always been to use MSAA when possible.
Since the shimmering seems to be limited to the edges of translucent foliage, it’s likely caused by the Alpha-to-Coverage feature not working. I’m not sure if it’s supported by the mobile renderer though, and googling it only seems to turn up others with the same issue.
I feel like that’s an understatement, the aliasing in VR with MSAA using unreal is insane. I don’t know why there is not more people talking about this.
Should we be forced to use TAA over MSAA when in-fact MSAA is the recommended method?
@VictorLerp Are there any upcoming changes that fixes aliasing? It doesn’t seem like Unity is suffering that much from this as it does in unreal
Alpha to coverage is enabled, which in by itself just created severe banding on the masked material. If we sharpen it according to Anti-aliased Alpha Test: The Esoteric Alpha To Coverage | by Ben Golus | Medium it gets sharpened but still flickers. Is MSAA even triggered when turning on alpha to coverage, it certainly doesn’t feel like so. Our foliage is flickering like crazy when on distance, even mip biasing down doesn’t help, it will still flicker.
Getting transparency banding means that the MSAA is working at least. The sharpening is good for when you have a small texture resolution relative to the view (like when you show your face into a bush), but may also make the aliasing worse at distance if it shrinks the gradient too much.
Testing this on PC with the default 5.6.0 VR template, and I am also experiencing more flickering than there should be. The higher the MSAA level the less it flickers, and looking at frames with RenderDoc it seems to render and resolve the MSAA correctly per-frame.
The effect seems to only affect the VR render. The Editor viewport is practically flicker-free, even with the camera moving. Setting the VR headset down on a stable surface shows that it still flickers, and turning off MSAA makes the flicker apparent on all geometry.
I’m not sure, but I suspect this issue isn’t related to MSAA or AtoC, just that they are only partially successful in mitigating it. Maybe it could be a TAA subpixel jitter function that gets left on for the VR render or something similar.
It’s possible that my issue is separate from yours though. It’s hard to say for sure without seeing it in action.
No I think you’re on to something there. I can’t verify with RenderDoc as it seems to be broken on 5.5.4 which is the current version we are on atm. Engine crashes as soon as I enabled RenderDoc
We’re using the sharpening on foliage as the apparent banding will remove geometry from the small foliage leaves/spruces. But even with sharpening the MSAA resolve should be applied for this material but it still flickers. As you say, if I turn off AA all geometry will alias quite bad, even opaque so I know MSAA is doing part of its job.
But there is something else thats causing unbearable flickering and it seems this is only an issue from Unreal VR.
Would be nice to have some more eyes on this with tests to see what is actually going on. Because this effectively stopping us from using masked materials at all in our shipped game. This increases overdraw, we have self sorting issues and its looking quite blurry but translucent materials atleast dont alias as bad.
The editor viewport is flicker free yes
@VictorLerp Do you have any information on this, any clues if you know about an upcoming fix? Because this definately seems like an Unreal Engine bug for the VR renderer
@Rectus_SA Could you be a absolute lad and try that CL? I never cloned the repo but downloaded from the launcher. Cloning a new repo would take to much disk space on my machine currently.
@VictorLerp I do want to clarify that we’re running forward renderer and this issue has been going on from atleast 5.3 which is around the time I started to do R&D for why we have so much aliasing on non-translucent objects. 5.4 is where I noticed alpha to coverage started to work but it still creates an aliased mess of the scene with masked materials. We are currently on 5.5.4 so if the bug you’re linking is introduced in 5.6 it has nothing to do with the issue I’m describing. Sure it could be the issue Rectus has, if we are suffering from different aliasing issues.
So this is not something new, however let’s first try this CL and see if there’s any improvement. I hope Rectus or someone else can test it.
Then this issue should be investigated. If you get more aliasing with default VR template than there should be something is off. As Recus says it’s only affecting the VR render as the editor viewport is flicker free, even with camera moving.
Maybe as he said, subpixel jitter function by TAA or something similar? We can’t use masked materials at all, even with alpha to coverage enabled, even if we sharpen alpha to coverage. Mip biasing doesn’t work, no matter how aggressive you are, it still flickers. This effectively forces us to use translucent for everything which is just right out bad.
Yeah, getting the same effect in ue5-main. I also noticed that adding some swaying with World Position Offset masks the effect somewhat, probably because it reads as actual movement.
I looked at it a bit more by hooking up the Meta XR Simulator. The aliasing is not visible when not actively moving, so it must be the minute tracking variations triggering it. Playing around with the materials, I can get the aliasing down a bit, but it is still much larger than any geometric aliasing, and increasing the MSAA level doesn’t help it much.
I also noticed that the wide custom MSAA resolve options using r.WideCustomResolve breaks rendering when using the default settings. When enabled together, the Separate Translucency pass will clobber the output and draw a black screen, both with 5.6.0 and ue5-main. Unreal seems to already use a custom MSAA resolve when HDR is enabled, so wide resolves could be a cheap quality improvement.
I’m not sure anymore, to be honest, it being temporally stable on a fully static render might point against it. I’ve noticed it’s hard to get good MSAA gradients on edges with most pixels ending up being either fully transparent or opaque. It could be due to my top level mip being too high res and sharp, but forcing smaller mips still results in aliasing.
It’s really hard to say, but it could be something with the coverage maps, or mip calculation (if Unreal isn’t using the default hardware based ones).
The effects might also be different if you are using the mobile renderer. I’ve only tested with the default VR template setup for Windows.
I don’t see aliasing when I use VR preview I couldn’t see much aliasing in the metaxr simulator either. So for me I notice it on target platform (quest3) which is the mobile renderer.