I’m working on a 2.5D game where characters are based on billboarded sprites. The animations are played by flipbooks using Paper2D/PaperZD. All in a pixel artstyle, in a 3D world.
The problem is that I obviously need some form of anti-aliasing. FXAA is simply bad, and TAA/TSR create heavy ghosting which completely ruins the animations. SMAA would be perfect, I think, but we don’t have that option here.
Are there any tricks or alternatives? Cause unless I’m misunderstanding something, rendering high quality pixel art animations with AA is straight up impossible in UE. They did it in Octopath Traveler/Triangle Strategy somehow, though.
I’ve tweaked every possible TAA cvar, and while some of them really help, they come with their own set of problems which end up ruining the AA for everything else.
The perfect thing would be to be able to disable AA for flipbooks, which is probably not possible, or render them separately, which sounds incredibly expensive.
Upscaling somewhat solves this issue. However, it seems like you’ve tried a lot of various things to achieve the result you are looking for. You lose a lot of functionality by disabling Temporal AA. So I would leave that on and instead focus on removing the ghosting, which is done by disabling motion blur. You can always implement your own aliasing method, however I wouldn’t recommend that since it is probably rather difficult.
Do you really need deferred rendering? Because if not you could probably use the forward renderer which will allow you to use MSAA
Oh yeah, you’re right!! I got MSAA mixed in my head with SMAA as usual, but yeah, I think switching to forward would be a great alternative. I was kinda excited to use Lumen though, so I’ll have to do some tests and weigh the pros and cons.
Edit: Tried it in a new project and it’s working. I must’ve been unlucky.
Last time I tried though, I was getting some complicated bugs with forward + MSAA. Whenever I enabled both the engine would crash and not boot back up. After switching to DX11, it would at least launch but having MSAA on would break the whole rendering. It’s an annoying little detail that will be a bit annoying to fix, as I’ve seen this bug reported before but without any definite fixes. I’ll try to get it working though!
Upscaling somewhat solves this issue. However, it seems like you’ve tried a lot of various things to achieve the result you are looking for. You lose a lot of functionality by disabling Temporal AA. So I would leave that on and instead focus on removing the ghosting, which is done by disabling motion blur. You can always implement your own aliasing method, however I wouldn’t recommend that since it is probably rather difficult.
I don’t have the skills to write my own AA solution indeed hahaha, but you gave me a nice idea that involves modifying UE5’s TAA shader. If I could just read from a custom stencil buffer, I could change the TAA weight inside the flipbook to a higher value like 1 and that would fix it. Sounds too easy to be true though, it might introduce some artifacts that will be harder to fix. But I’d love to try it anyway. I can’t find any information on modifying Unreal’s shaders however, so I have no idea how to include a reference to a custom stencil or another render texture inside of it.
Do you guys have experience with modifying source shaders?
If you want to modify the shader files, you can find them here: Program Files\Epic Games\UE_5.0\Engine\Shaders. Simply open the files inside of a text editor and it will work. When it comes to stencils, that needs to be enabled in the settings. You can then use the scene texture node inside to grab the stencil, and use the scene depth to mask out the subject. Here is a tutorial for the basics: How to Create Masks With the Custom Stencil Buffer | Tips & Tricks | Unreal Engine - YouTube
Thanks a lot!! What I’m confused about is how to access the stencil inside the global shader. You mention “you can then use the scene texture node inside to grab the stencil”, but since it’s not a material, I have no idea how to actually do that hahah. Is there a function for global shaders similar to the material node, or does it require a custom setup in TemporalAA.cpp? Thanks!