Why does ProcessScreenshots write all alpha to 255? And, why is alpha inverted otherwise?

Development Rendering
, ,
1

I’ve been experimenting with how to create high res screenshots with a transparent background. I’ve found out how to do it, but my method has created problems for colleagues who were using other screenshot functionality and so I clearly need a different solution.

I’m interested to know what is really going on here and how to best create transparent screenshots without knock on effects.

Essentially I enabled Alpha Output and worked to ensure that my post process materials and tonemapper replacing materials were propogating alpha through.

I found that it all worked right up until the end where in UGameViewportClient::ProcessScreenshots() ~line 2182 there was code where right before passing the screenshot data to a delegate, it iterated through everything and set the alpha to 255! No wonder the transparent background disappeared!

So I commented this code out.

Then I found that I was generating an image with inverted alpha, where the main object was transparent, and background fully opaque black.

I had no idea why this was the case but as an experiment in my post process tonemapper material I simply inverted the alpha and then suddenly everything worked perfectly.

This would all be fine except that my changes to the engine, commenting out this stomping of alpha, has caused unintentional side effects for other colleagues doing unrelated screenshot work. Whoops.

So why is alpha inverted here, why is it being stomped with 255 to make it opaque and what is the real fix for this problem?

I feel I’m doing kludges on kludges here and I’m not going to have a good outcome until I understand why this is setup the way it is with inverted alpha.

(As an aside, I tried an idea of keeping the engine change commented out and having my screenshot delegate invert the alpha in code instead of the tonemapper material but this didn’t work well with some other post process 2D texture overlays, which unintentionally had their alpha inverted too. So I discovered that the objects in the scene need to have their alpha fixed up prior to post process work of 2D textures being applied over the camera. This is why my earlier fix of adding the invert alpha into the tone mapper worked.)

(edit: I’m aware that there is an option to create screenshots with transparent backgrounds using depth mask but we are unsatisfied with this solution due to some visual artifact issues and that’s why we’re interested in other approaches)

2

Steps to Reproduce
Comment out the following code at 2182 of GameViewportClient.cpp and take a screenshot where options to Output Alpha and create a transparent background have been set.

for (auto& Color : Bitmap)
{
    Color.A = 255;
}
3

Hi Travis,

I checked in with the team that made this change, and it looks like this change was made as a part of fixing up the bug related to the rendering of the UI while also taking a screenshot, which is not completely relevant anymore. If you are interested in proposing an improvement to the system that would allow for respecting transparency while taking high-resolution screenshots, the team would be happy to take on a pull request through GitHub. Please let me know if this is an option for you.

4

I think it would make a lot of sense to remove this part where we stomp the alpha to 255 uniformly, but there would still remain the issue that for some reason alpha is inverted. I don’t know why that is the case or where best to make a change to fix that problem.

5

Yeah, that is a fair point. Would you happen to have a short set of steps for me to reproduce this on my end? I can then try to track down why we currently need to invert the alpha

6

Sure.

Step one is to make a simple level of something you can take a screenshot of. A cube and a directional light is enough. Then open up the level blueprint and have the Event BeginPlay connect to a Take High Res Screenshot node.

Now go to Project Settings and search for “alpha” and find Engine - Rendering - Alpha Output and mark that checked. This will propogate alpha through the post processor.

The goal here is to have a cube with a transparent background. Since we have no background and we’ve selected Alpha Output in the settings I would expect that we should see this however we don’t.

If we dig through the code we can see that when we take a screenshot then at GameViewportClient.cpp line 2034 we enter ProcessScreenshotData and then into GetHighResScreenshotConfig().MergeMaskIntoAlpha(Bitmap, FIntRect(0, 0, 0, 0));.

Since we are not going to be using the mask feature we eventually enter line 188 of HighResScreenshot.cpp

// Ensure that all pixels' alpha is set to 255
for (auto& Color : InBitmap)
{
	Color.A = TChannelType(255) * AlphaMultipler;
}

Here we see why we have an opaque image, which is that we’re setting alpha of everything to 255.

If we comment this out then we will get a curious image where the cube is transparent with alpha of 0 and still have a background. We can fix this and get the image we would expect if we invert the alpha.

for (auto& Color : InBitmap)
{
	Color.A = 255 - Color.A;
}

As mentioned before unfortunately this little hack of inverting alpha at the last minute just before a screenshot is taken is not good enough to genuinely fix the problem, as any sort of other screenspace elements added during the process will be incorrectly shown with inverted alpha.

It seems like the process to create the screenshot of the objects in the scene creates a result with an inverted alpha and there would need to be a fix up after that point but prior to other screenspace elements being added to the picture.

I noticed this because I had scripted the game to take screenshots using a world space CameraActor that was adding a texture overlay as a post process. The only way it looked right was to invert alpha as part of a post process in the scene.

7

Thanks for the extra repro steps. I confirmed the issue on my end, but I did not figure out a good way to properly invert the alpha when you take the high-resolution screenshot. I have escalated the case to a subject matter expert on the Rendering team, who will look at this early next week. If you have any questions, please don’t hesitate to contact us.

8

Hi Tavis,

So it’s been a few years since I last looked at this issue, but essentially iirc the HighResScreenshot was returning 0 in the alpha channel but only in some circumstances, which is why we stomped the 255 to make sure all cases were caught. Tbh it was a bit of a lazy solution on my part that I applied when attempting to fix a different issue. Could you please share some screenshots of what you and your colleagues are experiencing along with indicators of what settings are causing which image? This would really help me get my head around the problem before I repro and figure out the fix.

Thanks!

Jon

9

Here’s a screenshot of what a basic lit cube against a transparent background should ideally look like. In order to create this I had to add a post process volume and a material that simply inverted the alpha.[Image Removed]Here is my simple material [Image Removed]Now if I instead do not invert but directly set the alpha (or delete the post process volume) then the outputted image looks like this:

[Image Removed]Now the background is fully opaque and the cube itself is transparent, with just a thin rim of colour left from I guess some blending.

10

An added thing, in discussion with a colleague, he mentioned that they were aware of a r.AlphaInvertPass setting which could be helpful. Looking at the code this seems like it would add an extra inverse pass.

When I tried it, and the code only runs on the mobile rendering path, it instantly crashes.

[2025.06.05-22.29.58:732][596]LogRHI: Error: Failed to create graphics pipeline, hashes: Vertex: 4181D2545C52B4348E19D1C12C4CCFA36B67FC5D, Pixel: 741BA0E52EEC9C036115AD8A326C794A8C0D6A97, Pipeline: A7843F722B56456FC6CEFDF0FF1F9360BF2265A3.When I mentioned this to my colleague it reminded him that that happened to him to.

So I dunno. Maybe at some point this was added for some reason though not sure if it works anymore or how to get it working.

11

Hi Tavis, apologies again for the delay in response, had some hard deadlines this last week.

I’m afraid I’ve not been able to reproduce the issue but I am anticipating I’m doing something incorrectly in the repro, this is with screenshot or high res screen shot.

Could you please share your test project with me instead? If you go to file->zip project that’s the easiest way to share it.

Thanks,

Jon

12

I’ve added a project. There are two levels, one with a background and one without (where the hope would be to create a transparent background).

When you press play the level blueprint will take a highres screenshot. It should have inverted alpha. So for example the one with a background should appear entirely transparent.

In the level blueprint you can set InvertAlpha boolean to be true which will load in a sublevel that is a post process volume that inverts the alpha and fixes things.

Using that checkbox you should be able to create the desired image, of a box on a transparent background.

[Image Removed]

The key to all this, and I’m not sure if it’ll carry over in this zipped project, is that you need to comment out some code in Unreal.

As mentioned before at line 188 of HighResScreenshot.cpp there is a block of code that stomps all the alpha to 255 and that needs to be commented out.

// Ensure that all pixels' alpha is set to 255
for (auto& Color : InBitmap)
{
	Color.A = TChannelType(255) * AlphaMultipler;
}

So underlying question here is:

1) Is there an appropriate way to create transparent backgrounds that is less hacky? While it does feel deeply weird for Unreal to be stomping on the alpha, my fiddling of the Unreal code doesn’t feel like a good solution. Is there a way to more elegantly enable and disable alpha and to not have to resort to a post process volume to correct for the inverted alpha?

2) Why is the alpha inverted and such stomping of the alpha required? That’s interesting to me.

As it stands while I do have a work around, this postprocess volume approach is not ideal and it’s causing problems for other team members and breaking their work.

13

Hi Tavis,

Sorry for the delay yet again, we have Epic summer break now and I had a load of deadlines to wrap up, apologies for not responding to this sooner as a result but I’ve investigated this thoroughly now and I believe I have the solution required.

So to start off: I concur with all of your findings here, I can see how the data is coming back from the Viewport on readback, and how we are manipulating it. I think your concern about the fix being hacky is probably not something to worry about because the code itself appears to be a Frankenstein’s monster of combinations of hacks for different use cases. I think your approach to fixing this was hence along the right lines.

I did investigate the mask pathway of the code and a CustomStencil approach actually is an alternative to changing code here if you were to enable that process, but I think there’s 2 issues with that: 1) you would have to setup stencils for your scene to achieve this (maybe you could get away with setting the entire scene in an entire artificial skybox featuring the black background and only adding the stencil mask to that skybox?) and then 2) there appears to be some issues with this path, which I also encountered:

https://forums.unrealengine.com/t/high-res-screenshot-with-custom-depth-mask-not-working/572033

Anyway, it may still work if fully setup in this way, but regardless to be on the safe side, I spent a bit more time today fleshing out the implementation I anticipate is required if we are going to fix this in the engine. Please see perforce shelf 43945730.

This shelf adds an additional option to the high res screenshot config struct which tells the output to either stomp with 255 or simply adjust the existing value for the appropriate output:

[Image Removed]The image attached is the output from this setting being disabled.

To answer question 2: I’m not entirely clear on why the alpha is inverted to begin with but from my investigation I am pretty sure what is actually happening is you have a default black background in unreal with alpha pre-set to 255, and meanwhile the opaque base pass is not outputting the alpha for the material (emissive color is RGB only), so we may just be setting that to default to 0, and I know that the base pass occurs prior to translucent pass, so it may just be that the alpha is always ignored there. I think this is the most likely explanation.

The stomping of the alpha is a different answer, so interestingly because we have the way we write out the base pass I just mentioned, it means all the colours are essentially semi transparent. I believe this is accounted for in the Unreal pipeline, but for mapping to a screenshot/png file for example, those values are interpreted differently, hence the scene will be entirely black in the png if the alpha is not inverted. The 255 stomp hack (which turns out is not caused by me, which I was grateful to discover haha) is to account for this, when really this inversion process may be more appropriate (although questionable, just feels like more hacks).

Anyway, didn’t want to leave you in limbo over the two week summer break, I hope this gets you unblocked. Also, out of concern that you may not have perforce access (really hoping you do), here are some screenshots of the items required to get this additional option in the UI:

[Image Removed][Image Removed][Image Removed]Note: also needs the head file declaration ofc

[Image Removed]Note: the alpha multiplier must still be applied to account for any data mismatch in alpha channels (I believe that’s why you see the greyed out pink box in the final image).

Thanks, and I hope this gets you unblocked until we are back from summer break on July 14th.

Jon

14

Thanks for looking into this. I’ll ask around here about P4 access and see how this works.

15

I implemented what you wrote here into the engine (coincidentally I also had a task to make this toggle) and that is great, but we do still have the issue of the inverted alpha. In the sample project if I run this with ForceOpaqueAlpha to 0 the resulting image is totally transparent (as alpha is not inverted through any other means). Is that expected? What do you get when you run the sample project with your engine changes?

With the Mask approach are you referring to the “use custom depth as mask” option of the HighResScreenshot tool? We have investigated this in the path but found there was artifacting in the image and additionally it would not work well with some post process overlay images we wanted to add to the image (they’d be cropped out).

Two persons on our team have access to P4 so I could potentially be able to grab a shelved CL if I can get some of their help.

16

Tavis, apologies for the delay, I have been OOO travelling and came back to some fires, I am looking at this right now and will get a response to you later today.

17

Ok I had another look through and I think the problem is I set all this up whilst using your invert alpha material in the scene! My mistake sorry, I’ll get the shelf updated.

Did your colleagues manage to get access?

18

Ok so Im starting to think the PostProcess mask you’ve implemented might be necessary. In the basic Opaque Domain + DefaultLit shading model of the material, it doesn’t look like we write out the opacity at all to the final scene color and we compensate it earlier in the pipeline, then write out to a fully opaque render target. Hence I think your solution here is necessary, or a custom stencil method is likely required. I’m going to pass this question back over to a tech artist much more familiar with this sort of setup in case they can advise anything else.

I’m starting to think though that I may need to fix up the custom stencil screenshot path, that seems like the most appropriate long term solution here…

19

Hey, sorry for the delays, I’m just now getting caught up here. I’m usually a fan of custom stencil but it sounds like that’s not working? The other approach to this might be to use Movie Render Graph (maybe overkill for screenshots) since it’s got support for render layers that the virtual production folks use for offline compositing. Thoughts?

20

Yeah custom stencil in this case doesn’t work as it cuts away at elements that we’re adding to the image via post process and ultimately doesn’t render things correctly.

Movie Render Graph I have seen raised as a solution in other discussions and that’s interesting and I think well worth exploring in the long term. The problem is that it’s an ordeal to remarkably rework an image generation pipeline to use a new approach. But I think it makes sense to have a look at some point.

The reason I was asking about this particular issue of the inverted alpha is because it seems like the existing high res screenshot system almost works and is like 99% there haha. It was simply puzzling why some peculiar engine kludges seem to be required for this very last step to get transparent background images. I wondered if there was something I was missing or some relatively simple fix.

If it is ultimately a challenging thing to address, then I think for the immediate short term I’ll have to continue to manually invert the alpha via post process volume, and then in the long term have a look to see if we can create transparent images in a better way through Movie Render Graph.