Best method to capture static mesh metadata using SceneCapture2D

I am trying to use scene capture to record how many pixels of each static mesh are visible from a camera view. Here is the result of a SceneCapture in ‘BaseColor RGB’:

image

If I assign each static mesh a unique BaseColor (i.e. like in the image above), then I could simply count the pixels of each color in the resulting image, which would represent how much of each static mesh is visible to the camera, for example:

Green = 3,845 pixels
Blue = 2,714 pixels
Red = 3,109 pixels

If I keep a record of which static mesh has been assigned which color, I can then map the color counts back to the static mesh, which I can then use for further processing. This is all good and well…

HOWEVER…

This is not how I would like these objects to appear to the player. I would like any ‘assigned’ colors to be for the purpose of the scene capture analysis only, and not affect the appearance to the player.

Is there a spare/redundant channel in any of the below data types, which I could use to record a ‘unique ID’ for each static mesh without affecting visibility to the player and without restricting basic material editing functionality?

Ideally it would be great if was possible to capture a custom source, for example:

SCS_Custom UMETA(DisplayName = “Custom RGB dta defined by user, for SceneCapture purposes only”),

If this is not possible, would a practical workaround be to create two identical static meshes, one with a unique RGB BaseColor (which is hidden from the player but viewable to the SceneCapture), and another with the final material (which is viewable to the player but hidden from the SceneCapture).

I understand this method is essentially doubling the number of static meshes just to extract some basic scene info, which seems incredibly wasteful and doesn’t feel like the most elegant solution. However I’m wondering if it might work? However before I jump in and create a second copy of absolutely everything, are there any important considerations of this method I should consider beforehand? For example will this affect rendering speed, collisions etc?

From the top of my head, I’d go with some Post Process Material (only on the scene capture) that assigns different color to each primitive based on some data.

One I could think of is Custom Primitive Data (although I have never used it so I’m not sure if it’s viable, but maybe worth looking into?): Storing Custom Data in a Material Per Primitive | Unreal Engine Documentation

And another is Custom Depth Stencil, which you can for sure access in PP (although it only works if you have less than 256 instances on the screen at once).

But probably there might be other options too I can’t think of right now (or I don’t know about), my point is, you can try to do it from the capture’s PP so that it won’t effect the player’s camera.

Thanks! I’ve just read the Custom Primitive Data article, which looks promising. I’ll play around with post-processing to see if I can add this data to the capture. If so, problem solved :slight_smile:

OK I’ve followed the video below and managed to create a custom primitive data channel, which I can use to assign each actor a different color or value.

The video below shows how to add a post process material to a Scene Capture.

However how can I actually import custom primitive data into the post process material?

Hey, sadly I have never used them, so I couldn’t say without research. I’d look into the “Custom Data 0” material output, maybe you can access that buffer somehow in PP? (but maybe it’s not doable without engine modification)

This guy seems to have stored data in the SubsurfaceColor. Maybe you can do the same (or something similar) through a channel that you’d not otherwise use? https://answers.unrealengine.com/questions/873059/using-custom-data-outputs-and-accessing-them-throu.html

Thanks!

SubsurfaceColor worked a treat :slight_smile:

The spheres in the image below have a different BaseColor and SubsurfaceColor, and by accessing SubsurfaceColor in PP, I can render SubsurfaceColor to a separate target while keeping the main scene unchanged.

I guess this means I can’t use SubsurfaceColor for any rendering in the main scene, but it beats having to create a second copy of all assets, or editing engine code to create a new custom shader.

Yeah, this is definitely a hacky solution, it’d be much better if we had more access to the rendering pipeline from the editor and plugins; maybe in UE6 :smile: