My goal is to use images generated with unreal engine as input and ground truth data to image processing algorithms. I have actors moving around in the scene and means to export the captured images. Now I want to create segmentation images in which the moving actors are represented by white pixels while everything else is black (like in the following examples).
Unfortunately I don’t really have an idea where to start (e.g. if it is a custom shader or rather done in post processing).
Just using a white material which does not respond to lighting for the actors and a black one for everything else does not seem feasible. I’d like to be able to assign a flag to an actor which causes it to be displayed as white. Ideally the render mode should be switchable so that I can recrod a scene once in its regular look and once in segmentation mode.
We already have a custom SceneCapture to simulate the effects of a real video camera - maybe it can be extended to record in segmentation mode. I am grateful for any suggestion how these goals can be achieved.
What you probably need to do is create a post-process material with ‘SceneTexture’ node which you should set to CustomStencil in the dropdown in its properties. You should also enable Render Custom Depth on all static meshes in the scene and assign to each of them a different Custom Stencil (or assign them based on object type or whatever you want the objects to be segmented to).
Then what you do is create apply this post process material to a post process volume. As it’s quite hard to change post process materials in post process volumes from blueprints, as far as I know, it might be easier to just create a separate post process volume and switch between them. I’m posting a few screen shots below:
This is where I find all the static meshes in the scene and assign a random custom stencil between 10-255. I kept the stencils below 10 reserved for the landscape and the sky.
The properties of a static mesh in the rendering part. This is where you enable Custom depth rendering and where you can assign a custom stencil.
This is the final look. I’ve manually set the landscape’s custom stencil to 1, which is rendered green (using bitmask node) and the sky is set to custom stencil 0 (you can’t change that actually), which is rendered black.
I still have two problems now. The first one is the segmentation area has different RGB data. For instance, I may get (192,33,43) and (193,33,45) under the same mesh. So it’s hard to use it.
The second one is I can’t initialize the custom depth stencil because of error below.
It seems you are trying to convert actor references to static meshes. You have to select static mesh actor in the dropdown of Get All Actors of Class node.
