I would like to know how to make this effect of merged circle selection.
In Unity I used the Stencil Buffer for this (2 Passes).
In the first pass, I defined the Stincel areas (used to define drawablety)
In the second pass, I renderd the circle on the areas without “stincel values”
eg the green area is the stincel buffer mask and the red one is the decal area
How can I archive this effect in UE4?
You might be able to do this by rendering some circles to the custom depth buffer and then performing an edge filter on that buffer to get the edges you want, and then multiply that with your outline shader.
It definitely won’t be as easy as using a stencil buffer. To use a custom stencil buffer you are going to have to get down and dirty with the UE4 shader code and drawing pipeline.