I would like to know if there is a way to have 2 layers in composure that exclude each others characters from the garbage matte. So if there are two separate character groups in layers that are interwoven with each other. The majority of layer A is infront of layer B, but when there is a character from A that is behind someone from B this character needs to have a cutout of the B character infront of them. That way once it is composited you will see the right amount of each character without having to create many layers to have them separated.
In this screenshot you can see the red character in layer A is behind the green character in layer B. This character is the one I would like to be cutout in the garbage matte.
Here I have created a similar result to the garbage matte I would like to create. To get this to work I had to override all of the characters materials to be black for layer B and white for layer A. In the composure CG layer I included both layers and added the base color pass to the render passes. Even though this is similar to the result I would like. It’s quality compared to the actual garbage matte output is significantly worse as there is too much fringing along the edges to get a good result in the composite.
If it is possible to have this be done completely in a CG matte layer or some other way that’s less complicated that would be great!
I’d love to hear any suggestions. Thanks!