Sorry for the late reply! But I did get it to work eventually by making a mask and making a post process material that simply has the mask texture as an input into it. That way, when it’s rendered the mask will be applied and everything that is black in the mask will turn transparent.
It also works with semi-transparent objects so we made an automatic Nuke setup that can create a mask from cryptomattes. The limitation here is that we need to create the texture outside of Unreal but the result is very good.
We even made it so that we can mask out clean and soft shadows. 