Here is a little example:
Lets say you have a mesh that looks like this:
As you can see, I have isolated the right “arm” in the Uv space. That makes masking a bit easier.
In Photoshop, you create a simple black/white image (that is your mask).
That mask is now copied into the alpha channel of the diffuse texture.
Be sure to save the image as 32 bit file.
Now import the texture and create a material.
- Change the Blend mode to “masked”
- Connect the alpha channel of the texture sampler with the Opacity Mask pin
In the level, it looks then like this:
That should give you an idea about it
A few things to note though:
The hidden geometry does not leave a “hole” because the geometry of the arms and the torso are not connected
If you use a connected topology, you would see inside the mesh…
In this case, the mask channel data has the same resolution as the diffuse texture. For precise masking this is important.
Here however, a mask with lower resolution would actually suffice.
So it you create a second texture asset with a lower resolution (say 512*512), you can save some memory.
The masking would create a small gradient when sampled against the 4k diffuse texture, but that would be well outside the UV polygons…