You can put an actual mesh mask in front of the camera but you might choose to fake any 3D / lighting effect in the end using post processing, if a transparent material does not provide a high enough quality of detail (translucency / refraction / shadow etc.)