Line trace an opacity mask

UE3 had a feature called “Physical Material Mask”, that allows you to set black/white texture and apply 2 physical materials to surface. In UE4 its still in backlog, but maybe you can develop it on your own.

https://docs.unrealengine.com/udk/Three/PhysicalMaterialMask.html

One more option is to detect color of hit surface by getting it from color buffer. So the invisible part of image will be filled with solid color, like (0.5,0.5,0.5), and you will ignore line trace when it hit.