Yes, the problem is just that I can’t line trace in the material, so I can’t make a smooth version (only one that does a single linetrace for the entire enemy, neglecting specific spots).
I solved this by doing a line trace from my character in all directions, then the only challenge was to get the data into the material I’m trying to hide. I used a really elaborate workaround, desribed in this Reddit post. Then I could write a material function, which checks whether the distance on the xy-Plane from the specific point on the enemy to the player is smaller than my line trace result (thus visible by the player) and set the opacity mask to 1, else to 0.
Implementing that feature was quite a journey, but it worked out well!