If you turn off your directional light and use a static (ie. baked) spot light, coming in at the same kind of angle, do you still get the same problem?
If this fixes the issue then it could be light bleed from the dynamic lighting. If this is the case, try adjusting the shadow bias of the directional light.
Best wishes,
Matt