Smooth jagged edges in Render Target

I am working on a Line of Sight visualizer, using Line Traces and ISM. I capture this to a Render Target using Scene Capture 2D. The result is good, but not good enough when the trace angle is steep. This results in a jagged edge in the Render Target. This means that the player can see alone the very edge of walls that should be out of sight. Is there a way to smooth this out, or even fill in the “blanks”, thus creating a straight line along the edge? See pictures for ref.