How do I remove widget color?

Sounds like you tried adjusting Content Color and Opacity - which would affect all children; if you want a transparent border, setting the border’s Brush Color’s Alpha to 0 should do it: