You’re more likely to hit bugs and glitches on Linux, so I wouldn’t worry too much about that.
Of course you can’t build for Windows from Linux, so there’s no avoiding it when you need to create a Windows build. Same goes for XB1, PS4 and Switch since their tools are all Windows-only but you could (and should) have a Windows box setup as a build worker (using a CI tool like Jenkins, for example) and make scripts that fetch from your repository and build those other versions for you.