Bad to hear that you have such problems with haters :(. But just ignore bad comments or even block/report them when they are posted as a youtube comment, because mostly they just come from kiddies that don’t know what they are talking about ;).
Unfortunately such stuff nearly happens to everyone that shows something to other people or give some feedback to them. For example I always get 1-2 really bad comments on my youtube channel and even on Skype, but mostly I just report them to google and block the user. 
(I won’t post any examples now, because otherwise I will have to delete my post because of too many swear words ^^)
But unfortunately there is now way how you could avoid the stuff with the down-voting or the bad comment. You could just disable the voting and probably block the IDs from the users that are annoying you, but in my opinion that’s not a good solution.
I wish you the best that you will get that job!