Quite a while ago, I disabled the ability to rate my most popular videos. You can see my top 25 videos at bottom of this page.
http://socialblade.com/youtube/user/racegrooves/videos
Nowadays, I don't worry about the negatives unless they are really out-pacing the positives. If I believe that somebody is intentionally sabotaging a video, then I will turn off the ratings.
If one can suffer throught the negatives, allowing viewers to give it a thumbs up also winds up allowing them to promote your video if they have such options enabled. In other words, If they like a video, they might also have that auto-publish to their social feeds such as Twitter and Facebook.
Congrats on the growth of your video!