I got my very first "Dislikes" a couple weeks ago, both on the same day I think. I laughed when I saw them. Then I watched random YouTube videos for a while. I didn't give 'em a second thought. I find it hard to get upset that someone didn't like a video or two of mine when there are videos people make that I don't like. Sure, I keep my finger off that button and they don't, but that's what makes them who they are and me, well... I think I'm pretty awesome in my book!
As far as the people who might not watch your videos because of a few dislikes... think of YouTube like a pub where every week hundreds of people come just to see you, would you care if a couple people you didn't know, chose not to?