It's not only about the views. Even if the views are not there now, it doesn't automatically mean they will not come.
If you are annoyed by these few people now, you wouldn't be able to make it when you are big. When my channel was smaller I received some negativity here and there and I ignored it and kept going on. Now when I have 30 times more views per day, I get negative and insulting comments daily. Nowadays, I am slammed and insulted several times per day on my videos, including but not limited to my English accent (I am not a native English speaker), my hairy hands, my boots, the clothing, my filming or everything other imaginable and even unimaginable (the Internet is really good at inventing new kinds of insults, trust me). But as long as for each negative comment and dislike, I receive 10x-12x more positive comments and likes. I am happy and I keep going, while improving thanks to the negative comments - i.e. sometimes not speaking in the video at all, shaking the camera less (if possible), editing out the blurry parts of the video etc.