I thought a lot about this question before starting my channel. Bad comments are still activity on a video and good for analytics. And yes, loyal subscribers will often take care of the trolls themselves.
On the other hand, social media trolls abound and I refuse to let my channel be another platform them. I want the community I build to be a fun, troll-free space. So, I went to my comment filter and typed in all the nasty words I hate. I'm sure they'll come up with their own creative spellings that I will simply add to the growing list to guard against future trolls. And then I'll ban them. I don't care about analytics.
As others have said, people are free to express their disappointment, disagreement and dislike of me or the video as long as they do it respectfully. I'm not thin-skinned. But, if they start attacking my subscribers or being crude... they're gone.