Well it's a good point not to give up, but at the same time, I think it's important to evaluate your efforts properly.
This is important, especially if you have aspirations of being very successful on youtube. I will tell you right now that
if you've been working at your channel for several years, posting regularly, and only have a handful of views and/or subs, that you will likely never be very successful if you continue with that channel. That might sound harsh, but think about it. How many successful Youtubers do you know who worked hard on their channel for years with very little success and suddenly made it big 5 years later? I can't think of any. Now, if you don't care about success so much and just do it mostly for fun, then that's fine. Keep with it.
It's hard to be successful at Youtube, especially in certain oversaturated niches. Not giving up is admirable, and you should always give yourself and your channel as much of a chance to succeed as possible, but you can't be afraid to give up on a channel either. You have to understand that 'giving up' on a particular channel, after a year or two of relatively stagnant growth, doesn't mean you are a failure. It simply means that you've recognized that this one particular idea didn't go over as well as you thought it would, or that the content on this one particular channel isn't quite what people are looking for.
So I agree partly with the OP, you should never give up after a relatively short period of time. You have to have more patience than that. But also, there is no shame in calling it quits on a channel after years of stagnation, regrouping, coming up with a new idea or a new approach, and starting fresh either.