Personally I've turned losing subscribers into a positive thing. The way I see it is, just like how sub4sub subscribers are worthless and don't contribute, the same goes for people who are subscribed to your content but don't want to be anymore or don't resonate with it.
If no-one were to unsubscribe ever then that would mean a lot of people who no longer enjoy or resonate with your current style of content production sticking around, which means more dislikes, more negative comments, and extra stat skewing when you have a bunch of subscribers who never even watch your videos etc. So I'd rather anyone that ever fell into that bracket just unsubscribed from me.
It's bound to happen in any sort of niche too since you'll often be targeting a very specific need or view and people who fall outside of that might not want to stick around. However it's better to stick to that niche and please the ones who do fall into it and not please the ones who don't, than to try and branch out and do everything while not really creating any real fans since you never go into depth enough on a single subject/genre of games/type of videos/whatever it is you're making content about.
However that doesn't mean you should just ignore when people unsubscribe, you should think "Why now?" since often it means something has changed that makes your content no longer resonate with them. If it's because you're honing in more on your particular niche, that's probably a good thing - but if it's because your content is lower quality, or because you've offended a particular group, or done something else bad then you need to take notice and rectify it before you lose more.
Go to Creator Studio -> Analytics -> Subscribers -> Subscribers Lost, then at the bottom, where it says "Video", click that and have a look at what videos you lost subscribers on. What did you do different in those videos that made people unsubscribe?
Hope this helps