ultimately, do what you want
to crystallize what that means for you, it might be a good idea to think about the consequence that decision would have on your channel
simply, youll attract different audiences based on the content of your videos
this will create a divergence in viewership relative to your subscriber count
i cant confirm this with 100 % certainty, but ive read the ranking of your videos will be affected if theres too much of a discrepancy between your subscriber base, and the amount of views you get per video
a lot of subscribers, but low viewership, would then mean your videos will rank lower, even with optimized SEO and word-of-mouth promotion
to answer your question, i guess it all comes down to how seriously you take this, and what youre trying to achieve
with that said, having many different types of content doesnt doom your channel, it just makes it even harder to compete with other channels