I'll put in my 2 cents since I've been in this EXACT situation.
I went from 0 to 4K subscribers in about 6 months. During that time I had a primary focus on one game. It was popular enough to get viewers, but not popular enough to be over-saturated with creator. I owned that game on YouTube with only one real competitor (which I even did a collab with) and eventually beat him in terms of views at least.
After about 8 months, I noticed it (growth) was starting to slow down, I was running out of ideas, I didn't feel much joy for it anymore.
So I did less and less of them, and more of other content. Every video the comments were "where is (the game)" "why are you posting this?" etc... they all got a low amount of views. It will vary from creator to creator. I got hit hard because my sort of content was meant to be useful "how to do this, how to do that, the best of this," and so on. So naturally people were more there for the game. I did have a decent amount of fans who were there for me though, and they pushed me through.
Moral of the story: I'd never want to be stuck in one game unless it was so massively popular (Minecraft or CoD level) that the room for growth is practically unlimited.
As a viewer, I prefer people who are consistent in what they do (to a degree), it's the reason I subscribed. As a creator, I like to try new things from time to time, which probably hinders my channel a bit.