I would say keep your Subscribers view able I think your Subscribers would actually appreciate the fact that you are comfortable even with a smaller number and that they still matter to you, when you Subscribe to someone and see the number you feel like you are apart of a community no matter how big or small that community is.
and in the words of my favorite TV Show character Tyrion Lannister: Wear it like armor, then it can never be used to hurt you. (That was cheesy I know) but it provides a good point try and get comfortable with a low number that way if people insult you about it or don't Subscribe to you because of it you can brush it off and move on.