SEO is helping but there is a very important factor, especially for small channels: "Channel Authority" and "Watch Time". If you do not have lots of subscribers and almost zero watch time, YouTube is not going to rank you high in the beginning. So, your channel will develop slowly.
Social media and marketing help you get the initial watch time, as well as social cues that YouTube uses in its algorithm. I'm afraid, small channels HAVE TO do some marketing if they want viewers. Personally, I also do not like promoting. It is very boring to copy-paste the same text in Facebook into different groups, pages, etc. I also sent personal messages to my friends, whom I thought would be interested in my video. CTRL+V is your friend.