Not necessarily. Here's how it works: there are great ads, good ones, average ones, and s****y ones. Which ones you get depends on your audience retention, in comparison to the audience retention of your fellow network partners. If there are more small people, they won't do much.
1) smaller channels tend to have lower audience retention, because they suck and will never grow at all. These channels will have almost no impact on you.
2) some of those channels may have amazing audience retention, but if their views are so small, their space in the market will be too small. It's like how you don't worry about the price of Microsoft stock might plummet by adding your 10 shares to it.
The second one may become a worse problem if time continues to go on and tons of small channels with stellar audience retention are added (which is very unlikely), but at that point they have the same effect as the network adding one or two more big channels to their network. I doubt it will affect you that much, unless you were getting paid more than you were worth.