For me, it mainly happens from people who a few weeks/months back left a comment and then followed up with "Maybe check out our channel and subscribe if you get a chance". They see that I have not subscribed so they rage unsubscribe, that's what I can gather from my stuff