My feelings are similar to flag's. From time to time, I try to unfollow people who have not contributed in months and years. I find unfollowing time-consuming and tedious. If it wasn't so much work checking people to see when they last commented or posted or shared, I would probably only follow about 20 people + notable newbies who I want to encourage and find out more about.
The title "Unfollow Guilt" is hilarious though. Guilt is an important emotion that we should follow all the way down the rabbit hole. When we catch it, we should grab the scruff of its dirty collar, pull it up into the light, and interrogate it. If you feel guilty because you betrayed your own values, then your guilt tells you to smarten up or to question your values - maybe you've outlived them, maybe they're not even yours.
But I digress.
EDIT: By the way: In the last hour, someone "unfollowed" me. True. Should I feel guilty?