I love articles like this. Nice write up of your trial by fire. Web server/application infrastructure architecture is what I do for a living, though I work in the lending industry, so we get far more traffic and serve bigger loads, but we also get multi-million dollar budgets which makes things a little more easy to approach. None the less, the growing pains, be it small app or large app, still suffer these same scaling issues as you grow and take on new traffic. It's always quite fun to deal with and I love it personally. The favorite part of my jobs are being involved in Sev issues that arise after new code is dropped, or we add a new service, or instance something off. That on the fly problem solving when you're under pressure is the reason I love my job. My favorite quote, from a very talented Microsoft rep I trained with for a year, was "Heroics. Don't. Scale." He repeated this frequently, and with heart every time when my management would be like "We can't re-code that right now", "or we won't have those other servers in the farm for another quarter". I always disagreed with him. Heroics do scale, it's just more difficult and you don't sleep as much. :)
This sounds very familiar to when Hubski was the top link in a response on AskReddit about a month ago. The boys were up all night trying to do anything to get Hubski to not be dead. forwardslash mk I think you'll like this.
I'm glad you found it interesting :) Reddit can be very overwhelming. The amount of traffic I received in 4 hours was greater than what i received on from hackernews over 12 hours. My little blog does not compare to hubski, I sure did learn a lot from this experience.
Thanks. That was an interesting read. After being hugged by AskReddit we also put more static content on the shoulders of nginx, and drastically improved caching. Prior to our hug, we weren't even using CSS sprites for our images, which resulted in many unnecessary requests. We also use DO. That's too bad. I read your article, and enjoyed it. We won't punish your servers nearly as much, but you'll never see us pull down an article either: http://hubski.com/pub?id=100957(Un)fortunately after 4 hours of being number one on the technology subreddit, my article was marked as SPAM and yanked of the page. The only explanation I got from the moderators was that this post was an opinion, not news. In some way I was relieved as I was seeing the traffic going down. It was still a lot but my server could handle it.