What happened yesterday…

December 2, 2010

Yesterday we posted the video of DHH’s Keynote presentation from Ruby Conf X. David promptly wrote his own tweet in his voice and notified all his followers of its availability, and then they started retweeting.

This caused a huge rush of users to try and watch the streaming video and/or download the various sized MP4 files on the site, and what do you know… lack of stress testing and the site went belly up.

After reworking our Apache configuration (currently the rails 2.3.8 app is running behind apache/passenger) we restarted things with a little improvement. Yes we are looking at Nginx.

A number of people have asked me why I host on dreamhost? why not S3? Right now I charge conferences for recording the event. I do not charge the community for downloading the videos, or charge any sort of recurring fees for hosting them. I pulled up the Amazon S3 Calculator and took a look at what it would cost to serve my videos from there (I didn’t try to calculate anything fancy with Cloudfront, because I don’t have the knowledge at the moment to figure out the configuration). I have about 750GB of videos stored on dreamhost and on an average month without a conference release I get use about 600gb of bandwidth. The current billing cycle at Dreamhost will use about 2500gb of bandwith. So with the S3 calculator this would cost me anywhere from $180 a month to $410+ a month.

Currently confreaks is hosted on a Virtual Private Server at Dreamhost and apparently was on a 100mbit network connection. The rush of traffic put the throughput beyond the pipes capacity and the server became unreachable.

I received notice from Dreamhost at 1:05 PM that they had successfully moved my server to a gigabit server, and that we were averaging 140mbits of traffic.

Below is a graph of our bandwidth usage for confreaks.net (the dynamic app) from 10/29 to 12/1. You can see the total traffic consumption for yesterday was about 287gb, almost double the previous busiest day.

Anyway I have to get back to producing videos. I’m planning on spending time in January and February doing software updates to the site and addressing scaleability issues for the future.

Also I want to raise a thanks to the guys at TekPub who put up a mirror of the RubyConf Keynote talks.