Outage debriefing

posted by simon on Wednesday, January 16, 2008

I promised to write a little on our outage earlier this month. In short what happened was that our scripts suddenly stopped working. They would crash when reading a CGI parameter. The @params var in CGI was being set to an empty hash instead of an object with a special [] method (see here under Parameters. We suspect that Dreamhost had perhaps updated something or modified some Apache configuration to cause this sudden change. So I made a little "hello world" test page to attempt to recreate the problem. To my surprise the test page worked fine. After a lengthy period of making the real page successively more identical to the test page while swearing repeatedly I found that removing require 'haml' made the problem go away. Now Haml is a nifty little templating engine that I was using to render our html. (Actually I'd had a conversation with Daniel on the plane just days earlier about how I wanted to stop using Haml since it's benefits were minimal and it was one more place where things could go wrong. So I was right about that). Luckily it wasn't hard to revert back to the plain old html templates that I was using before installing Haml.

In defence of Haml, we were using 1.5 and the latest version was 1.7 at the time (now 1.8). Perhaps the bug (if it is a Haml bug) is fixed in a new release. But at this point I don't care. I still believe Dreamhost must have done some kind of update to something ruby related for this to have suddenly happened like it did, though their (excellent) support team were not able to verify this.

Thanks for the many messages of good-will and support we received via email, twitter and this blog. Surprisingly not a single negative message arrived during the outage. Which perversely makes me want to encourage some negative feedback. Come on guys, we can take it. :)


Anonymous Anonymous said...

What is wrong with you guys! I couldn't update my site for four days!! Why did it take so long to solve this problem. Like the dog in your awesome new commercial I will be shaking Tiddlyspot out of my fur. Excuse me while I paste all my tiddlers into Google Notes. Good bye and good luck.
Disgruntled Ex-user

Jan 16, 2008, 9:58:00 AM  
Anonymous Anonymous said...

Also why haven't you installed TW2.3 yet? It's been out for ages.

Jan 16, 2008, 9:59:00 AM  
Anonymous Anonymous said...

Thanks for this, Simon.
It really does sound a little odd; why would Haml stop working all of a sudden if there were no changes of any kind?! Oh well, I guess we'll never know now...

Jan 16, 2008, 5:07:00 PM  
Anonymous Anonymous said...

me2 i am outta here! this place sucks! i don't care if it is blue!

Jan 20, 2008, 9:53:00 AM  
Anonymous Anonymous said...

Argh, still not saving :(

So you want negative criticism, hmm?
Let's see, how about...

Your mother was a hamster and your father smelt of elderberries!

Jan 25, 2008, 1:59:00 AM  
Blogger simon said...

Jack, saving seems to be working okay now. If you're still having trouble please email support with some more details (and taunts).

Jan 28, 2008, 11:22:00 AM  

Post a Comment

<< Home