Telling the Truth
November 13th, 2007
Without intending to throw so much love towards 37signals tonight, I feel this is worth mentioning. Over the last few hours, I was unable to access the 37signals website in addition to our Highrise and Basecamp accounts (both 37signals web-based products). Here was the explanation just posted on their blog and in the top section of Highrise:
Downtime summary
On the evening of Monday, November 12, we experienced a few of hours of downtime due to an explosion at our main data center in Dallas, TX. This event lead to the eventual failure of a backup cooling system. Without adequate cooling, our servers had to be shut down to prevent permanent damage. We have detailed the events that led to the downtime. We deeply apologize for any inconveniences this may have caused and will work hard to make sure we reduce the likelihood of this happening again.
What they did well:
- The decidedly non-techie tone.
- They told the truth (as far as I know) and provided a detailed account of what happened.
- The notification posts were up as soon as the servers were.
- 37signals did not throw blame directly on Rackspace, their hosting provider.
Even though it was obviously a Rackspace issue. - Their apology and offer to credit concerned customers.
Thanks for hanging in there.
Will be using this situation as both a wake-up call and a learning experience. While our systems are engineered to chug through major failure, this “perfect storm” chain of events beat both our set-up and our data center’s sophisticated back-up systems. We will work hard to further diversify our systems in order to make an future downtime event like this even more rare.
We apologize for any inconvenience this downtime caused your business. If you feel you were significantly impacted by this downtime please send an email to support and we’ll credit you for the downtime.
Will I be asking for a credit? Absolutely not, because I was not significantly impacted (now ask me if I would take a credit from Verizon or Comcast that I did not technically deserve). Well done, 37signals.