Lightening Storm Causes Data center Outage


Companies spend countless hours trying to find the location that they feel is right for their data center, they evaluate earthquake zones, hurricane zones, tornados, etc. and it turns out lately severe lighting storms have been the culprit of several outages. Late last year both Microsoft and Amazon had suffered due to a data center in Dublin having an entire power outage due to a transformer getting struck by lighting, causing an explosion and fire that knocked out the utility and enabled the generators to start. Now recently in the news Amazon in Virginia had almost the same problem. So now all the latest headlines are doubting cloud computing, and that’s just silly. Kelly Clay said it best on Forbes, ” So is the public cloud reliable? It seems to me that the public cloud is as reliable as the systems its tenants have in place to rebound when things go wrong.” In the data center world you plan to have a failure. She goes on saying, “Back in 2010, Netflix posted five tips for survival in the AWS public cloud. Their point number three tells us that the best way to avoid failure is to fail constantly. They even built a tool, dubbed Chaos Monkey, that causes random things to fail in their infrastructure so they can learn to avoid outages like this one. It will be interesting to see if they share a remedy for this latest failure once they engineer around it.”

So what actually happened? Here is the cliff note verison “Normally, upon dropping the utility power provided by the transformer, electrical load would be seamlessly picked up by backup generators,” Amazon said in an update on its status dashboard. “The transient electric deviation caused by the explosion was large enough that it propagated to a portion of the phase control system that synchronizes the backup generator plant, disabling some of them.” In this instance i’m not sure how you could prevent the lighting strike but I do believe there were things that Amazon’s customers could do to prevent being down.

So how can we prevent it from happeneing again? I think it’s impossible to prevent the actual lighting striking but if you are a company who is utilizing the cloud TIA-942 should be your best friend.  TIA-942 goes over these key elements when building a data center:

  • Network architecture
  • Electrical design
  • File storage, backup and archiving
  • System redundancy
  • Network access control and security
  • Database management
  • Web hosting
  • Application hosting
  • Content distribution
  • Environmental control
  • Protection against physical hazards (fire, flood, windstorm)
  • Power management

“The principal advantages of designing data centers in accordance with TIA-942 include standard nomenclature, failsafe operation, robust protection against natural or humanmade disasters, and long-term reliability, expandability and scalability”

I would love to hear your thoughts!



Related Posts Plugin for WordPress, Blogger...

  • Larry Farmer - July 4, 2012 - 11:28 am

    Well said! Beats the heck out the hysterical articles I ‘ve been seeing all over the net.ReplyCancel

Your email is never published or shared. Required fields are marked *