Sorry you are down, wait we are down, too! (Or: The sad state of HA)

September 6, 2013

This week I learned about an outage that happened at a provider of Microsoft Office applications. Which reminded me about the sad state of high availability across the industry.


 

More on the provider

The provider is a medium sized infrastructure vendor that is successful in providing hosted Microsoft Officer applications, basically running the servers for Outlook for their clients. They are  not small with 5 corporate locations on both sides of the Atlantic, and 10 datacenters in the US and Europe. The provider is professional and has e.g. achieved SOC2 and SOC3 compliance.

The vendor offers a a 99.999% up-time guarantee - but that was definitively broken by being down for most of a workday from 7 AM till 3:30 PM.
 

What happened

Clients noticed in the morning, that they were not able to get their emails, send emails and work with their calendars. When calling the provider, calls went dead, the provider's website and support applications were not available. The first provider to client communication happened then over... Twitter. And Twitter remained the lifeline between provider and customers till - you may have guessed it - the Twitter account went into Twitter jail for hitting the daily limit of 1000 tweets. And while that seems generous - it's not much if Twitter is your only ways of communication with multiple hundreds customers.
 

What went right

The provider got the system back, tried all they can do to get customer informed (so they were obviously in the dark), offered the usual letter form the CEO in the next 24 hours and had that followed up by the COO. The vendor communicated pro-actively that they had broken the service levels, and that that they would waive the requirement to ask for re-imbursement, and re-imburse customers diretly based on their SLAs.
 

What went wrong

In the CEO letter the provider already offered an issue with their routers as the root cause of the outage. And while it's fine to not have the ultimate reasons 24 hours post an outage event - you need to do better than the following from the letter of the COO- 48 hours later:
 

"the routers connecting all our systems each received an invalid update”
 
As my colleague Frank Scavo pointed out - that is pretty passive language - no one did the update. Was it the provider, was it the router manufacturer etc - we do not know. No one is taking responsibility. 
 
Moreover there was no mention why all the routers went down, why the update was not tested separately and routers were not switched in groups, why were no backup routers held back etc. 
 
And the provider explained that the phones (VoIP) and customer support systems were down - because the provider is using its own infrastructure. And while drink your own champagne is a good argument - it is an empty glass when you have a system outage. The provider missed to explain how this happened and why e.g. their DR for their operational systems did not kick in.
 
The lack of an answer on both of these areas does not instill confidence to customers.
 

The sorry state of HA

We all know that data center components should only be switched in groups and with redundancy - but obviously this went wrong at the provider. Equally running your critical customer systems on the same infrastructure with your customers is a disaster waiting to happen - and it happened to this provider.
 
So why do well known and proven HA principles get broken? The reasons are manifold. Human error, overconfidence (both are my bets in this case), cutting corners cost wise, not thinking the impossible, etc are all likely. And human nature is good at discrediting highly unlikely events - but when they happen we too often do not think of it was the decision makers looking the other way back then when they came up.
 

MyPOV

Outages are always unfortunate and can't be planned as the Dilbert cartoon requests. It comes back to how a provider reacts, investigates, communicates and then remedies the sources. On reaction and investigation the provider was solid with a B rating - but on communication and remedies they only deserve an F.
 
And HA on Twitter is easy - get a 2nd and 3rd account and switch over when your main account goes to Twitter jail. Yours truly knows about the 200 tweets per hour limit well. So follow holgermu1, too. Just in case. Happened twice this year - so far. And I won't beat 400 tweets per hour - promised... now wait...

 

Holger Mueller
Vice President and Principal Analyst Constellation Research
Holger Mueller is VP and Principal Analyst for Constellation Research for the fundamental enablers of the&nbsp;cloud, IaaS, PaaS and next generation Applications, with forays up the tech stack into BigData and Analytics, HR Tech, and sometimes SaaS. Holger provides strategy and counsel to key clients, including Chief Information Officers, Chief Technology Officers, Chief Product Officers, Chief HR Officers, investment analysts, venture capitalists, sell-side firms, and technology buyers.<br>&nbsp; Coverage Areas: Future of Work Tech Optimization &amp; Innovation<br>&nbsp; Background: Before joining Constellation Research, Mueller was VP of Products for NorthgateArinso, a KKR company. There, he led the transformation of products to the cloud and laid the foundation for new Business… Read more
Published
September 06, 2013
Author
Holger Mueller
Categories
Business Themes
New C-Suite Tech Optimization
Audience Role
Chief Customer Officer Chief Executive Officer Chief Information Officer
Related Blog Posts

Results

Analyst Blog
January 20, 2026

What's Agentic Commerce Without Infrastructure? (Distillation Aftershots)

Future of Work
In this issue, we have to talk about agentic commerce, given how much it was discussed at NRF this past week. The last time we addressed it was an intro to the topic; this time, we have more in-depth links for you to read. First, my take.
Esteban Kolsky
Analyst Blog
January 15, 2026

How Should Executives Think of AI in 2026?

Digital Safety, Privacy, and Cybersecurity
Plenty has been said about how AI should be done over the next 10+ years. What about today&amp;#039;s concerns for the next 12 months? How should executives discuss AI in the enterprise for the next 12 months? What are the questions they should be asking, and the steps they should be taking? What are the biggest concerns?
Esteban Kolsky