The beginning of July was a big week for data center outages with some notable Internet impact. I've written before about cloud resiliency and uptime in my 'Changing a tire while going 60mph' and 'Big, white puffy clouds can still evaporate' posts.
Amongst those outages was a Google App Engine interruption, which just goes onto the tally sheet as yet another failure of a supposed infallible cloud platform service. Ted Dziuba has some interesting viewpoints on the App Engine outage, and brings up a key element that makes all the difference with these types of outages: transparency. If something fails, people like to know why it happened (especially in light of all the resiliency/redundancy claims of the cloud) and what is being done so it doesn't happen again. Google's transparency was a basic statement that something bad happened with GFS, and that affected BigTable, which affected App Engine's DataStore, and they are going to fix it. Now, compare that level of detail to the information offered by Amazon during their S3 outage from July 2008 and February 2008. There is a notable difference. So a message to SaaS vendors: just because customers don't need to know the platform and implementation details behind your SaaS, doesn't mean they don't want to know...especially in light of a catastrophic failure. Telling customers what went wrong let's them enact their own post-mortem and due-diligence analysis processes to ensure they haven't shackled themselves to a SaaS vendor who is experiencing outages due to silly oversights that are likely to repeat as life goes on.
Actually, transparency and efficient communication regarding the outage event was also a key issue for some of the other outages last week. Jeremy Irish blogged about the fire at Fisher Plaza data center that left his systems offline for 29 hours. Apparently news of what exactly was happening, even for the first responders that were physically present at the building, was scarce and little effort was made to keep folks efficiently informed. Eventually they found out that it was a fire in a basement-level electrical room, which triggered the sprinklers and in turn ruined additional electrical equipment and generators, which ultimately caused significant damage to the power infrastructure.
Overall, Murphy and his laws are a fact of life and vendors (cloud, SaaS, or other) must do what is in their power to have contingency plans if you truly want to maintain availability and business continuity. And SaaS vendors, take note: just because you can hide the implementation and platform details of your service, doesn't mean you should hide the operational details of your service...especially during a service disruption.
Until next time,
- Jeff
Showing posts with label cloud. Show all posts
Showing posts with label cloud. Show all posts
Tuesday, July 7, 2009
Thursday, March 26, 2009
Changing a tire while going 60mph
In my previous post Big, white, puffy clouds can still evaporate, I mentioned the Google Gmail outage from February and pointed out the implied resiliency (or lack there-of) of cloud-based platforms. Rich Miller over at DataCenterKnowledge.com just posted an interview with Urs Holzle from Google entitled "How Google Routes Around Outages." The title of this post is a quote from the referenced article, and it seems rather fitting.
On the one hand, I was a bit surprised to read Holzle admit the preference to have human involvement in remediating certain types of outages. It seems Google has automated technological controls to handle single system and small-scale failures (a whole equipment rack, etc.) within a single data center; but larger scale outages (such as a whole data center) are mitigated manually. On the other hand, I can understand the desire to have explicit human oversight; large-scale failures tend to have unpredictable cascading effects that are hard to account for with automated systems. In fact, Holzle mentions that the very systems designed to handle larger-scale outages tend to be the same systems responsible for the outages; the February GMail outage is an explicit example.
Having said that, I still believe clouds need to be as self-sustaining as possible (preferably in automated fashion) despite the current limits of technology and engineering. That implied resiliency and redundancy is one of the core value propositions of clouds. Without it, clouds get devalued into not much more than a traditional managed hosting platform/service.
My suggestion to organizations considering a purchase from a cloud-based vendor is to have a due-diligence discovery session regarding the cloud architecture that vendor has engineered. You want to see something akin to a fairly detailed Visio diagram depicting multiple datacenter/regional locations, with appropriate systems in each location and the roles they provide to the overall cloud platform and offered services. You want to see redundancy in systems and roles, with distribution and backups across disparate regions. Look for points of failure, and ask them how the entire cloud handles an outage and thwarts a service interruption during the failure for any given node you happen to point to. Especially point to an entire datacenter/region and ask what happens if that entire area of the cloud system goes down due large-scale power outage, natural disaster, etc. Also be wary of how cloud entrance points could become affected during an outage; it is great if the overall service will naturally shift processing to a secondary region if the entire first region experiences an outage, but it is also not so great if you have to explicitly go back and reconfigure your organization to point to the secondary region. Even though the processing survived, your point of entrance changed and thus you still had an outage. You definitely want to ensure all points of presence/entrance are fault tolerant.
Overall, going with a cloud vendor still requires you to think about redundancy, fault tolerance, business continuity, etc. And it might require a little extra effort up front in order to ferret out the appropriate details from your cloud vendor during the initial evaluation. Fortunately, once you are satisfied with the resiliency offered by the vendor's cloud architecture, you can then focus your attention elsewhere--the vendor is the one that has to deal with the implementation and ongoing maintenance of the necessary architecture redundancy. And that is one headache that is definitely nice to outsource.
Until next time,
- Jeff
On the one hand, I was a bit surprised to read Holzle admit the preference to have human involvement in remediating certain types of outages. It seems Google has automated technological controls to handle single system and small-scale failures (a whole equipment rack, etc.) within a single data center; but larger scale outages (such as a whole data center) are mitigated manually. On the other hand, I can understand the desire to have explicit human oversight; large-scale failures tend to have unpredictable cascading effects that are hard to account for with automated systems. In fact, Holzle mentions that the very systems designed to handle larger-scale outages tend to be the same systems responsible for the outages; the February GMail outage is an explicit example.
Having said that, I still believe clouds need to be as self-sustaining as possible (preferably in automated fashion) despite the current limits of technology and engineering. That implied resiliency and redundancy is one of the core value propositions of clouds. Without it, clouds get devalued into not much more than a traditional managed hosting platform/service.
My suggestion to organizations considering a purchase from a cloud-based vendor is to have a due-diligence discovery session regarding the cloud architecture that vendor has engineered. You want to see something akin to a fairly detailed Visio diagram depicting multiple datacenter/regional locations, with appropriate systems in each location and the roles they provide to the overall cloud platform and offered services. You want to see redundancy in systems and roles, with distribution and backups across disparate regions. Look for points of failure, and ask them how the entire cloud handles an outage and thwarts a service interruption during the failure for any given node you happen to point to. Especially point to an entire datacenter/region and ask what happens if that entire area of the cloud system goes down due large-scale power outage, natural disaster, etc. Also be wary of how cloud entrance points could become affected during an outage; it is great if the overall service will naturally shift processing to a secondary region if the entire first region experiences an outage, but it is also not so great if you have to explicitly go back and reconfigure your organization to point to the secondary region. Even though the processing survived, your point of entrance changed and thus you still had an outage. You definitely want to ensure all points of presence/entrance are fault tolerant.
Overall, going with a cloud vendor still requires you to think about redundancy, fault tolerance, business continuity, etc. And it might require a little extra effort up front in order to ferret out the appropriate details from your cloud vendor during the initial evaluation. Fortunately, once you are satisfied with the resiliency offered by the vendor's cloud architecture, you can then focus your attention elsewhere--the vendor is the one that has to deal with the implementation and ongoing maintenance of the necessary architecture redundancy. And that is one headache that is definitely nice to outsource.
Until next time,
- Jeff
Friday, February 27, 2009
Big, white, puffy clouds can still evaporate
If you haven't heard by now, Google GMail had a 2.5 hour outage on Tuesday. Acacio Cruz, Gmail’s Site Reliability Manager, explained in a little more detail regarding the outage. My interpretation of his explanation is that they had a bug in how data is shifted around in the cloud, and during a shift one of data centers couldn't handle the new burden, failed, and eventually the effect cascaded.
I have a few comments on this. First, GMail still carries the 'beta' moniker...and aren't service disruptions like this kind of expected/tolerated with beta software? But beta or not, real people use the service--including Google themselves (according to Acacio Cruz's post). So maybe it's time to promote GMail out of beta and designate it as production, since that's what it is to most people anyways.
Second, running a resilient cloud is hard, particularly when it comes to outages and the resulting automatic rebalancing to cover the gap and maintain availability. It's like a tire on a car: everything rotates routinely and drives smoothly when the tire is balanced. But if the center of balance of the tire shifts dramatically at a random moment, things quickly deteriorate and the high-speed rotation can cause significant instability and damage to the rest of the vehicle. Computing clouds are conceptually similar. What happens when an outage takes out a piece of the cloud and the finely orchestrated balance is randomly skewed...is the capacity of the rest of the cloud sufficient to absorb the load? Can the cloud rebalance itself gracefully? These are the same types of questions one will hear when investigating fault tolerance, failover, and redundancy capabilities of any normal technology component (servers, firewalls, WAN links, etc.). But the questions take on a whole new level of scale when you apply them to a cloud.
Plus let's not overlook the fact that we have different fault tolerance expectations for services in the cloud. The typical perception of clouds seems to imply internal failover capabilities. After all, that's part of the appeal of clouds: all that fault tolerance, redundancy, etc. is magically taken care of within the cloud. The cloud is infallible. The cloud cannot fail. Those are the dominant marketing messages. No one talks about having a contingency plan for when the cloud is unavailable (what would that be exactly? A second hot-spare cloud?).
So when choosing a cloud-based vendor, you should try to investigate their cloud resiliency. Vendors who take non-cloud specific software, run copies of it in two or three datacenters, and call it a cloud service are likely not delivering the true expected experience to customers. Software needs to be purpose-built to run in a cloud, including multi-tenancy design and strong internal failover capability. That's because most people wind up putting all of their eggs into the cloud basket. If you are going to be one of those people, you should try to make sure the basket offers the implied safety that you expect.
Until next time,
- Jeff
I have a few comments on this. First, GMail still carries the 'beta' moniker...and aren't service disruptions like this kind of expected/tolerated with beta software? But beta or not, real people use the service--including Google themselves (according to Acacio Cruz's post). So maybe it's time to promote GMail out of beta and designate it as production, since that's what it is to most people anyways.
Second, running a resilient cloud is hard, particularly when it comes to outages and the resulting automatic rebalancing to cover the gap and maintain availability. It's like a tire on a car: everything rotates routinely and drives smoothly when the tire is balanced. But if the center of balance of the tire shifts dramatically at a random moment, things quickly deteriorate and the high-speed rotation can cause significant instability and damage to the rest of the vehicle. Computing clouds are conceptually similar. What happens when an outage takes out a piece of the cloud and the finely orchestrated balance is randomly skewed...is the capacity of the rest of the cloud sufficient to absorb the load? Can the cloud rebalance itself gracefully? These are the same types of questions one will hear when investigating fault tolerance, failover, and redundancy capabilities of any normal technology component (servers, firewalls, WAN links, etc.). But the questions take on a whole new level of scale when you apply them to a cloud.
Plus let's not overlook the fact that we have different fault tolerance expectations for services in the cloud. The typical perception of clouds seems to imply internal failover capabilities. After all, that's part of the appeal of clouds: all that fault tolerance, redundancy, etc. is magically taken care of within the cloud. The cloud is infallible. The cloud cannot fail. Those are the dominant marketing messages. No one talks about having a contingency plan for when the cloud is unavailable (what would that be exactly? A second hot-spare cloud?).
So when choosing a cloud-based vendor, you should try to investigate their cloud resiliency. Vendors who take non-cloud specific software, run copies of it in two or three datacenters, and call it a cloud service are likely not delivering the true expected experience to customers. Software needs to be purpose-built to run in a cloud, including multi-tenancy design and strong internal failover capability. That's because most people wind up putting all of their eggs into the cloud basket. If you are going to be one of those people, you should try to make sure the basket offers the implied safety that you expect.
Until next time,
- Jeff
Subscribe to:
Posts (Atom)
Posts
