Re: Reliable Cloud host ?

[Kevin Day <toasty () dragondata com>]

On Feb 26, 2012, at 4:56 PM, Randy Carpenter wrote:
> We have been using Rackspace Cloud Servers. We just realized that they have absolutely no redundancy or failover 
> after experiencing a outage that lasted more than 6 hours yesterday. I am appalled that they would offer something 
> called "cloud" without having any failover at all.
>
> Basic requirements:
>
> 1. Full redundancy with instant failover to other hypervisor hosts upon hardware failure (I thought this was a given!)

This is actually a much harder problem to solve than it sounds, and gets progressively harder depending on what you 
mean by "failover".

At the very least, having two physical hosts capable of running your VM requires that your VM be stored on some kind of 
SAN (usually iSCSI based) storage system. Otherwise, two hosts have no way of accessing your VM's data if one were to 
die. This makes things an order of magnitude or higher more expensive.

But then all you've really done is moved your single point of failure to the SAN. Small SANs aren't economical, so you 
end up having tons of customers on one SAN. If it dies tons of VMs are suddenly down. So you now need a redundant SAN 
capable of live-mirroring everyone's data. These aren't cheap either, and add a lot of complexity to things. (How to 
handle failover if it died mid-write, who has the most recent data after a total blackout, etc)

And this is really just saying "If hardware fails, i want my VM to reboot on another host." If what you're defining 
high availability to mean "even if a physical host fails, i don't want a second of downtime, my VM can't reboot" you 
want something like VMware's ESXi High Availability modules where your VM is actually running on two hosts at once, 
running in lock-step with each other so if one fails the other takes over transparently. Licenses for this are 
ridiculously expensive, and requires some reasonably complex networking and storage systems.

And I still haven't touched on having to make sure both physical hosts capable of running your VM are on totally 
independent switches/power/etc, the SAN has multiple interfaces so it's not all going through one switch, etc.

I also haven't run into anyone deploying a high-availability/redundant system where they haven't accidentally ended up 
with a split-brain scenario (network isolation causes the backup node to think it's live, when the primary is still 
running). Carefully synchronizing things to prevent this is hard and fragile.

I'm not saying you can't have this feature, but it's not typical in "reasonably priced" cloud services, and nearly 
unheard-of to be something automatically used. Just moving your virtual machine from using local storage to ISCSI 
backed storage drastically increases disk latency and caps the whole physical host's disk speed to 1gbps (not much 
deployment for 10GE adapters on the low-priced VM provider yet). Any provider who automatically provisions a virtual 
machine this way will get complaints that their servers are slow, which is true compared to someone selling VMs that 
use local storage. The "running your VM on two hosts at once" system has such a performance penalty, and costs so much 
in licensing, you really need to NEED it for it not to be a ridiculous waste of resources.

Amazon comes sorta close to this, in that their storage is mostly-totally separate from the hosts running your code. 
But they have had failures knock out access to your storage, so it's still not where I think you're saying you want to 
be.

The moral of the story is that just because it's "in the cloud", it doesn't gain higher reliability unless you're 
specifically taking steps to ensure it. Most people solve this by taking things that are already distributable (like 
DNS) and setting up multiple DNS servers in different places - that's where all this "cloud stuff" really shines.


(please no stories about how you were able to make a redundant virtual machine run using 5 year old servers in your 
basement, i'm talking about something that's supportable on a provider scale, and isn't adding more 
single-points-of-failure)

-- Kevin