Cloud resilience beyond high availability

When an organization moves to the cloud, one of the first architecture decisions is to deploy across several availability zones. It is a sound decision: a zone is one or more data centers with independent power, cooling and networking, and spreading the load across zones protects against the loss of any one of them. The problem starts when that decision is presented to the risk committee as "the continuity plan".

Multi-zone high availability answers a specific question: what happens if one site fails? Resilience answers a broader one: what happens if anything the service depends on fails, including the whole region, a managed service, the identity provider or a corrupted record that was replicated everywhere? This article walks through that gap and how to close it.

High availability is not resilience

The cloud providers' own frameworks draw a clear line between the two. The reliability pillar of the AWS Well-Architected Framework distinguishes between a workload's availability and its disaster recovery, and notes that a strategy built for losing one data center falls short if your definition of disaster includes losing a Region. The Azure Well-Architected Framework makes the same point in its reliability pillar: design for resilience and, separately, design for recovery.

In practice, multi-zone leaves three common gaps:

• Regional failures. They are rare, but they happen: a problem in a regional service, in the control plane or in the backbone network hits every zone at once.
• Data corruption and deletion. Synchronous replication across zones copies the error just as efficiently as it copies good data. An accidental deletion or malicious encryption reaches every replica in seconds.
• Change errors. A faulty configuration rolled out to every zone at the same time turns redundancy into an impact multiplier.

None of these scenarios is solved by adding a fourth zone. They are solved with isolation, with copies the failure cannot reach and with tested recovery procedures.

The dependencies nobody drew

A service's architecture diagram usually shows load balancers, compute and databases. It rarely shows what all of that needs in order to work. Those hidden dependencies are often the real single point of failure:

• DNS. If failover between regions relies on changing DNS records, the availability of the DNS service, the configured time to live (TTL) and client cache behavior all matter.
• Identity. A regional identity provider or a directory that lives in a single site can keep users, and even operators, from signing in during the contingency.
• Queues and messaging. Messages in flight in a regional queue may become unreachable. Decide in advance whether they are lost, reprocessed or reconciled.
• Managed services. Databases, secret stores or managed functions have their own scope (zonal, regional or global) and their own guarantees. You read that scope in the documentation; you do not assume it.
• Third parties. Payment gateways, verification services, SMS providers. Their continuity plan is part of yours, whether you know it or not.
• Control plane. AWS disaster recovery guidance recommends relying on data plane operations during failover, because the control plane (creating or scaling resources) usually has lower availability design goals and may be exactly what is failing.

A useful exercise is to take one critical service and ask, for each dependency: where does it live, what happens if it does not respond, and how would we find out? The answers usually reveal more risk than any topology review. The same discipline used to find root cause in distributed systems applies here, only before the incident.

RTO and RPO per service, not per platform

The recovery time objective (RTO) is the maximum acceptable time to restore the service. The recovery point objective (RPO) is the maximum amount of data, measured in time, the business accepts losing. A common mistake is to set a single RTO and a single RPO for "the cloud" or for the whole data center.

We recommend defining them per business service and, when a service has flows of different criticality, per flow. Balance inquiries and account-to-account transfers can live in the same application and have very different tolerances. The Azure Well-Architected Framework suggests classifying workloads into criticality tiers and deriving recovery targets from them; it is a good starting point.

Three rules of thumb:

• The business sets the target; Technology says what it costs to meet it. A near-zero RPO has a price in architecture, licensing and operations. That conversation should be explicit.
• A service's RTO cannot be shorter than its dependencies'. If the identity provider takes longer to recover than the application, the real RTO is the identity provider's.
• Targets are recorded next to service levels. If you already have SLAs that measure what the business cares about and SLOs with error budgets, RTO and RPO should live in the same catalog, with the same owner.

Recovery patterns: choose by objective

Cloud providers describe a range of patterns, from the simplest and cheapest to the most complex. The table summarizes the most common ones. Values are qualitative and relative to each other: actual RTO and RPO depend on the implementation, on data volume and, above all, on what testing shows.

Pattern	How it works	Typical RTO	Typical RPO	Cost and complexity
Backup and restore	Copies of data, configuration and infrastructure as code in another region or account; everything is rebuilt on activation	Highest	Depends on backup frequency	Low
Active-passive cold (pilot light)	Data replicated continuously; compute exists only as a definition and is switched on at activation	Medium	Low	Moderate
Active-passive warm (warm standby)	A complete, working copy at reduced capacity; activation only requires scaling up	Low	Low	Moderate to high
Multi-region active-active	Several regions serve traffic at once; losing one redistributes the load	Near zero for a regional failure	Near zero with replication, but not for data corruption	Highest

Two warnings the providers' documentation repeats and that are worth underlining. First, no replication pattern replaces point-in-time backups: if the data is corrupted, so is the replica. Second, active-active is not "the best pattern", just the most expensive. It makes sense for the few services whose RTO demands it; for the rest, a warm standby or a well-tested backup is usually the right call.

In financial services, the choice of pattern and secondary region also has regulatory implications (data residency, provider concentration, notifying supervisors). We cover this in cloud governance in financial services.

Test for real: game days, chaos and restores

A recovery plan that has not been tested is a hypothesis. Three kinds of testing complement each other.

Game days

A game day is a planned exercise in which a team simulates a specific failure (losing a region, an identity provider outage, an expired certificate) and runs the recovery procedure with a stopwatch in hand. The Azure Well-Architected Framework recommends combining tabletop exercises, dry runs in non-production environments and production drills, because only the latter verify whether RTO and RPO are met under real conditions.

Chaos engineering

The principles of chaos engineering, published at principlesofchaos.org, propose an experimental method: define the system's steady state with output metrics (throughput, error rate, latency), hypothesize that this state will hold, introduce failures that mimic real events and look for differences. They stress minimizing the blast radius and automating experiments so they run continuously. In regulated organizations we recommend starting outside production, with a limited scope and with whatever approval your change enablement process requires.

Real restores

The data integrity chapter of Google's SRE book sums it up in one idea: nobody wants backups; what people want are restores. Confirming that the backup job finished "green" proves nothing. What proves something is restoring periodically into an isolated environment, validating the integrity of the restored data and measuring how long it took. That measured time, not the estimate, is what you compare against the RTO.

Where to start

A reasonable sequence for an organization that already runs multi-zone:

1. Pick the critical services. Start with three to five business services, not the whole platform.
2. Agree RTO and RPO with the business owner of each service, and record the estimated cost of meeting them.
3. Map the hidden dependencies of each service: DNS, identity, queues, managed services, third parties and control plane.
4. Choose the pattern that meets the target at the lowest cost and complexity, without forgetting point-in-time backups.
5. Test and measure. Real restores first, game days next, chaos experiments once the process is mature.
6. Fix and repeat. Every test produces findings; treat them as problems with an owner and a date, and test again.

Continuity plans with regulatory impact should be validated with Risk and Compliance before they are presented to the supervisor.