--- title: "From High Availability to a Real Plan B: Lessons from the Latest AWS Outage and a Practical Guide" description: "AWS’s October 20 outage reminded us of a simple truth: resilience can’t be improvised . High availability (HA) within a single region helps, but it doesn’t replace a plan B that keeps services running..." url: https://www.stackscale.com/blog/from-high-availability-to-a-real-plan-b/ date: 2025-10-21 modified: 2025-10-21 author: "Marketing Stackscale" image: https://www.stackscale.com/wp-content/uploads/2022/03/what-is-a-data-center-Stackscale.jpg categories: ["Business continuity", "Computing"] tags: ["high availability", "Infrastructure", "mission critical"] type: post lang: en --- # From High Availability to a Real Plan B: Lessons from the Latest AWS Outage and a Practical Guide AWS’s October 20 outage reminded us of a simple truth: **resilience can’t be improvised**. **High availability (HA)** within a single region helps, but it doesn’t replace a **plan B** that keeps services running when the **common element** too many components depend on fails (region, control plane, DNS, queues, identity, etc.). This guide lays out a **practical path** from “HA” to **business continuity**, with patterns we see working in production—and that we at Stackscale implement with **two active-active data centers**. When required, these architectures **combine smoothly with other providers**. It all starts with the basics: **define RTO/RPO** and **rehearse** failover. > David Carrero (Stackscale co-founder): “HA is necessary, but if everything hinges on a single common point, HA fails. The difference between a scare and a crisis comes down to a rehearsed plan B.” --- ## 1) Align business and tech: RTO/RPO and a dependency map - **RTO (Recovery Time Objective):** how long a service can be **down**. - **RPO (Recovery Point Objective):** how much **data loss** (in time) you can accept when restoring. With those targets **signed off by the business**, draw the **dependency map**: what breaks what? Where do **identity**, **DNS**, **queues**, **catalogs**, or **global tables** really **anchor**? The output is your list of **services that can’t go down**—and **what they depend on**. --- ## 2) Continuity patterns that actually work ### Active–active across two data centers (RTO=0 / RPO=0) For **payments**, **identity**, or **core transactions**: - **[Synchronous storage replication](https://www.stackscale.com/products/synchronous-georeplication-storage/)** between both DCs → **RTO=0 / RPO=0**. - **Distributed / quorum databases**, with conflict handling (CRDTs / sagas). - **DNS/GTM** with **real service health checks** (not just pings). - **Degrade modes** pre-defined (read-only, feature flags). At Stackscale we offer **active–active** across **two European data centers** with **synchronous replication** as a mission-critical foundation. You can **complement** that core with third parties (hyperscalers or other DCs) if you need a **third continuity path** or sovereignty/localization guarantees. ### Multi-site warm standby (hot passive) For most important workloads: - **Asynchronous data replication** to a second site. - **Pre-provisioned infrastructure** (templates / IaC) to promote site B in **minutes**. - **Automatic failover** via **DNS/GTM** and **rehearsed runbooks**. ### Minimum footprint at an alternate provider (formerly “pilot-light”) When you want to **lower concentration risk** at a reasonable cost: - Keep the **bare minimum running** outside your primary provider (e.g., **DNS**, **observability**, **immutable backups**, **break-glass identity**, **status page**). - Promote to **active-passive** or **active-active** only what’s **truly critical**, based on RTO/RPO. > Carrero: “It’s not about abandoning hyperscalers—it’s about balancing. The European ecosystem—Spain included—is mature and a strong complement for resilience and sovereignty.” --- ## 3) Three layers that move the needle (and how to handle them) **Data** - **Transactional:** **quorum-based distribution** + **conflict control**. - **Objects:** **versioning + inter-site replication**, **immutability** (WORM lock) and, where needed, **air-gap**. - **Catalogs/queues:** avoid “global” services **anchored in a single external region** if your RTO/RPO can’t tolerate it. **Network / DNS / CDN** - **Two DNS providers** and **GTM** with **business-level probes** (a real “health transaction,” not a ping). - **Multi-CDN** and **alternate origins** (A/A or A/P) with **origin shielding**. - **Redundant private connectivity** (overlay SD-WAN) between cloud and DC. **Identity & access** - **IdP** with **key (JWKS) caching** and **contextual re-auth**. - **Break-glass accounts** **outside the failure domain**, protected by **strong MFA** (hardware keys). - **App governance** to stop **consent phishing** and **OAuth abuse**. --- ## 4) Observability, backups, and drills: without these, there’s no plan B - **Observability outside the same failure domain:** at least one **mirror** of metrics/logs and a **status page** that **don’t** depend on the primary provider. - **Immutable backups** and **timed restore drills** (with **recent success**). - **Quarterly gamedays:** region/IdP/DNS/queue/DB failures. Measure **real RTO**, **dwell time**, and **MTTR**. --- ## 5) Two reference architectures (and how they fit with Stackscale) ### All-in with Stackscale continuity - **DC A + DC B** (Stackscale) in **active–active** with **synchronous replication** (RTO=0/RPO=0). - **Immutable backups** in a **third domain** (another DC or isolated object storage). - **Multi-provider DNS/DNSSEC** + **GTM** with business health checks. - **Observability** mirrored out (dedicated provider or a second Stackscale site). - **Runbooks** and **regular gamedays** with local, 24/7 support. ### Hybrid continuity (Stackscale + another location/provider) - **DC A + DC B** (Stackscale) **active–active** for the **core**. - **Minimum footprint** at another provider for **DNS**, **status**, **logs/SIEM**, and **immutable object storage** (or vice versa). - **Non-critical workloads** at the other location, with **DR back to Stackscale** if sovereignty or cost requires it. - **Private connectivity** and **portable policies** (identity, logging, backup). **What Stackscale brings** (without the sales pitch): - **Two European data centers** with **low latency**, **redundant power & networks**, and **nearby 24/7 support**. - **Synchronous replication** for mission-critical apps (**RTO=0/RPO=0**). - **High-performance storage** (block/object) with **versioning** and **WORM lock (Object Lock)** for **immutable backups**. - **Bare-metal and private cloud** to consolidate workloads, plus **dedicated connectivity** with carriers and public clouds via partners. - Easy integration with **external providers** (DNS, CDN, observability, hyperscalers) for **hybrid or selective multicloud** strategies. --- ## 6) A realistic 30-60-90 day roadmap **Days 1–30** - Get **RTO/RPO** approved per service. - Build the **dependency & global-anchor map**. - **Immutable backups** and your **first timed restore**. **Days 31–60** - **Multi-provider DNS/GTM**, **multi-CDN**, and **observability outside** the same failure domain. - **Minimum footprint** (emergency identity, status, SIEM). - First **gameday** (DNS + DB/queues). **Days 61–90** - **Active–active** or **warm standby** between Stackscale’s two DCs. - Integrations with **third parties** (if needed). - **Full-region failure gameday** and metrics review. --- ## 7) Executive metrics that actually say something - **% of services** with **signed RTO/RPO** and **met in drills**. - **Restore OK** (last 30 days) and **average restore time**. - **Failover time** (gamedays) and **dwell time** by scenario. - **Observability coverage “outside”** the failure domain (yes/no by domain). - **“Global” dependencies with alternatives** (yes/no). --- ## Final word: resilience with a clear head (and without handcuffs) The lesson isn’t to “flee the cloud,” but to **design for failure** and **de-concentrate** single points of breakage. **HA** is necessary, but not sufficient: you need a **complete alternate route** to the same outcome. With **two active–active DCs** (**RTO=0/RPO=0**) as the base and **continuity layers**—DNS, immutable copies, observability, and break-glass identity—**outside the same failure domain**, your platform will stay up when a provider or region stumbles. At Stackscale, we support that transition every day. And when continuity goals or regulations call for it, we **combine** our **two-data-center** infrastructure with **other providers**. That way, plan B isn’t a PDF in a drawer—it’s a **path you’ve already walked**.