CX High Availability: Across Two Sites

EFCX High Availability Across Two Sites

Introduction

Running a platform across two data centers can be architected two ways: active/active, where both sites stay live and continuously clustered, or active/passive, where one site carries all production traffic and the second is provisioned but stopped until a failover triggers it. The two differ in what infrastructure they require, how failover actually happens, and what it costs to build and run, so this note lays out both before recommending how to choose between them.

This note covers the cross-site axis only: what happens when a whole site is lost. What happens within a single site, a pod, node, or host loss while the site itself stays up, is a separate, independent decision covered in EFCX-High-Availability-Within-a-Site.

Highlights and pros/cons

Option 1, Active/Active: both sites run live at once. Each site runs its own compute cluster, but the platform’s stateful data-tier services (database, cache, message broker) are joined into a single continuously-synchronized cluster spanning both sites, with no promotion step; components outside that data tier keep their own separate cross-site failover mechanism. A witness node in a third location casts the tie-breaking vote for data-tier quorum, so failover there is automatic, an election completes in seconds. That continuous readiness is also what makes it the more expensive, more demanding option: a witness node, a tight inter-DC link, and full standing capacity at both sites all the time.

Option 2, Active/Passive: one site carries all production traffic while the second is kept current through asynchronous replication and stays stopped, or scaled to a minimal footprint, until triggered. There’s no live quorum to arbitrate, so no witness node is needed and the inter-DC link requirement relaxes considerably, replication doesn’t have to stay inside a tight RTT/jitter budget the way live clustering does. The trade-off is recovery speed: bringing the second site up is a sequence, not an election, and can involve a manual promotion step or an orchestrated one.

Option 1: Active/Active

Both sites run live. Each site runs its own compute cluster, but the stateful data-tier services, MongoDB, Redis, PostgreSQL and Artemis, are joined into a single continuously-synchronized cluster spanning both sites, with no promotion step: whichever site holds a given component’s primary, the other stays in sync across the inter-DC link. Components outside that data tier, such as the voice/session-border layer, have their own separate cross-site failover mechanism rather than being part of this cluster. A witness node in a third location casts the tie-breaking vote for the data-tier cluster’s quorum: with exactly two sites and no third vote, a lost inter-site link would otherwise be indistinguishable from “the other site died,” so neither side could safely self-promote without risking two masters both accepting writes.

• Infra required: a witness/arbiter node in a third location, plus an inter-DC link meeting under 50ms RTT, under 30ms jitter, under 0.5% packet loss.

• RTO: seconds, a cluster election once the surviving site and witness hold majority. Fully automatic, no manual intervention.

• RPO: near-zero, bounded only by however asynchronous the cross-site replication turns out to be.

• Cost/complexity: higher. Both sites run full production capacity continuously, and whichever site doesn’t currently hold the cluster’s primary reaches across the inter-DC link for every read/write, all the time, not just during a failure.

Option 2: Active/Passive

One site carries all production traffic. The second site is stopped or scaled to zero, kept current via asynchronous storage-array replication, and only starts when triggered.

• Infra required: an inter-DC link sized for asynchronous replication (no strict RTT/jitter ceiling the way live clustering needs), plus a replication mechanism between sites. Platform-native storage/hypervisor-layer replication is the preferred mechanism (see below); where that isn’t available, database-native replication configured per engine is a fallback, at the cost of separate configuration and monitoring per engine instead of one platform-level mechanism.

• RTO: 7 to 15 minutes end to end: detection, storage handover, service start, traffic switch, to be confirmed by DR testing. Can run as a manual runbook step or as orchestrated automation, either way it’s a sequence, not an election.

• RPO: bounded by replication lag at the moment of failure, whatever was written in the seconds-to-minutes immediately before the failure can be lost.

• Cost/complexity: lower. The passive site doesn’t need full standing capacity, and there’s no continuous cross-site read/write path to budget for.

Replication and failover infrastructure

Covered on its own page: EFCX Replication and Failover Infrastructure. Shared with EFCX-Hardware-Sizing-Across-Two-Sites, which carries the same content locally as efcx-replication-and-failover-infrastructure.

Side by side

Choosing between them

The right choice hinges on what the underlying infrastructure and inter-site link can actually support, and how much downtime and data loss the business can tolerate during a site failure. If the sites are close enough for a sub-50ms, low-jitter link and the extra standing capacity is affordable, active/active gives automatic, near-instant failover. If the link is longer or the second site is meant to stay cost-light until needed, active/passive is the simpler, cheaper option, at the cost of a slower, RTO-bound recovery. Either way, confirm the actual inter-DC link characteristics and the infrastructure platform’s replication/failover capabilities before committing, since both change what each option can realistically deliver.