CX High Availability: Within a Site

Control plane

Multiple master nodes (an odd number, three is the tested minimum) with a clustered consensus store for the control plane’s own state

A single master has no quorum to fall back on if it’s lost. At higher workload, master nodes are typically dedicated to control-plane duty only rather than also scheduling application pods.

Worker nodes

Multiple worker nodes (three is the tested minimum)

On a worker loss, pods reschedule to survivors within Kubernetes’ default relocation window (5 to 10 minutes, deliberately not instant, to avoid pods hopping on a transient blip). Workload does not automatically rebalance back once a recovered node rejoins, only new work lands there; evening the load back out needs a deliberate, planned restart of the affected workloads, which itself causes an outage.

External load balancer

A failover pair fronting the cluster, not a single instance

A single load balancer is a documented single point of failure; production sizing calls for a failover cluster of at least two.

Storage under the CSI layer

Storage that is itself replicated across nodes within the site, not tied to a single node

So a node loss doesn’t strand a volume behind it. Earlier on-prem material documents this as an explicit add-on layer (raw block devices pooled into replicated storage) built specifically for platforms that don’t already provide multi-node storage redundancy; a converged, cloud-native platform typically provides that natively through its own CSI driver instead, without needing a separate add-on. Confirm which is actually the case for the platform in use before assuming either.

MongoDB (conversation and customer data)

N-node replica set plus one arbiter for the tie-breaking vote, replica count set per available worker nodes, anti-affinity keeping the replica off the same node as the primary

On primary-pod loss, a surviving replica is automatically promoted

The arbiter ran out of memory under test, and Mongo also attempted to bring the original primary back online on another node some hours later; the documented fix was to drop the arbiter from the connection string once elections had already been proven to work without it in the loop

Redis (session and cache state)

Sentinel-managed replica set, anti-affinity between replicas

On primary-pod loss, a surviving replica is automatically promoted

None recorded

PostgreSQL (License Manager, Keycloak data)

Managed replica set with automated failover, three replicas by default

On primary-pod loss, a surviving replica is automatically promoted; running in asynchronous mode was the documented performance tuning

None recorded beyond the async-mode tuning note

MinIO (attachments, recordings, object storage)

Distributed mode, an even replica count of four or more, one drive per node, anti-affinity

Tested successfully

None recorded

ActiveMQ / Artemis (message bus)

A single pod backed by replicated storage rather than a peer cluster of its own; recovery depends on the storage layer reattaching to a new pod, not on an application-level election

A new pod is created and takes over the replicated storage after the default relocation window

The replacement pod sometimes got stuck in scheduling and needed a forced pod deletion to recover

Keycloak (authentication)

No built-in HA of its own; sits entirely on top of the HA PostgreSQL above

A new pod is created after the relocation window and reconnects to PostgreSQL

Recovery is gated on PostgreSQL’s own recovery completing first, not just on Keycloak’s pod restarting; the same stuck-pod caveat as ActiveMQ applies if the new pod doesn’t schedule cleanly

CX application pods (conversation manager, routing engine, and the rest of core)

Stateless; conversation and routing state lives in MongoDB, Redis, and the message bus rather than in the pod

A fresh pod picks the work back up once rescheduled

Spin-up issues were seen in two components’ init containers after a cross-node move, worked around at the time by disabling those init containers