VRS PostgreSQL High Availability (Replication) | Voice Recording

VRS PostgreSQL High Availability (Replication) — Team Brief

Product: Voice Recording Solution (VRS)
Audience: Team lead / engineering
Date: 22 Oct 2026
Status: Lab validated; production inputs pending

1. Executive summary

Today production uses one PostgreSQL server with (at least) two databases:

Database	Consumers
`vrs`	Mixer, JTAPI, APIs, Archival
`keycloak_vrs`	Keycloak

We will move to a 3-node self-managed HA cluster (Patroni + etcd + HAProxy) on Recorder 1, Recorder 2, and Replay, so that:

All three sites share one logical Postgres (same vrs + keycloak_vrs databases).
Every confirmed write exists on at least two servers (synchronous replication).
If the primary DB or its server fails, another node becomes primary in ~30–45 seconds (≤ 60 s target).
Applications keep the same connection style after failover: localhost:5000 (read-write) via local HAProxy.

Data is not discarded by replication. Existing data is migrated with a verified backup and a controlled cutover. The old Postgres is kept offline for rollback (≥ 7 days).

2. Goals and non-goals

Goals

Zero loss of confirmed (acknowledged) writes.
Automatic failover; no manual VIP move for apps.
Full copy of data on all three servers.
Same app config forever: point at local HAProxy, not at a fixed primary IP.

Out of scope (separate work)

Replicating recording audio files (WAV) — DB HA does not copy media.
Active-active multi-master writes.
Kubernetes.
Off-site DR site (pgBackRest off-box is in scope; remote DR cluster is not).

3. Target architecture

Recorder 1                    Recorder 2                    Replay
+------------------+          +------------------+          +------------------+
| VRS apps         |          | VRS apps         |          | APIs / Archival  |
| → localhost:5000 |          | → localhost:5000 |          | → :5000 RW       |
| [HAProxy]        |          | [HAProxy]        |          | → :5001 RO       |
+--------+---------+          +--------+---------+          +--------+---------+
         |                             |                             |
         +--------------+--------------+--------------+--------------+
                        |     all RW traffic → CURRENT primary
         +--------------v--------------v--------------v--------------+
         |  PG+Patroni PRIMARY  ===sync==>  STANDBY   ===sync==>  STANDBY |
         |  etcd (quorum 2 of 3) — leader lock, prevents split-brain       |
         +-----------------------------------------------------------------+
                        |
                        +--> pgBackRest backups (NFS / MinIO — outside the 3 nodes)

Component	Role
PostgreSQL + Patroni	Database; automatic promote / rejoin
etcd (3 members)	Distributed lock; 2-of-3 quorum
HAProxy	`:5000` → primary only; `:5001` → read-only; health via Patroni `:8008`

What runs on each host today vs with DB HA

Today (no DB HA): Postgres is typically a single server (one host). Recorders / Replay run VRS apps and connect to that one Postgres (often 5432 on that DB host).

With DB HA: each of the three VRS servers (Recorder 1, Recorder 2, and Replay) runs the same set of DB-stack components, in addition to whatever VRS apps already run there:

On every host (×3)	What it is	Notes
1× PostgreSQL + Patroni	DB instance managed by Patroni	Only one of the three is primary at a time; the other two are standbys (full data copy)
1× etcd	Cluster membership / leader lock	Needs 2 of 3 etcd members up (quorum)
1× HAProxy	Local DB entrypoint for apps	Apps use `localhost:5000` (RW) / `:5001` (RO); HAProxy sends RW to the current primary

Plus existing VRS workloads (unchanged by this design):

Host	Typical apps (examples)	DB connection after HA
Recorder 1	Mixer, JTAPI, recorder services, …	→ local HAProxy `:5000`
Recorder 2	Same class of recorder services	→ local HAProxy `:5000`
Replay	APIs, Archival, search/replay, …	→ local HAProxy `:5000` (RW); optional `:5001` for read-only

So per machine you add 3 processes/containers for the DB stack (Postgres+Patroni, etcd, HAProxy). Across the fleet that is 9 containers total for HA, replacing the single standalone Postgres server.

Important: Which recorder handles calls and which node is DB primary are independent. Moving call traffic to Recorder 2 does not require moving the DB primary.

4. Replication design (fixed decisions)

Decision	Value	Meaning
Topology	1 primary + 2 standbys	Single writer
Sync	`synchronous_mode: true`, `synchronous_node_count: 1`	Commit waits for any 1 of 2 standbys
Strict	`synchronous_mode_strict: false`	If both standbys down, primary still accepts writes (async)
Failover timers	ttl 30 / loop_wait 10 / retry_timeout 10	Unplanned failover ~30–45 s
Rejoin	`use_pg_rewind: true` + data checksums	Old primary rejoins without full re-copy
App endpoint	Local HAProxy only	Apps never chase the primary IP

Failure behaviour (expected)

Failure	Result	Confirmed data
Primary container crash	Failover if not back before TTL	No loss
Primary server down	Failover ~30–45 s	No loss
One standby down	Other standby becomes sync	No loss
Both standbys down	Primary continues (non-strict)	Risk only if primary then dies
Network isolates primary	Isolated node demotes; majority promotes	No split-brain

5. What changes for VRS applications?

No new Patroni/etcd libraries in recorder code.

Only connection settings (per host):

# VRS metadata DB (mixer, jtapi, apis, archival)
DB_HOST=localhost
DB_PORT=5000
DB_NAME=vrs
DB_USER=...
DB_PASSWORD=...

# Keycloak (same cluster, different database)
KC_DB=postgres
KC_DB_URL=jdbc:postgresql://localhost:5000/keycloak_vrs
KC_DB_USERNAME=...
KC_DB_PASSWORD=...

Recommended JDBC extras (APIs)

?targetServerType=primary&connectTimeout=5&socketTimeout=60&tcpKeepAlive=true

Pool should validate connections so dead sockets are dropped after failover.

Vendor / product behaviour to confirm

Retry DB writes ≥ 60 seconds on connection errors.
Retried inserts idempotent (unique call id / ON CONFLICT).
Replay may use :5001 for read-only searches if desired.

6. Existing production data — will anything be lost?

Situation	Outcome
Correct migration (7A / 7B / 7C) + verified backup	No loss of existing `vrs` / `keycloak_vrs` data
Streaming replication / Patroni clone	Copies data; does not delete it
Wipe volumes / `patronictl reinit` / wrong empty cluster	Can destroy — forbidden without explicit approval
Cutover to empty cluster by mistake	Looks like lost (wrong target)

Both databases (vrs and keycloak_vrs) live on the same Postgres instance today and will live on the same Patroni cluster after cutover.

Recommendation: keep Keycloak on the same replicated Postgres unless login isolation is required.

7. Migration plan (high level)

Phase A — Prepare (no cutover)

Fill inventory (IPs, disks, PG version, backup path).
Full backup of current Postgres; test-restore once.
Install Docker/Compose, disks, time sync on all three servers.
Build Patroni image; start etcd → Patroni → HAProxy on all three.
Prove lab / staging: 1 leader + 2 streaming replicas.

Phase B — Cutover (maintenance window)

Option	When	Downtime
7A In place	Current DB already on one of the 3 servers; same PG major	Minutes
7B Dump/restore	DB elsewhere, or version change, or small DB	Dump + restore time
7C Logical replication	Large DB; need short final switch	Minutes at switch

Cutover steps: stop writers → final backup → migrate both DBs → verify → point apps to localhost:5000 → start apps → keep old Postgres ≥ 7 days.

Phase C — Prove & operate

Failover tests; pgBackRest off-box; runbooks.

8. Lab proof (already done)

Result: 1 Leader + 2 Replicas streaming, lag 0.
Stack: PostgreSQL 17 + Patroni 4.1.3 + etcd v3.5.21 + HAProxy.
Repo: vrs-pg-ha/lab/patroni-3node/

9. Host prerequisites (each of 3 servers)

Linux x86_64; Docker Engine ≥ 24 + Compose v2; time sync; static IPs; dedicated PG disk; firewall between nodes only (5432, 8008, 2379, 2380, local 5000/5001); enough RAM for Postgres and apps.

10. Open inputs (blocked for production deploy)

Recorder1/2/Replay IPs; subnet; PG major; paths; EXISTING_DB_HOST; secrets; maintenance windows; vendor retry/idempotency/Keycloak answers.

11. Risks and mitigations

Risk	Mitigation
Cutover mistake	Verified backup; old DB rollback window
App does not retry 60 s	Confirm with product
Audio on dead recorder	Separate media replication
Port 5000 conflict	Map free host port in lab
Disk full (WAL)	Slots + monitoring

12. Recommendation for team lead

Approve Patroni + etcd + HAProxy on all three servers; apps → localhost:5000.
Keep vrs + keycloak_vrs on the same HA cluster.
Migrate existing data (not empty cluster).
Fill inventory; staging rehearsal; production window.
Run acceptance suite before go-live.

13. References

vrs-pg-ha/docs/SPEC.md
vrs-pg-ha/lab/patroni-3node/README.md
vrs-pg-ha/lab/patroni-3node/scripts/acceptance-tests.sh

Prepared for Expertflow VRS engineering — PostgreSQL HA / replication briefing.