End-User Guide: Upgrading MongoDB Replica Set to 8.x for CX Solution ( UNVERIFIED )

Audience: System Administrators & DevOps Engineers Time Estimate: 1-2 hours (depending on environment)

This guide provides a comprehensive, step-by-step procedure for upgrading a high-availability MongoDB replica set from version 6.x to 8.x within a Kubernetes environment. The process uses a rolling upgrade strategy to ensure minimal downtime.

Section 1: Pre-Upgrade Checklist & Critical Precautions

Careful preparation is the key to a successful upgrade. Do not skip these steps.

✅ 1.1: Critical - Back Up Your Database

This is the most important step. Before making any changes, you must have a complete, verified backup of your MongoDB database.

Action: Follow your organization's established backup procedures. Recommended methods are:
- Mongo, PostgreSQL Backup/Restore Procedure for EF-CX on Kubernetes (Link placeholder)
- Kubernetes Backup/Restore using Velero (Link placeholder)
Verification: Confirm that the backup process completed successfully and that you know the procedure for restoring from it if necessary.

✅ 1.2: Verify CX Solution and MongoDB Version

Action (CX Version): Ensure you are running CX 4.6 Release or newer. This guide is not compatible with earlier versions.

Action (MongoDB Version): Connect to your database and verify you are running a 6.x version and that the replica set is healthy.

# (Instructions to connect to your current mongo shell)
# Once inside the shell, run these commands:

# This should return a version string starting with "6."
db.version()

# This command checks the health of your replica set.
rs.status()

Verification: The output of rs.status() should show:
- One member with the state PRIMARY.
- All other members with the state SECONDARY.
- No members in an error, unknown, or down state.
- The "ok": 1 field at the end of the output.
If your replica set is not healthy, resolve all issues before proceeding.

✅ 1.3: Schedule a Maintenance Window

Although this guide uses a rolling upgrade to minimize downtime, it is best practice to perform the upgrade during a period of low traffic.

Action: Announce a maintenance window to all stakeholders.

Section 2: Stage 1 - Upgrade to MongoDB 7.x

We will first perform a rolling upgrade to version 7.x.

Step 2.1: Prepare Helm Chart for MongoDB 7.x

Download the Helm Chart:

mkdir -p mongodb-v7
cd mongodb-v7
helm pull --untar oci://registry-1.docker.io/bitnamicharts/mongodb --version 15.6.9

Modify values.yaml: Navigate into the chart directory (cd mongodb) and open values.yaml for editing. Update the following sections:

Architecture & Replica Count:

## MongoDB(&reg;) architecture. Allowed values: standalone or replicaset
architecture: replicaset
## Number of replicas in the MongoDB(&reg;) replica set
replicaCount: 3 # Adjust to your replica count

Authentication (Crucial): Use your existing root password. The replica set key should also match your existing key if one is set, or be a new secure key if this is the first time setting it.

auth:
  ## @param auth.rootPassword Password for the root user
  ##
  rootPassword: "Expertflow123" # CHANGE THIS to your existing password

  ## @param auth.replicaSetKey Key for the replica set
  ##
  replicaSetKey: "YourSecureReplicaSetKey123" # CHANGE THIS to a secure key

TLS Configuration: Point the chart to your existing secret containing your TLS certificates.

tls:
  enabled: true
  mTLS:
    enabled: true
  autoGenerated: false
  existingSecret: "mongo-mongodb-ca" # The name of your existing TLS secret
  caCert: "mongodb-ca-cert"
  caKey: "mongodb-ca-key"
  pemChainIncluded: true

Enable StatefulSet:

## @param useStatefulSet Use a StatefulSet instead of a Deployment
##
useStatefulSet: true

Step 2.2: Perform the Rolling Upgrade to 7.x

Execute the Upgrade: This command applies the new chart values and begins the rolling upgrade.
```
helm upgrade --install --namespace ef-external --values ./values.yaml mongo .
```
Verification: Monitor the pod status in a separate terminal. You will see each pod terminate and restart one by one with the new version. Wait for all pods to be in the Running 1/1 state.
```
kubectl get pods --namespace ef-external -l app.kubernetes.io/instance=mongo -w
```

Step 2.3: Update Feature Compatibility Version (FCV) to 7.0

Launch a MongoDB 7.x Client Pod:

export MONGODB_ROOT_PASSWORD=$(kubectl get secret --namespace ef-external mongo-mongodb -o jsonpath="{.data.mongodb-root-password}" | base64 -d)

kubectl apply -f - << EOF
apiVersion: v1
kind: Pod
metadata:
  name: mongo7-mongodb-client
  namespace: ef-external
spec:
  containers:
  - command: ["/bin/sh"]
    args: ["-c", "mkdir /tmp/mongodb_certs && cd /tmp/mongodb_certs && openssl req -nodes -newkey rsa:2048 -keyout client.key -out client.csr -subj '/C=SW/ST=Bern/L=Jagerweg/O=efcx/OU=EfCx/CN=mongo' && openssl x509 -req -sha256 -days 3650 -in client.csr -CA /tmp/CERTS/mongodb-ca-cert -CAkey /tmp/CERTS/mongodb-ca-key -set_serial 01 -out client.crt && cat /tmp/CERTS/mongodb-ca-cert /tmp/CERTS/mongodb-ca-key > combined.pem && cat client.crt client.key > client.pem && sleep infinity"]
    env:
    - name: MONGODB_ROOT_PASSWORD
      value: $MONGODB_ROOT_PASSWORD
    image: docker.io/bitnami/mongodb:7.0.11-debian-12-r0
    name: mongo7-mongodb-client
    volumeMounts:
    - mountPath: /tmp/CERTS
      name: mongo-certs
  volumes:
  - name: mongo-certs
    secret:
      secretName: mongo-mongodb-ca
  restartPolicy: Always
EOF

Connect to the Replica Set Primary:

kubectl -n ef-external exec -ti mongo7-mongodb-client -- bash

# Inside the client pod, run:
mongosh admin --host "mongo-mongodb" --authenticationDatabase admin -u root -p $MONGODB_ROOT_PASSWORD --tls --tlsAllowInvalidHostnames --tlsAllowInvalidCertificates --tlsCertificateKeyFile /tmp/mongodb_certs/client.pem --tlsCAFile /tmp/mongodb_certs/combined.pem

Verify and Set FCV: Inside the mongosh shell:

// Verification: Check the current FCV. It should report "6.0".
db.adminCommand( { getParameter: 1, featureCompatibilityVersion: 1 } )
// Expected output: { "featureCompatibilityVersion" : { "version" : "6.0" }, ... }

// Action: Set the FCV to "7.0".
db.adminCommand( { setFeatureCompatibilityVersion: "7.0", confirm: true } )

Verification: The setFeatureCompatibilityVersion command must return an "ok": 1 status. Do not proceed if it fails.

Cleanup:

// Exit the mongo shell
quit()
```bash
# Exit the client pod
exit

# Delete the client pod
kubectl -n ef-external delete pod mongo7-mongodb-client
cd ..

Section 3: Stage 2 - Upgrade to MongoDB 8.x

The process is identical to Stage 1, but uses the 8.x versions.

Step 3.1: Prepare Helm Chart for MongoDB 8.x

Download the Helm Chart:

mkdir -p mongodb-v8
cd mongodb-v8
helm pull --untar oci://registry-1.docker.io/bitnamicharts/mongodb --version 16.4.4

Modify values.yaml: Navigate into cd mongodb and edit values.yaml with the exact same architecture, replicaCount, auth, tls, and useStatefulSet values you used in Step 2.1.

Step 3.2: Perform the Rolling Upgrade to 8.x

Execute the Upgrade:

helm upgrade --install --namespace ef-external --values ./values.yaml mongo .

Verification: Monitor the pods until all have been upgraded and are Running.

kubectl get pods --namespace ef-external -l app.kubernetes.io/instance=mongo -w

Step 3.3: Update Feature Compatibility Version (FCV) to 8.0

Launch a MongoDB 8.x Client Pod:

export MONGODB_ROOT_PASSWORD=$(kubectl get secret --namespace ef-external mongo-mongodb -o jsonpath="{.data.mongodb-root-password}" | base64 -d)

kubectl apply -f - << EOF
apiVersion: v1
kind: Pod
metadata:
  name: mongo8-mongodb-client
  namespace: ef-external
spec:
  containers:
  - command: ["/bin/sh"]
    args: ["-c", "mkdir /tmp/mongodb_certs && cd /tmp/mongodb_certs && openssl req -nodes -newkey rsa:2048 -keyout client.key -out client.csr -subj '/C=SW/ST=Bern/L=Jagerweg/O=efcx/OU=EfCx/CN=mongo' && openssl x509 -req -sha256 -days 3650 -in client.csr -CA /tmp/CERTS/mongodb-ca-cert -CAkey /tmp/CERTS/mongodb-ca-key -set_serial 01 -out client.crt && cat /tmp/CERTS/mongodb-ca-cert /tmp/CERTS/mongodb-ca-key > combined.pem && cat client.crt client.key > client.pem && sleep infinity"]
    env:
    - name: MONGODB_ROOT_PASSWORD
      value: $MONGODB_ROOT_PASSWORD
    image: docker.io/bitnami/mongodb:8.0.4-debian-12-r3
    name: mongo8-mongodb-client
    volumeMounts:
    - mountPath: /tmp/CERTS
      name: mongo-certs
  volumes:
  - name: mongo-certs
    secret:
      secretName: mongo-mongodb-ca
  restartPolicy: Always
EOF

Connect and Set FCV:

kubectl -n ef-external exec -ti mongo8-mongodb-client -- bash

# Inside the client pod, run:
mongosh admin --host "mongo-mongodb" --authenticationDatabase admin -u root -p $MONGODB_ROOT_PASSWORD --tls --tlsAllowInvalidHostnames --tlsAllowInvalidCertificates --tlsCertificateKeyFile /tmp/mongodb_certs/client.pem --tlsCAFile /tmp/mongodb_certs/combined.pem

Inside the mongosh shell:

// Verification: Check the current FCV. It should report "7.0".
db.adminCommand( { getParameter: 1, featureCompatibilityVersion: 1 } )

// Action: Set the FCV to "8.0".
db.adminCommand( { setFeatureCompatibilityVersion: "8.0", confirm: true } )

Verification & Cleanup: Ensure the command returns "ok": 1. Then, exit and delete the client pod.
```
quit()
```bash
exit
kubectl -n ef-external delete pod mongo8-mongodb-client
```

Section 4: Post-Upgrade - Reconnecting CX Applications

Your applications need to be updated to connect to the upgraded database securely.

Step 4.1: Recreate Client TLS Secret in Application Namespace

This provides the necessary certificates for your applications to establish a trusted connection.

Extract Certificates and Create Client PEM:

# Create a temporary directory
mkdir -p /tmp/mongodb_certs_final

# Extract the CA certificate from the server's secret
CERTFILES=($(kubectl get secret mongo-mongodb-ca -n ef-external -o go-template='{{range $k,$v := .data}}{{$k}}{{"\n"}}{{end}}'))
for f in ${CERTFILES[*]}; do
  kubectl get secret mongo-mongodb-ca -n ef-external -o go-template='{{range $k,$v := .data}}{{ if eq $k "'$f'"}}{{$v | base64decode}}{{end}}{{end}}' > /tmp/mongodb_certs_final/${f} 2>/dev/null
done

# Navigate to the directory
cd /tmp/mongodb_certs_final

# Generate a new client key and signing request
openssl req -nodes -newkey rsa:2048 -keyout client.key -out client.csr -subj "/C=SW/ST=Bern/L=Jagerweg/O=efcx/OU=EfCx/CN=mongo"

# Sign the client certificate with the CA
openssl x509 -req -sha256 -days 3650 -in client.csr -CA mongodb-ca-cert -CAkey mongodb-ca-key -set_serial 01 -out client.crt

# Combine the client certificate and key into a single PEM file
cat client.crt client.key > client.pem

Create the Secret in the expertflow Namespace:

kubectl -n expertflow create secret generic mongo-mongodb-ca \
  --from-file=mongodb-ca-cert=./mongodb-ca-cert \
  --from-file=mongodb-ca-key=./mongodb-ca-key \
  --from-file=client-pem=./client.pem \
  --dry-run=client -o yaml | kubectl apply -f -

Step 4.2: Verify Application Connection String

Your applications must use a replica set connection string to leverage high availability.

Action: Check the configuration for your CX deployments (in ConfigMaps, Secrets, or environment variables).
Verification: The connection string must be in the replica set format, listing the headless service addresses of the pods.

Example Format: mongodb://mongo-mongodb-0.mongo-mongodb-headless.ef-external.svc.cluster.local:27017,mongo-mongodb-1.mongo-mongodb-headless.ef-external.svc.cluster.local:27017,mongo-mongodb-2.mongo-mongodb-headless.ef-external.svc.cluster.local:27017/?replicaSet=rs0

Update it if it's pointing to a single host.

Step 4.3: Restart CX Deployments

This forces the applications to pick up the new secret and connection string.

Action:

kubectl -n expertflow rollout restart deploy

Final Verification: Check the logs of your application pods to confirm they connect successfully to MongoDB. Look for any TLS or connection error messages.
```
# Get the name of one of your app pods
kubectl get pods -n expertflow

# Tail the logs of that pod
kubectl logs -n expertflow <your-app-pod-name> -f
```
Perform a full functionality test of your CX solution.

Congratulations! Your MongoDB replica set has been successfully upgraded to version 8.x.