Skip to main content
Skip table of contents

End-User Guide: Upgrading MongoDB Replica Set to 8.x for CX Solution ( UNVERIFIED )

Audience: System Administrators & DevOps Engineers Time Estimate: 1-2 hours (depending on environment)

This guide provides a comprehensive, step-by-step procedure for upgrading a high-availability MongoDB replica set from version 6.x to 8.x within a Kubernetes environment. The process uses a rolling upgrade strategy to ensure minimal downtime.

Section 1: Pre-Upgrade Checklist & Critical Precautions

Careful preparation is the key to a successful upgrade. Do not skip these steps.

✅ 1.1: Critical - Back Up Your Database

This is the most important step. Before making any changes, you must have a complete, verified backup of your MongoDB database.

✅ 1.2: Verify CX Solution and MongoDB Version

  • Action (CX Version): Ensure you are running CX 4.6 Release or newer. This guide is not compatible with earlier versions.

  • Action (MongoDB Version): Connect to your database and verify you are running a 6.x version and that the replica set is healthy.

    CODE 
    # (Instructions to connect to your current mongo shell)
# Once inside the shell, run these commands:

# This should return a version string starting with "6."
db.version()

# This command checks the health of your replica set.
rs.status()

  • Verification: The output of rs.status() should show:

    • One member with the state PRIMARY.

    • All other members with the state SECONDARY.

    • No members in an error, unknown, or down state.

    • The "ok": 1 field at the end of the output.

    If your replica set is not healthy, resolve all issues before proceeding.

✅ 1.3: Schedule a Maintenance Window

Although this guide uses a rolling upgrade to minimize downtime, it is best practice to perform the upgrade during a period of low traffic.

  • Action: Announce a maintenance window to all stakeholders.

Section 2: Stage 1 - Upgrade to MongoDB 7.x

We will first perform a rolling upgrade to version 7.x.

Step 2.1: Prepare Helm Chart for MongoDB 7.x

  1. Download the Helm Chart:

    CODE 
    mkdir -p mongodb-v7
cd mongodb-v7
helm pull --untar oci://registry-1.docker.io/bitnamicharts/mongodb --version 15.6.9

  2. Modify values.yaml: Navigate into the chart directory (cd mongodb) and open values.yaml for editing. Update the following sections:

    • Architecture & Replica Count:

      CODE 
      ## MongoDB(®) architecture. Allowed values: standalone or replicaset
architecture: replicaset
## Number of replicas in the MongoDB(®) replica set
replicaCount: 3 # Adjust to your replica count

    • Authentication (Crucial): Use your existing root password. The replica set key should also match your existing key if one is set, or be a new secure key if this is the first time setting it.

      CODE 
      auth:
  ## @param auth.rootPassword Password for the root user
  ##
  rootPassword: "Expertflow123" # CHANGE THIS to your existing password

  ## @param auth.replicaSetKey Key for the replica set
  ##
  replicaSetKey: "YourSecureReplicaSetKey123" # CHANGE THIS to a secure key

    • TLS Configuration: Point the chart to your existing secret containing your TLS certificates.

      CODE 
      tls:
  enabled: true
  mTLS:
    enabled: true
  autoGenerated: false
  existingSecret: "mongo-mongodb-ca" # The name of your existing TLS secret
  caCert: "mongodb-ca-cert"
  caKey: "mongodb-ca-key"
  pemChainIncluded: true

    • Enable StatefulSet:

      CODE 
      ## @param useStatefulSet Use a StatefulSet instead of a Deployment
##
useStatefulSet: true

Step 2.2: Perform the Rolling Upgrade to 7.x

  1. Execute the Upgrade: This command applies the new chart values and begins the rolling upgrade.

    CODE 
    helm upgrade --install --namespace ef-external --values ./values.yaml mongo .

  2. Verification: Monitor the pod status in a separate terminal. You will see each pod terminate and restart one by one with the new version. Wait for all pods to be in the Running 1/1 state.

    CODE 
    kubectl get pods --namespace ef-external -l app.kubernetes.io/instance=mongo -w

Step 2.3: Update Feature Compatibility Version (FCV) to 7.0

  1. Launch a MongoDB 7.x Client Pod:

    CODE 
    export MONGODB_ROOT_PASSWORD=$(kubectl get secret --namespace ef-external mongo-mongodb -o jsonpath="{.data.mongodb-root-password}" | base64 -d)

kubectl apply -f - << EOF
apiVersion: v1
kind: Pod
metadata:
  name: mongo7-mongodb-client
  namespace: ef-external
spec:
  containers:
  - command: ["/bin/sh"]
    args: ["-c", "mkdir /tmp/mongodb_certs && cd /tmp/mongodb_certs && openssl req -nodes -newkey rsa:2048 -keyout client.key -out client.csr -subj '/C=SW/ST=Bern/L=Jagerweg/O=efcx/OU=EfCx/CN=mongo' && openssl x509 -req -sha256 -days 3650 -in client.csr -CA /tmp/CERTS/mongodb-ca-cert -CAkey /tmp/CERTS/mongodb-ca-key -set_serial 01 -out client.crt && cat /tmp/CERTS/mongodb-ca-cert /tmp/CERTS/mongodb-ca-key > combined.pem && cat client.crt client.key > client.pem && sleep infinity"]
    env:
    - name: MONGODB_ROOT_PASSWORD
      value: $MONGODB_ROOT_PASSWORD
    image: docker.io/bitnami/mongodb:7.0.11-debian-12-r0
    name: mongo7-mongodb-client
    volumeMounts:
    - mountPath: /tmp/CERTS
      name: mongo-certs
  volumes:
  - name: mongo-certs
    secret:
      secretName: mongo-mongodb-ca
  restartPolicy: Always
EOF

  2. Connect to the Replica Set Primary:

    CODE 
    kubectl -n ef-external exec -ti mongo7-mongodb-client -- bash

# Inside the client pod, run:
mongosh admin --host "mongo-mongodb" --authenticationDatabase admin -u root -p $MONGODB_ROOT_PASSWORD --tls --tlsAllowInvalidHostnames --tlsAllowInvalidCertificates --tlsCertificateKeyFile /tmp/mongodb_certs/client.pem --tlsCAFile /tmp/mongodb_certs/combined.pem

  3. Verify and Set FCV: Inside the mongosh shell:

    CODE 
    // Verification: Check the current FCV. It should report "6.0".
db.adminCommand( { getParameter: 1, featureCompatibilityVersion: 1 } )
// Expected output: { "featureCompatibilityVersion" : { "version" : "6.0" }, ... }

// Action: Set the FCV to "7.0".
db.adminCommand( { setFeatureCompatibilityVersion: "7.0", confirm: true } )

  4. Verification: The setFeatureCompatibilityVersion command must return an "ok": 1 status. Do not proceed if it fails.

  5. Cleanup:

    CODE 
    // Exit the mongo shell
quit()
```bash
# Exit the client pod
exit

# Delete the client pod
kubectl -n ef-external delete pod mongo7-mongodb-client
cd ..

Section 3: Stage 2 - Upgrade to MongoDB 8.x

The process is identical to Stage 1, but uses the 8.x versions.

Step 3.1: Prepare Helm Chart for MongoDB 8.x

  1. Download the Helm Chart:

    CODE 
    mkdir -p mongodb-v8
cd mongodb-v8
helm pull --untar oci://registry-1.docker.io/bitnamicharts/mongodb --version 16.4.4

  2. Modify values.yaml: Navigate into cd mongodb and edit values.yaml with the exact same architecture, replicaCount, auth, tls, and useStatefulSet values you used in Step 2.1.

Step 3.2: Perform the Rolling Upgrade to 8.x

  1. Execute the Upgrade:

    CODE 
    helm upgrade --install --namespace ef-external --values ./values.yaml mongo .

  2. Verification: Monitor the pods until all have been upgraded and are Running.

    CODE 
    kubectl get pods --namespace ef-external -l app.kubernetes.io/instance=mongo -w

Step 3.3: Update Feature Compatibility Version (FCV) to 8.0

  1. Launch a MongoDB 8.x Client Pod:

    CODE 
    export MONGODB_ROOT_PASSWORD=$(kubectl get secret --namespace ef-external mongo-mongodb -o jsonpath="{.data.mongodb-root-password}" | base64 -d)

kubectl apply -f - << EOF
apiVersion: v1
kind: Pod
metadata:
  name: mongo8-mongodb-client
  namespace: ef-external
spec:
  containers:
  - command: ["/bin/sh"]
    args: ["-c", "mkdir /tmp/mongodb_certs && cd /tmp/mongodb_certs && openssl req -nodes -newkey rsa:2048 -keyout client.key -out client.csr -subj '/C=SW/ST=Bern/L=Jagerweg/O=efcx/OU=EfCx/CN=mongo' && openssl x509 -req -sha256 -days 3650 -in client.csr -CA /tmp/CERTS/mongodb-ca-cert -CAkey /tmp/CERTS/mongodb-ca-key -set_serial 01 -out client.crt && cat /tmp/CERTS/mongodb-ca-cert /tmp/CERTS/mongodb-ca-key > combined.pem && cat client.crt client.key > client.pem && sleep infinity"]
    env:
    - name: MONGODB_ROOT_PASSWORD
      value: $MONGODB_ROOT_PASSWORD
    image: docker.io/bitnami/mongodb:8.0.4-debian-12-r3
    name: mongo8-mongodb-client
    volumeMounts:
    - mountPath: /tmp/CERTS
      name: mongo-certs
  volumes:
  - name: mongo-certs
    secret:
      secretName: mongo-mongodb-ca
  restartPolicy: Always
EOF

  2. Connect and Set FCV:

    CODE 
    kubectl -n ef-external exec -ti mongo8-mongodb-client -- bash

# Inside the client pod, run:
mongosh admin --host "mongo-mongodb" --authenticationDatabase admin -u root -p $MONGODB_ROOT_PASSWORD --tls --tlsAllowInvalidHostnames --tlsAllowInvalidCertificates --tlsCertificateKeyFile /tmp/mongodb_certs/client.pem --tlsCAFile /tmp/mongodb_certs/combined.pem

    Inside the mongosh shell:

    CODE 
    // Verification: Check the current FCV. It should report "7.0".
db.adminCommand( { getParameter: 1, featureCompatibilityVersion: 1 } )

// Action: Set the FCV to "8.0".
db.adminCommand( { setFeatureCompatibilityVersion: "8.0", confirm: true } )

  3. Verification & Cleanup: Ensure the command returns "ok": 1. Then, exit and delete the client pod.

    CODE 
    quit()
```bash
exit
kubectl -n ef-external delete pod mongo8-mongodb-client

Section 4: Post-Upgrade - Reconnecting CX Applications

Your applications need to be updated to connect to the upgraded database securely.

Step 4.1: Recreate Client TLS Secret in Application Namespace

This provides the necessary certificates for your applications to establish a trusted connection.

  1. Extract Certificates and Create Client PEM:

    CODE 
    # Create a temporary directory
mkdir -p /tmp/mongodb_certs_final

# Extract the CA certificate from the server's secret
CERTFILES=($(kubectl get secret mongo-mongodb-ca -n ef-external -o go-template='{{range $k,$v := .data}}{{$k}}{{"\n"}}{{end}}'))
for f in ${CERTFILES[*]}; do
  kubectl get secret mongo-mongodb-ca -n ef-external -o go-template='{{range $k,$v := .data}}{{ if eq $k "'$f'"}}{{$v | base64decode}}{{end}}{{end}}' > /tmp/mongodb_certs_final/${f} 2>/dev/null
done

# Navigate to the directory
cd /tmp/mongodb_certs_final

# Generate a new client key and signing request
openssl req -nodes -newkey rsa:2048 -keyout client.key -out client.csr -subj "/C=SW/ST=Bern/L=Jagerweg/O=efcx/OU=EfCx/CN=mongo"

# Sign the client certificate with the CA
openssl x509 -req -sha256 -days 3650 -in client.csr -CA mongodb-ca-cert -CAkey mongodb-ca-key -set_serial 01 -out client.crt

# Combine the client certificate and key into a single PEM file
cat client.crt client.key > client.pem

  2. Create the Secret in the expertflow Namespace:

    CODE 
    kubectl -n expertflow create secret generic mongo-mongodb-ca \
  --from-file=mongodb-ca-cert=./mongodb-ca-cert \
  --from-file=mongodb-ca-key=./mongodb-ca-key \
  --from-file=client-pem=./client.pem \
  --dry-run=client -o yaml | kubectl apply -f -

Step 4.2: Verify Application Connection String

Your applications must use a replica set connection string to leverage high availability.

  • Action: Check the configuration for your CX deployments (in ConfigMaps, Secrets, or environment variables).

  • Verification: The connection string must be in the replica set format, listing the headless service addresses of the pods.

    Example Format: mongodb://mongo-mongodb-0.mongo-mongodb-headless.ef-external.svc.cluster.local:27017,mongo-mongodb-1.mongo-mongodb-headless.ef-external.svc.cluster.local:27017,mongo-mongodb-2.mongo-mongodb-headless.ef-external.svc.cluster.local:27017/?replicaSet=rs0

    Update it if it's pointing to a single host.

Step 4.3: Restart CX Deployments

This forces the applications to pick up the new secret and connection string.

  • Action:

    CODE 
    kubectl -n expertflow rollout restart deploy

  • Final Verification: Check the logs of your application pods to confirm they connect successfully to MongoDB. Look for any TLS or connection error messages.

    CODE 
    # Get the name of one of your app pods
kubectl get pods -n expertflow

# Tail the logs of that pod
kubectl logs -n expertflow <your-app-pod-name> -f

    Perform a full functionality test of your CX solution.

Congratulations! Your MongoDB replica set has been successfully upgraded to version 8.x.

JavaScript errors detected

Please note, these errors can depend on your browser setup.

If this problem persists, please contact our support.

Contact Support Close