Mongo, PostgreSQL Backup/Restore Procedure for EF-CX on Kubernetes ( manual procedure )

In this guide, we will evaluate procedures to backup from Mongo and PostgreSQL databases. This procedure requires the end-user to be comfortable with the common interfacing with the Kubernetes and understand the basics of the Kubernetes operations.

there is an actively developed guide for backups using Velero for EF-CX solution on Kubernetes. This procedure is only valid for manual backup and does not ascertain the continued approach for backup. Velero-based backups are considered more appropriate for continuous and effective back solutions. Please refer to Kubernetes Backup/Restore using Velero

Backups

To save backups on your local system, create a folder under $HOME/backups

CODE

mkdir -p  $HOME/backups/{mongo,postgresql}

Mongo Backup

To take backup from the current mongo databases, follow these steps

step-1 Run the MongoDB client POD

CODE

kubectl run --namespace ef-external  --tty -i   mongo-mongodb-client  --env="MONGODB_ROOT_PASSWORD=$MONGODB_ROOT_PASSWORD" --image docker.io/bitnami/mongodb:6.0.2-debian-11-r1 --command -- sleep infinity

wait for the mongo-client pod to run. Sample run given below

CODE

#kubectl get pods -n ef-external mongo-mongodb-client
NAME                   READY   STATUS    RESTARTS   AGE
mongo-mongodb-client   1/1     Running   0          16s

step-2 Exec into the mongo client pod

CODE

kubectl -n ef-external exec -i -t mongo-mongodb-client -- bash

step-3 take the backup of all databases from the current host

CODE

mongodump --host "mongo-mongodb" --gzip --out /tmp/mongo-backup-$(date +%Y-%m-%d)

Please note that the path of the backup has to under /tmp folder. the above command will create a folder under /tmp with the date in YYYY-MM-DD appended e-g /tmp/mongo-backup-2023-09-12

exit out of the mongo-client pod

CODE

exit

step-4 retrieve the backup file from the POD to actual storage

CODE

kubectl -n ef-external cp  mongo-mongodb-client:/tmp/mongo-backup-2023-09-12  $HOME/backups/mongo/mongo-backup-2023-09-12

If the above command does not work, replace the mongo-backup-$(date +%Y-%m-%d) with the actual path e-g /tmp/mongo-backup-2023-09-12 as the time/date inside the pod might have a different folder name created

PostgreSQL Backup

For PostgreSQL backups, please follow along these steps.

step-1 export the admin username/password pair for postgresql-client pod

CODE

export POSTGRES_ADMIN_PASSWORD=$(kubectl get secret --namespace ef-external ef-postgresql -o jsonpath="{.data.postgres-password}" | base64 -d)

and for sa user ( default user for EF-CX component using PostgreSQL database )

CODE

export POSTGRES_PASSWORD=$(kubectl get secret --namespace ef-external ef-postgresql -o jsonpath="{.data.password}" | base64 -d)

step-2 launch the postgresql client

CODE

 kubectl run ef-postgresql-client  --tty -i  --namespace ef-external --image docker.io/bitnami/postgresql:14.5.0-debian-11-r21 --env="PGPASSWORD=$POSTGRES_PASSWORD"  --command -- sleep infinity

step-3 verify the pod is running

CODE

# kubectl -n ef-external get pods "ef-postgresql-client"
NAME                   READY   STATUS    RESTARTS       AGE
ef-postgresql-client   1/1     Running   15 (15d ago)   120d

step-4 exec into the postgresql-client pod

CODE

kubectl -n ef-external exec -it ef-postgresql-client -- bash

step-5 Evolve the pod Env script

execute the environment setup for postgresql-client ( Only needed when the postgresql is running in non-HA mode , like no pgpool and multiple replicas of postgresql are running )

CODE

/opt/bitnami/scripts/postgresql/entrypoint.sh /bin/bash

the above step will output something like below:

CODE

I have no name!@ef-postgresql-client:/$ /opt/bitnami/scripts/postgresql/entrypoint.sh /bin/bash
postgresql 14:55:10.29
postgresql 14:55:10.29 Welcome to the Bitnami postgresql container
postgresql 14:55:10.30 Subscribe to project updates by watching https://github.com/bitnami/containers
postgresql 14:55:10.30 Submit issues and feature requests at https://github.com/bitnami/containers/issues
postgresql 14:55:10.30

postgres@ef-postgresql-client:/$

step-6 list databases to take backups.

CODE

psql --host ef-postgresql -U sa -d licenseManager -p 5432 -q -A -t -c  "SELECT datname FROM pg_database"

identify the databases in the first column of the output in previous step and continue with their backup

step-7 Archive the backup

take backup of the keycloak_db and licenseManager Databases.

CODE

# keycloak_db database

pg_dump --host ef-postgresql -U sa  -F c -b -v -f  "/tmp/licenseManager.backup" "licenseManager"

#licenseManager database
pg_dump --host ef-postgresql -U sa  -F c -b -v -f  "/tmp/keycloak_db.backup" "keycloak_db"

exit the postgresql-client pod

CODE

exit; exit

step-8 Restore the backup onto the host

On the host System. once all the databases from the postgresql are backed up succcessfully, exit out of the postgresql client pod and recove the files onto your host system to be further saved at a secure location.

CODE

kubectl -n ef-external cp ef-postgresql-client:/tmp/keycloak_db.backup $HOME/backups/postgresql/keycloak_db.backup

kubectl -n ef-external cp ef-postgresql-client:/tmp/licenseManager.backup $HOME/backups/postgresql/licenseManager.backup

Please repeat step7 and step-8 for all required databases if other databases' backup are also required.

Restore

PostgreSQL Restore

step-1 Admin and sa Password

export the admin username/password pair for postgresql-client pod

CODE

export POSTGRES_ADMIN_PASSWORD=$(kubectl get secret --namespace ef-external ef-postgresql -o jsonpath="{.data.postgres-password}" | base64 -d)

and for sa user ( default user for EF-CX component using PostgreSQL database )

CODE

export POSTGRES_PASSWORD=$(kubectl get secret --namespace ef-external ef-postgresql -o jsonpath="{.data.password}" | base64 -d)

step-2 launch the postgresql client

CODE

 kubectl run ef-postgresql-client  --tty -i  --namespace ef-external --image docker.io/bitnami/postgresql:14.5.0-debian-11-r21 --env="PGPASSWORD=$POSTGRES_PASSWORD"  --command -- sleep infinity

step-3 verify the pod is running

CODE

# kubectl -n ef-external get pods "ef-postgresql-client"
NAME                   READY   STATUS    RESTARTS       AGE
ef-postgresql-client   1/1     Running   15 (15d ago)   120d

step-4 Copy the dump files from host

Copy all the backup files from the host. Here we are restoring only 2 backups for keycloak_db and licensemanage databases

CODE

# keycloak_db database
kubectl -n ef-external cp $HOME/backups/postgresql/keycloak_db.backup  ef-postgresql-client:/tmp/keycloak_db.backup

#licenseManager database
kubectl -n ef-external cp $HOME/backups/postgresql/licenseManager.backup  ef-postgresql-client:/tmp/licenseManager.backup

step-5exec into the postgresql-client pod

once the pod is running, exec into the postgresql-client pod

CODE

kubectl -n ef-external exec -it ef-postgresql-client -- bash

step-6 evolve the env script.

execute the environment setup for postgresql-client ( Only needed when the postgresql is running in non-HA mode , like no pgpool and multiple replicas of postgresql are running )

CODE

/opt/bitnami/scripts/postgresql/entrypoint.sh /bin/bash

the above step will output something like below:

CODE

I have no name!@ef-postgresql-client:/$ /opt/bitnami/scripts/postgresql/entrypoint.sh /bin/bash
postgresql 14:55:10.29
postgresql 14:55:10.29 Welcome to the Bitnami postgresql container
postgresql 14:55:10.30 Subscribe to project updates by watching https://github.com/bitnami/containers
postgresql 14:55:10.30 Submit issues and feature requests at https://github.com/bitnami/containers/issues
postgresql 14:55:10.30

postgres@ef-postgresql-client:/$

step-7 restore

Restore all databases one by one by running

For keyclock_db;

CODE

pg_restore --host ef-postgresql -U sa  -d "keycloak_db"  -v "/tmp/keycloak_db.backup"

For licenseManager databases, DROP the existing database

CODE

dropdb --host ef-postgresql -U sa  "licenseManager"

Create the licenseManager Databases with 'sa' user

CODE

createdb  --host ef-postgresql -U sa  "licenseManager"

verify the database is created successfully by running

CODE

psql --host ef-postgresql -U sa keycloak_db   -p 5432 -t  -c  "SELECT datname FROM pg_database"

it will display something like below:

CODE

postgres@ef-postgresql-client:/tmp$ psql --host ef-postgresql -U sa keycloak_db   -p 5432 -t  -c  "SELECT datname FROM pg_database"
 postgres
 template1
 template0
 keycloak_db
 superset

Restore the "lienceManager" Database by running

CODE

pg_restore --host ef-postgresql -U sa  -d "licenseManager"  -v "/tmp/licenseManager.backup"

Exit the postgresql-client pod.

CODE

exit

Mongo Restore

step-1 Start the client pod

Run the mongoDB client POD

CODE

kubectl run --namespace ef-external  --tty -i   mongo-mongodb-client  --env="MONGODB_ROOT_PASSWORD=$MONGODB_ROOT_PASSWORD" --image docker.io/bitnami/mongodb:6.0.2-debian-11-r1 --command -- sleep infinity

wait for the mongo-client pod to run. Sample run given below

CODE

#kubectl get pods -n ef-external mongo-mongodb-client
NAME                   READY   STATUS    RESTARTS   AGE
mongo-mongodb-client   1/1     Running   0          16s

step-2 copy the dump from host

copy the backup into the mongo-client pod

CODE

kubectl -n ef-external cp /root/backups/mongo mongo-mongodb-client:/tmp/

step-3 exec into the client pod

Exec into the mongo client pod

CODE

kubectl -n ef-external exec -i -t mongo-mongodb-client -- bash

If this is a complete restore of the MongoDB, including users and access rights, proceed to Step#4 , However if this is not the case, please remove the admin db from the archive so the new users and database rights are not over written

CODE

rm -rf /tmp/mongo/mongo-backup-<date-stamp>/admin

For the pre 4.5.1 release

step-4 restore

restore the backup of all databases from the current host

CODE

mongorestore --host "mongo-mongodb" --gzip --verbose  /tmp/mongo

step-5 Verify the restore

login into the mongodb and verify the databases

CODE

mongosh --host "mongo-mongodb"
#and run
test> show dbs

For 4.5.2 and onwards release

Enable TLS + Auth based connectivity by following steps mentioned here TLS Enablement for Stateful Components. Once these steps are complete then load the mongo dump using

CODE

mongorestore  \
--host "mongo-mongodb" \
--authenticationDatabase admin -u root -p"Expertflow123"  \
--ssl   \--tlsInsecure   \
--sslPEMKeyFile  /tmp/mongodb_certs/client-pem   \
--sslCAFile  /tmp/mongodb_certs/client-pem \
--gzip \
--verbose  /tmp/mongo/mongo-backup-2024-09-05/

step-6 exit the mongo client pod

exit out of the mongo-client pod

CODE

exit