TLS Enablement for Stateful Components

This procedures requires redeployment of existing solution to enable SSL/TLS support in mongodb, redis, postgresql and activeMQ. Please take backup before proceeding with the procedure below and then restore when completed. For further details please consult Mongo, PostgreSQL Backup/Restore Procedure for EF-CX on Kubernetes ( manual procedure )

Deploy Static Certificates:-

Copy the CX-4.5.2/cim-solution/kubernetes/external-tls-secrets and paste in current release cim-solution/kubernetes

CD into the following directory of the current release:-

CODE

cd cim-solution/kubernetes

apply TLS secrets for external components:-

CODE

kubectl apply -f external-tls-secrets/

Redis

Update the following values in the values.yaml file under cim-solution/kubernetes/external/bitnami/redis

CODE

tls:
  ## @param tls.enabled Enable TLS traffic
  ##
  enabled: true
  ## @param tls.authClients Require clients to authenticate
  ##
  authClients: true
  ## @param tls.autoGenerated Enable autogenerated certificates
  ##
  autoGenerated: false
  ## @param tls.existingSecret The name of the existing secret that contains the TLS certificates
  ##
  existingSecret: "redis-crt"
  ## @param tls.certificatesSecret DEPRECATED. Use existingSecret instead.
  ##
  certificatesSecret: ""
  ## @param tls.certFilename Certificate filename
  ##
  certFilename: "tls.crt"
  ## @param tls.certKeyFilename Certificate Key filename
  ##
  certKeyFilename: "tls.key"
  ## @param tls.certCAFilename CA Certificate filename
  ##
  certCAFilename: "ca.crt"

Uninstall the Redis Helm Chart

CODE

helm -n ef-external delete redis

Delete the existing PVC for Redis

CODE

kubectl -n ef-external delete pvc redis-data-redis-master-0

Deploy the Redis using the helm command

CODE

helm upgrade --install=true --wait=true --timeout=10m0s --debug --namespace=ef-external --values=external/bitnami/redis/values.yaml redis external/bitnami/redis/

Once the Redis is deployed successfully, migrate the TLS certificate to Expertflow namespace by executing

CODE

kubectl get secret redis-crt -n ef-external  -o yaml | sed 's/namespace: ef-external/namespace: expertflow/' | kubectl create -f -

Manual Verification ( Optional)

Export all cert files using the following commands:-

CODE

mkdir /tmp/redis_certs/
CERTFILES=($(kubectl get secret redis-crt -n ef-external -o go-template='{{range $k,$v := .data}}{{$k}}{{"\n"}}{{end}}'))
for f in ${CERTFILES[*]}; do   kubectl get secret redis-crt  -n ef-external -o go-template='{{range $k,$v := .data}}{{ if eq $k "'$f'"}}{{$v  | base64decode}}{{end}}{{end}}' > /tmp/redis_certs/${f} 2>/dev/null; done

Export Redis Password:-

CODE

export REDIS_PASSWORD=$(kubectl get secret --namespace ef-external redis -o jsonpath="{.data.redis-password}" | base64 -d)

Start a Redis client pod:-

CODE

kubectl run --namespace ef-external redis-client   --env REDIS_PASSWORD=$REDIS_PASSWORD  --image gitimages.expertflow.com/general/redis:CIM-4292-6.2-debian-10-k8s --command -- sleep infinity

Now you can mount the secret redis-crt inside the client pods and use TLS certificates.

CODE

kubectl cp --namespace ef-external /tmp/redis_certs/tls.crt redis-client:/tmp/tls.crt
kubectl cp --namespace ef-external /tmp/redis_certs/tls.key redis-client:/tmp/tls.key
kubectl cp --namespace ef-external /tmp/redis_certs/ca.crt redis-client:/tmp/ca.crt

Exec into client pod:-

CODE

kubectl exec --tty -i redis-client \
   --namespace ef-external -- bash

verify the connection using the following command in the client pod:-

CODE

I have no name!@redis-client:/$ REDISCLI_AUTH="$REDIS_PASSWORD" redis-cli -h redis-master --tls --cert /tmp/tls.crt --key /tmp/tls.key --cacert /tmp/ca.crt              redis-master:6379>
redis-master:6379>
redis-master:6379> CONFIG GET databases
1) "databases"
2) "16"

PostgreSQL

Update the following values in the values.yaml file under cim-solution/kubernetes/external/bitnami/postgresql

CODE

tls:
  ## @param tls.enabled Enable TLS traffic support
  ##
  enabled: true
  ## @param tls.autoGenerated Generate automatically self-signed TLS certificates
  ##
  autoGenerated: false
  ## @param tls.preferServerCiphers Whether to use the server's TLS cipher preferences rather than the client's
  ##
  preferServerCiphers: true
  ## @param tls.certificatesSecret Name of an existing secret that contains the certificates
  ##
  certificatesSecret: "ef-postgresql-crt"
  ## @param tls.certFilename Certificate filename
  ##
  certFilename: "tls.crt"
  ## @param tls.certKeyFilename Certificate key filename
  ##
  certKeyFilename: "tls.key"
  ## @param tls.certCAFilename CA Certificate filename
  ## If provided, PostgreSQL will authenticate TLS/SSL clients by requesting them a certificate
  ## ref: https://www.postgresql.org/docs/9.6/auth-methods.html
  ##
  certCAFilename: ""

Uninstall the PostgreSQL helm chart

CODE

helm -n ef-external delete ef-postgresql

Delete the PVC for postgresql

CODE

kubectl -n ef-external delete pvc data-ef-postgresql-0

Deploy the Postgresql using the helm command

CODE

helm upgrade --install=true --wait=true --timeout=10m0s --debug --namespace=ef-external --values=external/bitnami/postgresql/values.yaml ef-postgresql external/bitnami/postgresql

Once the Postgresql is deployed successfully, migrate the TLS certificate to Expertflow namespace by executing

CODE

kubectl get secret ef-postgresql-crt -n ef-external  -o yaml | sed 's/namespace: ef-external/namespace: expertflow/' | kubectl create -f -

Manual Verification ( optional )

export all cert files using the following commands:-

CODE

mkdir /tmp/postgresql_certs/
CERTFILES=($(kubectl get secret ef-postgresql-crt -n ef-external -o go-template='{{range $k,$v := .data}}{{$k}}{{"\n"}}{{end}}'))
for f in ${CERTFILES[*]}; do   kubectl get secret ef-postgresql-crt  -n ef-external -o go-template='{{range $k,$v := .data}}{{ if eq $k "'$f'"}}{{$v  | base64decode}}{{end}}' > /tmp/postgresql_certs/${f} 2>/dev/null; done

Export Postgres Password:-

CODE

export POSTGRES_PASSWORD=$(kubectl get secret --namespace ef-external ef-postgresql -o jsonpath="{.data.password}" | base64 -d)

Start a Postgresql client pod by running this command:-

CODE

kubectl run ef-postgresql-client --rm --tty -i --restart='Never' --namespace ef-external --image docker.io/bitnami/postgresql:14.5.0-debian-11-r21 --env="PGPASSWORD=$POSTGRES_PASSWORD" \
      --command -- psql --host ef-postgresql -U sa -d licenseManager -p 5432

Now restore the backup for postgresql using Mongo, PostgreSQL Backup/Restore Procedure for EF-CX on Kubernetes ( manual procedure )

Keycloak

1. Update Keycloak Helm Chart Values

To update the Helm chart values of your Bitnami Keycloak, you need to edit the values.yaml file. You can locate this file in the “cim-solution/kubernetes/external/bitnami/keycloak” directory. You need to modify the database connection string to include SSL parameters and mount the secret containing the certificates. Find the 'extraEnv' , 'extraVolumes' and 'extraVolumeMounts' sections in values.yaml. Add the following properties to specified sections:

YAML

## Add TLS parameters
extraEnvVars:
  - name: KEYCLOAK_JDBC_PARAMS
    value: "sslmode=verify-ca&sslrootcert=/opt/keycloak/certs/ca.crt"

## Mount the secret containing the TLS certificates
extraVolumes:
  - name: ef-postgresql-crt
    secret:
      secretName: ef-postgresql-crt

extraVolumeMounts:
  - name: ef-postgresql-crt
    mountPath: /opt/keycloak/certs/
    readOnly: true

Uninstall the Keycloak Helm Chart

CODE

helm -n ef-external delete keycloak

Deploy or Upgrade the Keycloak Release with Helm

Deploy or upgrade your Keycloak release using the values.yaml file by running the following command:

CODE

helm upgrade --install=true --wait=true --timeout=10m0s --debug --namespace=ef-external --values=external/bitnami/keycloak/values.yaml keycloak external/bitnami/keycloak/

The following components use Auto Generated certs

MongoDB

Auth Enablement

open values.yaml file in MongoDB helm chart.

CODE

vi external/bitnami/mongodb/values.yaml

Update the following values as mentioned below:-

CODE

auth:
  ## @param auth.enabled Enable authentication
  ## ref: https://docs.mongodb.com/manual/tutorial/enable-authentication/
  ## EXPERTFLOW
  enabled: true
  ## @param auth.rootUser MongoDB(&reg;) root user
  ##
  rootUser: root
  ## @param auth.rootPassword MongoDB(&reg;) root password
  ## ref: https://github.com/bitnami/containers/tree/main/bitnami/mongodb#setting-the-root-user-and-password-on-first-run
  ##
  rootPassword: "Expertflow123"  # Change this to match the requirements

TLS Enablement

open values.yaml file in MongoDB helm chart.

CODE

vi external/bitnami/mongodb/values.yaml

Deploy the mongoDB helm chart with these values changed

CODE

tls:
  ## @param tls.enabled Enable MongoDB(&reg;) TLS support between nodes in the cluster as well as between mongo clients and nodes
  ##
  enabled: true
  ## @param tls.autoGenerated Generate a custom CA and self-signed certificates
  ##
  autoGenerated: true
  extraDnsNames:
       - "mongo-mongodb.ef-external.svc"
       - "mongo-mongodb.ef-external.svc.cluster.local"

Uninstall the MongoDB helm chart

CODE

helm -n ef-external delete mongo

Remove the PVC/PV

Make sure the backup is secured before proceeding with this step, as the PVC/PV will be recreated

CODE

kubectl -n ef-external delete pvc datadir-mongo-mongodb-0

Then deploy the solution using the standard helm command

CODE

helm upgrade --install=true --wait=true --timeout=10m0s --debug --namespace=ef-external --values=external/bitnami/mongodb/values.yaml mongo external/bitnami/mongodb/

Once the mongoDB chart is deployed successfully, copy the secret containing TLS certificates using

CODE

kubectl get secret mongo-mongodb-ca -n ef-external  -o yaml | sed 's/namespace: ef-external/namespace: expertflow/' | kubectl create -f -

Restore Backup

Once the mongodb statefulset is deployed successfully and is in ready state, please proceed with the restore procedure of mongo backup.

when restoring mongo from backup, please do not restore the admin db.

Manual Verification ( optional )

export all the cert files in ef-external namespace using

CODE

mkdir /tmp/mongodb_certs
CERTFILES=($(kubectl get secret mongo-mongodb-ca -n ef-external -o go-template='{{range $k,$v := .data}}{{$k}}{{"\n"}}{{end}}'))
for f in ${CERTFILES[*]}; do   kubectl get secret mongo-mongodb-ca  -n ef-external -o go-template='{{range $k,$v := .data}}{{ if eq $k "'$f'"}}{{$v  | base64decode}}{{end}}{{end}}' > /tmp/mongodb_certs/${f} 2>/dev/null; done

The above script will export all the certs to local directory /tmp/mongodb_certs.

Run the following command to export MongoDB Password:-

CODE

kubectl get secret --namespace ef-external mongo-mongodb -o jsonpath="{.data.mongodb-root-password}" | base64 -d

Run the mongoDB client pod

CODE

kubectl run --namespace ef-external mongo-mongodb-client --env="MONGODB_ROOT_PASSWORD=$MONGODB_ROOT_PASSWORD" --image docker.io/bitnami/mongodb:6.0.2-debian-11-r1 --command -- sleep infinity

copy the certificate files inside the client pod

CODE

kubectl -n ef-external cp /tmp/mongodb_certs mongo-mongodb-client:/tmp/

Connect to the mongoDB pod using SSL/TLS certs

CODE

kubectl -n ef-external exec -it mongo-mongodb-client  -- bash

once inside the mongodb-client pod, combine both cert and key file using

CODE

cat /tmp/mongodb_certs/mongodb-ca-cert /tmp/mongodb_certs/mongodb-ca-key > /tmp/mongodb_certs/combined.pem

verify the connection using tls

CODE

mongosh admin --host "mongo-mongodb" \
--authenticationDatabase admin \
-u root \
-p $MONGODB_ROOT_PASSWORD \
--tls  \
--tlsAllowInvalidHostnames  \
--tlsAllowInvalidCertificates \
--tlsCertificateKeyFile /tmp/mongodb_certs/client-pem  \
--tlsCAFile /tmp/mongodb_certs/client-pem

Sample Run

CODE

$body

Sometimes, the mongodb client pod doesn’t inherit the MONGODB_ROOT_PASSWORD environment variable, and user will have to enter the password manually.