TLS enablement for Redis, MongoDB , PostgreSQL and ActiveMQ with Static Certificates

This document describes the procedure to enable TLS encryption for Expertflow CX stateful components mainly Redis, MongoDB, PostgreSQL and ActiveMQ.

WARNINIG

This procedures requires redeployment of existing solution to enable SSL/TLS support in mongodb, redis, postgresql and activeMQ. Please take backup before proceeding with the procedure below and then restore when completed. For further details please consult Mongo, PostgreSQL Backup/Restore Procedure for EF-CX on Kubernetes ( manual procedure )

Deploy Static Certificates:-

Change the directory:-

CODE

cd cim-solution/kubernetes

apply TLS secrets for external components:-

CODE

kubectl apply -f external-tls-secrets/

Redis

Redis should be deployed with these values in the values.yaml file

CODE

tls:
  ## @param tls.enabled Enable TLS traffic
  ##
  enabled: true
  ## @param tls.authClients Require clients to authenticate
  ##
  authClients: true
  ## @param tls.autoGenerated Enable autogenerated certificates
  ##
  autoGenerated: false
  ## @param tls.existingSecret The name of the existing secret that contains the TLS certificates
  ##
  existingSecret: "redis-crt"
  ## @param tls.certificatesSecret DEPRECATED. Use existingSecret instead.
  ##
  certificatesSecret: ""
  ## @param tls.certFilename Certificate filename
  ##
  certFilename: "tls.crt"
  ## @param tls.certKeyFilename Certificate Key filename
  ##
  certKeyFilename: "tls.key"
  ## @param tls.certCAFilename CA Certificate filename
  ##
  certCAFilename: "ca.crt"

Deploy the Redis using the helm command mentioned in the deployment guide.

Manual Verification:-

Export all cert files using the following commands:-

CODE

mkdir /tmp/redis_certs/
CERTFILES=($(kubectl get secret redis-crt -n ef-external -o go-template='{{range $k,$v := .data}}{{$k}}{{"\n"}}{{end}}'))
for f in ${CERTFILES[*]}; do   kubectl get secret redis-crt  -n ef-external -o go-template='{{range $k,$v := .data}}{{ if eq $k "'$f'"}}{{$v  | base64decode}}{{end}}{{end}}' > /tmp/redis_certs/${f} 2>/dev/null; done

Export Redis Password:-

CODE

export REDIS_PASSWORD=$(kubectl get secret --namespace ef-external redis -o jsonpath="{.data.redis-password}" | base64 -d)

Start a Redis client pod:-

CODE

kubectl run --namespace ef-external redis-client   --env REDIS_PASSWORD=$REDIS_PASSWORD  --image gitimages.expertflow.com/general/redis:CIM-4292-6.2-debian-10-k8s --command -- sleep infinity

Now you can mount the secret redis-crt inside the client pods and use TLS certificates.

CODE

kubectl cp --namespace ef-external /tmp/redis_certs/tls.crt redis-client:/tmp/tls.crt
kubectl cp --namespace ef-external /tmp/redis_certs/tls.key redis-client:/tmp/tls.key
kubectl cp --namespace ef-external /tmp/redis_certs/ca.crt redis-client:/tmp/ca.crt

Exec into client pod:-

CODE

kubectl exec --tty -i redis-client \
   --namespace ef-external -- bash

verify the connection using the following command in the client pod:-

CODE

I have no name!@redis-client:/$ REDISCLI_AUTH="$REDIS_PASSWORD" redis-cli -h redis-master --tls --cert /tmp/tls.crt --key /tmp/tls.key --cacert /tmp/ca.crt              redis-master:6379>
redis-master:6379>
redis-master:6379> CONFIG GET databases
1) "databases"
2) "16"

MongoDB

Deploy the mongoDB helm chart with these values changed

CODE

tls:
  ## @param tls.enabled Enable MongoDB(&reg;) TLS support between nodes in the cluster as well as between mongo clients and nodes
  ##
  enabled: true
  ## @param tls.autoGenerated Generate a custom CA and self-signed certificates
  ##
  autoGenerated: false
  ## @param tls.existingSecret Existing secret with TLS certificates (keys: `mongodb-ca-cert`, `mongodb-ca-key`, `client-pem`)
  ## NOTE: When it's set it will disable certificate creation
  ##
  existingSecret: "mongo-mongodb-ca"
  ## Add Custom CA certificate
  ## @param tls.caCert Custom CA certificated (base64 encoded)
  ## @param tls.caKey CA certificate private key (base64 encoded)
  ##
  caCert: "mongodb-ca-cert"
  caKey: "mongodb-ca-key"

Then deploy the solution using standard helm command as mentioned in the CX deployment guide.

Manual Verification

export all the cert files in ef-external namespace using

CODE

mkdir /tmp/mongodb_certs
CERTFILES=($(kubectl get secret mongo-mongodb-ca -n ef-external -o go-template='{{range $k,$v := .data}}{{$k}}{{"\n"}}{{end}}'))
for f in ${CERTFILES[*]}; do   kubectl get secret mongo-mongodb-ca  -n ef-external -o go-template='{{range $k,$v := .data}}{{ if eq $k "'$f'"}}{{$v  | base64decode}}{{end}}{{end}}' > /tmp/mongodb_certs/${f} 2>/dev/null; done

The above script will export all the certs to local directory /tmp/mongodb_certs.

Run the following command to export MongoDB Password:-

CODE

kubectl get secret --namespace ef-external mongo-mongodb -o jsonpath="{.data.mongodb-root-password}" | base64 -d

Run the mongoDB client pod

CODE

kubectl run --namespace ef-external mongo-mongodb-client --env="MONGODB_ROOT_PASSWORD=$MONGODB_ROOT_PASSWORD" --image docker.io/bitnami/mongodb:6.0.2-debian-11-r1

copy the certificate files inside the client pod

CODE

kubectl -n ef-external cp /tmp/mongodb_certs mongo-mongodb-client:/tmp/

Connect to the mongoDB pod using SSL/TLS certs

CODE

kubectl -n ef-external exec -it mongo-mongodb-client  -- bash

once inside the mongodb-client pod, combine both cert and key file using

CODE

cat /tmp/mongodb_certs/mongodb-ca-cert /tmp/mongodb_certs/mongodb-ca-key > /tmp/mongodb_certs/combined.pem

verify the connection using tls

CODE

mongosh admin --host "mongo-mongodb" \
--authenticationDatabase admin \
-u root \
-p $MONGODB_ROOT_PASSWORD \
--tls  \
--tlsAllowInvalidHostnames  \
--tlsAllowInvalidCertificates \
--tlsCertificateKeyFile /tmp/mongodb_certs/client-pem  \
--tlsCAFile /tmp/mongodb_certs/client-pem

Sample Run

CODE

I have no name!@mongo-mongodb-client:/$ mongosh admin --host "mongo-mongodb" --authenticationDatabase admin -u root -p $MONGODB_ROOT_PASSWORD --tls  --tlsAllowInvalidHostnames  --tlsAllowInvalidCertificates --tlsCertificateKeyFile /tmp/client-pem  --tlsCAFile /tmp/mongo/client-pem
Current Mongosh Log ID: 663b303a12c4a32b93ff8546
Connecting to:          mongodb://<credentials>@mongo-mongodb:27017/admin?directConnection=true&authSource=admin&tls=true&tlsAllowInvalidHostnames=true&tlsAllowInvalidCertificates=true&tlsCertificateKeyFile=%2Ftmp%2Fmongo%2Fcombined.pem&tlsCAFile=%2Ftmp%2Fmongo%2Fclient-pem&appName=mongosh+1.6.0
Using MongoDB:          6.0.2
Using Mongosh:          1.6.0
For mongosh info see: https://docs.mongodb.com/mongodb-shell/
------
   The server generated these startup warnings when booting
   2024-05-08T07:42:12.444+00:00: /sys/kernel/mm/transparent_hugepage/enabled is 'always'. We suggest setting it to 'never'
   2024-05-08T07:42:12.445+00:00: /sys/kernel/mm/transparent_hugepage/defrag is 'always'. We suggest setting it to 'never'
   2024-05-08T07:42:12.445+00:00: vm.max_map_count is too low
------
admin> show dbs;
admin   100.00 KiB
config   12.00 KiB
local    72.00 KiB
admin>

Sometimes, the mongodb client pod doesn’t inherit the MONGODB_ROOT_PASSWORD environment variable, and user will have to enter the password manually.

PostgreSQL

Postgresql should be deployed with these values in the values.yaml file

CODE

tls:
  ## @param tls.enabled Enable TLS traffic support
  ##
  enabled: true
  ## @param tls.autoGenerated Generate automatically self-signed TLS certificates
  ##
  autoGenerated: false
  ## @param tls.preferServerCiphers Whether to use the server's TLS cipher preferences rather than the client's
  ##
  preferServerCiphers: true
  ## @param tls.certificatesSecret Name of an existing secret that contains the certificates
  ##
  certificatesSecret: "ef-postgresql-crt"
  ## @param tls.certFilename Certificate filename
  ##
  certFilename: "tls.crt"
  ## @param tls.certKeyFilename Certificate key filename
  ##
  certKeyFilename: "tls.key"
  ## @param tls.certCAFilename CA Certificate filename
  ## If provided, PostgreSQL will authenticate TLS/SSL clients by requesting them a certificate
  ## ref: https://www.postgresql.org/docs/9.6/auth-methods.html
  ##
  certCAFilename: ""

Deploy the Postgresql using the helm command mentioned in the deployment guide.

Manual Verification:-

export all cert files using the following commands:-

CODE

mkdir /tmp/postgresql_certs/
CERTFILES=($(kubectl get secret ef-postgresql-crt -n ef-external -o go-template='{{range $k,$v := .data}}{{$k}}{{"\n"}}{{end}}'))
for f in ${CERTFILES[*]}; do   kubectl get secret ef-postgresql-crt  -n ef-external -o go-template='{{range $k,$v := .data}}{{ if eq $k "'$f'"}}{{$v  | base64decode}}{{end}}' > /tmp/postgresql_certs/${f} 2>/dev/null; done

Export Postgres Password:-

CODE

export POSTGRES_PASSWORD=$(kubectl get secret --namespace ef-external ef-postgresql -o jsonpath="{.data.password}" | base64 -d)

Start a Postgresql client pod by running this command:-

CODE

kubectl run ef-postgresql-client --rm --tty -i --restart='Never' --namespace ef-external --image docker.io/bitnami/postgresql:14.5.0-debian-11-r21 --env="PGPASSWORD=$POSTGRES_PASSWORD" \
      --command -- psql --host ef-postgresql -U sa -d licenseManager -p 5432