Upgrade guide CX4.4.10 to CX4.10.2

This upgrade guide serves for Non-Helm to Helm-based deployments

Existing CX deployment along with other groups ( for details about groups see CX Deployment Groups ) must be converted to their helm-based deployments. For this to work properly two approaches are possible

Retain External Components – Recommended
Migrate the complete solution – resulting longer downtime

This guide focuses on the Retain External Components approach

Clone the CX-4.10 revision

git clone -b CX-4.10.2 https://efcx:RecRpsuH34yqp56YRFUb@gitlab.expertflow.com/cim/cim-solution.git $HOME/CX-4.10.2

cd CX-4.10.2/kubernetes

Create a directory to hold values files for all the Helm charts

mkdir helm-values

Add helm repository

helm repo add expertflow https://expertflow.github.io/charts/

update helm repo

helm repo update expertflow

Expertflow ETL Deployment

The Expertflow ETL is deployed in a temporary namespace transflux to support MongoDB data migration prior to the CX upgrade. Once the solution is upgraded, ETL must be redeployed as part of the standard deployment process.

Expertflow ETL

For Expertflow ETL deployment, please refer to Expertflow ETL Deployment
For MongoDB data migration, refer to this guide

WARNING:

After the MongoDB data migration, please verify the associated MRD type and the Interruptible flag for MRDs (Unified Admin → Routing Engine → Channel Category Settings). Below are the default values for the three system-configured MRDs:

CHAT
1. MRD Type: CHAT
2. Interruptible: TRUE
CX VOICE
1. MRD Type: CX_VOICE
2. Interruptible: FALSE
CISCO CC
1. MRD Type: CISCO_CC
2. Interruptible: FALSE

Upgrade External Components

Change the directory to

cd CX-4.10.2/kubernetes

Redis Upgrade

Delete the Old Redis Secrets

Run the following commands to delete the old TLS certificate of Redis from both the expertflow and ef-external namespaces

kubectl delete secret -n ef-external redis-crt
kubectl delete secret -n expertflow redis-crt

Uninstall the Redis Helm Chart

helm -n ef-external delete redis

Delete the existing PVC for Redis

kubectl -n ef-external delete pvc redis-data-redis-master-0

Clone the values file to update the parameter values

helm show values expertflow/redis > helm-values/ef-redis-custom-values.yaml

Update the following values in helm-values/ef-redis-custom-values.yaml as mentioned below

Make sure to add any custom configurations, such as resource allocation, in the helm-values/ef-redis-custom-values.yaml

auth:
  password: "Expertflow123"  # Change this to match the requirements

For Worker HA deployments, add the following tolerations

tolerations:
    - key: "node.kubernetes.io/unreachable"
      operator: "Exists"
      effect: "NoExecute"
      tolerationSeconds: 60 # Evict after 60 seconds of being unreachable
    - key: "node.kubernetes.io/not-ready"
      operator: "Exists"
      effect: "NoExecute"
      tolerationSeconds: 60 # Evict after 60 seconds of being not-ready

Apply a static TLS secret for Redis

kubectl apply -f pre-deployment/static-tls/redis-tls.yaml

Run the following command to deploy Redis

helm upgrade --install=true  --namespace=ef-external --values=helm-values/ef-redis-custom-values.yaml  redis expertflow/redis

Copy TLS Cert to `Expertflow` Namespace

run the following command to copy Redis TLS Certificate to Expertflow namespace by running the following command:-

kubectl get secret redis-crt -n ef-external  -o yaml | sed 's/namespace: ef-external/namespace: expertflow/' | kubectl create -f -

PostgreSQL and Keycloak Upgrade

Ensure that Keycloak is down before proceeding with the PostgreSQL upgrade.

TLS enablement and upgrade guide

Bring down the CX solution by following these steps

Change the directory to the folder containing the 4.4.10 release

cd CX-4.4.10/cim-solution/kubernetes

Delete deployments

kubectl delete -f cim/Deployments

Delete config-maps

kubectl delete -f cim/ConfigMaps

Delete services

kubectl delete -f cim/Services

Delete ActiveMQ Statefulset

kubectl delete -f cim/StatefulSet

Delete ingress resources (for ingress-nginx )

kubectl delete  -f cim/Ingresses/nginx

OR if Traefik Ingress Controller is used, then delete ingress resources ( for Traefik )

kubectl delete  -f cim/Ingresses/traefik

Uninstall the Grafana Helm charts, as it is now part of the AgentDesk group chart in the expertflow namespace

helm -n ef-external uninstall grafana

Delete the crons

kubectl -n expertflow delete -f pre-deployment/reportingConnector/ef-reporting-connector-cron.yaml

kubectl -n expertflow delete -f pre-deployment/team-announcement/ef-team-announcement-cronjob.yaml

Delete Conversation Controller ConfigMaps

kubectl -n expertflow delete configmap ef-conversation-controller-actions-utils-cm
kubectl -n expertflow delete configmap ef-conversation-controller-actions-cm
kubectl -n expertflow delete configmap ef-conversation-controller-actions-pycache-cm

Custom Password Interpolations

Below are the interpolations when using a custom or non-default password for MongoDB, Minio, Redis, PostgreSQL and ActiveMQ

Component with a custom password	Update required in
MongoDB	Update `CX -> values.yaml -> efConnectionVars -> MONGODB_PASSWORD`
PostgreSQL	Update `keycloak -> keycloak-custom-values.yaml -> externalDatabase -> password` Update `CX -> values.yaml -> license-manager -> extraEnvVars-> DB_PASS`
minio	Update `CX -> values.yaml -> file-engin -> extraEnvVars -> ACCESSKEY,SECRETKEY`
Redis	Update `CX -> values.yaml -> efConnectionVars ->` update `ActiveMQ -> active-custom-values.yaml -> extraEnvVars -> REDIS_PASSWORD`
keycloak	N/A
activeMQ	N/A

Setup APISIX

Change to CX-4.10 release directory

cd CX-4.10.2/kubernetes

Clone the apisix values.yaml file

helm show values expertflow/apisix --version 4.10.1  > helm-values/apisix-custom-values.yaml

update the apisix-custom-values.yaml file for given parameters

global:
  ingressRouter: "devops.ef.com"   
  ingressClassName: "nginx"
  ingressTlsCertName: "ef-ingress-tls-secret"

Deploy the apisix using updated custom-values.yaml file

helm upgrade --install --namespace ef-external --values helm-values/apisix-custom-values.yaml apisix expertflow/apisix --version 4.10.1

Verify the deployment of the apisix

kubectl -n ef-external get deploy

ActiveMQ

Apply TLS secret for ActiveMQ

kubectl apply -f pre-deployment/static-tls/activemq-tls.yaml

Clone the values file to update the parameters required

helm show values expertflow/activemq > helm-values/ef-activemq-custom-values.yaml

Make sure to add any custom configurations, such as resource allocation, in the helm-values/ef-activemq-custom-values.yaml

helm upgrade --install=true  --namespace=ef-external --values=helm-values/ef-activemq-custom-values.yaml activemq expertflow/activemq --version 4.10.0

Wait for ActiveMQ to start running

kubectl wait pods activemq-0 -n ef-external --for condition=Ready --timeout=600s

CX Clamav

Clamav is an optional scanning service to scan the files before uploading to the file engine. You can enable/disable the scanning in the file engine’s environment variable IS_SCAN_ENABLED; by default, it’s enabled.

Customise the deployment by fetching the values.yaml file and edit it as per the requirements.

helm show values expertflow/clamav --version 4.9.0   > helm-values/cx-clamav-values.yaml

Edit/update the values file helm-values/cx-clamav-values.yaml with

global:
  ingressRouter: <DEFAULT-FQDN>

Deploy the Clamav helm chart by

helm upgrade --install --namespace ef-external --set global.efCxReleaseName="ef-cx"  clamav --debug --values helm-values/cx-clamav-values.yaml   helm/clamav --version 4.9.0

Setup Vault

Clone the values file to update the parameters required

helm show values expertflow/vault --version 0.28.0 > helm-values/ef-vault-custom-values.yaml

helm upgrade --install --namespace vault --create-namespace vault --debug --values helm-values/ef-vault-custom-values.yaml expertflow/vault

Use the following vault configuration guide

Deploy Expertflow CX

Custom Configuration

For detailed guidelines on applying environment-specific configurations using custom values.yaml layering, refer to the CX Helm Chart Custom Configuration Strategy guide

CX Core

Transfer the Mongo Certificates from the ef-external namespace

kubectl get secret mongo-mongodb-ca -n ef-external  -o yaml | sed 's/namespace: ef-external/namespace: expertflow/' | kubectl create -f -

Apply the tls certs for File-Engine and Unified-Admin

kubectl apply -f pre-deployment/static-tls/ef-file-engine-https-certs.yaml 
kubectl apply -f pre-deployment/static-tls/ef-unified-admin-https-certs.yaml

Set up the default translation file for the customer widget

kubectl -n expertflow  create configmap ef-widget-translations-cm --from-file=pre-deployment/app-translations/customer-widget/i18n/

Apply ConfigMap to enable log masking for all components in the expertflow namespace

kubectl apply -f pre-deployment/logback/
kubectl -n expertflow create configmap ef-logback-cm --from-file=pre-deployment/logback/logback-spring.xml

Set up GraphQL schemas and MongoDB rules configmaps

 kubectl create configmap -n expertflow conversation-manager-graphql-schemas --from-file=pre-deployment/conversation-manager/graphql/schemas

kubectl create configmap -n expertflow conversation-manager-graphql-mongodb-rules --from-file=pre-deployment/conversation-manager/graphql/graphql-mongodb-rules.json

Create and Customise `ef-cx-custom-values.yaml`

Customise ef-cx-custom-values.yaml

You must first create custom values.yaml file to define your minimum required configurations.

Step 1: Create the Values File

Run the following command to create a new file:

vi helm-values/ef-cx-custom-values.yaml

Step 2: Add Required Minimum Configuration

In the opened file, add the following section to define the FQDN (Fully Qualified Domain Name) for ingress routing:

global:   
  ingressRouter: <CUSTOM-FQDN>

🔁 Replace <CUSTOM-FQDN> with your actual domain, e.g., devops.example.com.

This is the minimum required customisation for the CX Helm chart to work.

Optional: Add Further Customisations

You can extend the same ef-cx-custom-values.yaml file with additional configurations as needed, including environment variables, replica counts, etc.

To view all available default configurations and decide what you want to override:

helm show values expertflow/cx --version 4.10.2

This command prints the full default values.yaml file used by the CX chart, which serves as a reference for all configurable parameters.

We recommend only overriding the values you need in your custom file to keep the configuration lean and maintainable as mentioned in the custom configuration guide

In real-time reports, change the following extraEnvVars as per your reporting DB

DATASOURCE_URL
DATASOURCE_USERNAME
DATASOURCE_PASSWORD

Deploy the CX Core using default values.

helm upgrade --install --namespace expertflow --create-namespace   ef-cx  --debug --values helm-values/ef-cx-custom-values.yaml expertflow/cx --version 4.10.2

“ef-cx” in the above command is the release name which will be referenced in all subsequent functional groups' deployments.

Check the status of CX components

kubectl -n expertflow get pods

CX AgentDesk

Delete the existing translations config-map

kubectl delete cm ef-app-translations-cm -n expertflow

Set up the default translation file for Agent Desk

kubectl -n expertflow  create configmap ef-app-translations-cm --from-file=pre-deployment/app-translations/unified-agent/i18n

Set up the default canned messages translations file for Agent Desk

kubectl -n expertflow  create configmap ef-canned-messages-cm --from-file=pre-deployment/app-translations/unified-agent/canned-messages

Apply CRM ConfigMap for Agent Desk

kubectl -n expertflow create configmap ef-crm-service-cm --from-file=pre-deployment/crm-service/

Update the FQDN of the machine against the url field and other DB configurations file post-deployment/config/grafana/supervisor-dashboards/datasource.yml

Apply the Grafana data-source manifest.

kubectl -n expertflow  create secret generic ef-grafana-datasource-secret --from-file=post-deployment/config/grafana/supervisor-dashboards/datasource.yml

Apply the Grafana provider manifest.

kubectl -n expertflow create cm ef-grafana-dashboard-provider-cm --from-file=post-deployment/config/grafana/supervisor-dashboards/dashboard.yml

For Supervisor Dashboard

Apply the config-map for the supervisor dashboard files using the steps below.

kubectl -n expertflow create configmap ef-grafana-supervisor-dashboard --from-file=post-deployment/config/grafana/supervisor-dashboards/Supervisor_Dashboard_CIM.json

For Agent Dashboard

kubectl -n expertflow create configmap ef-grafana-agent-dashboard --from-file=post-deployment/config/grafana/supervisor-dashboards/Agent_Dashboard_CIM.json

Create and Customise `cx-agent-desk-custom-values.yaml`

Customise cx-agent-desk-custom-values.yaml

You must first create custom values.yaml file to define your minimum required configurations.

Step 1: Create the Values File

Run the following command to create a new file:

vi helm-values/cx-agent-desk-custom-values.yaml

Step 2: Add Required Minimum Configuration

In the opened file, add the following section to define the FQDN (Fully Qualified Domain Name) for ingress routing:

global:   
  ingressRouter: <CUSTOM-FQDN>

🔁 Replace <CUSTOM-FQDN> with your actual domain, e.g., devops.example.com.

This is the minimum required customisation for the CX Helm chart to work.

Optional: Add Further Customisations

You can extend the same cx-agent-desk-custom-values.yaml file with additional configurations as needed, including environment variables, replica counts, etc.

To view all available default configurations and decide what you want to override:

helm show values expertflow/agent-desk --version 4.10.1

This command prints the full default values.yaml file used by the CX chart, which serves as a reference for all configurable parameters.

We recommend only overriding the values you need in your custom file to keep the configuration lean and maintainable as mentioned in the custom configuration guide

Install the AgentDesk using the Helm chart

helm upgrade --install --namespace expertflow   --set global.efCxReleaseName="ef-cx"  cx-agent-desk  --debug --values helm-values/cx-agent-desk-custom-values.yaml expertflow/agent-desk --version 4.10.1

CX Channels (Optional)

Create and Customise `cx-channels-custom-values.yaml`

Customise cx-channels-custom-values.yaml

You must first create custom values.yaml file to define your minimum required configurations.

Step 1: Create the Values File

Run the following command to create a new file:

vi helm-values/cx-channels-custom-values.yaml

Step 2: Add Required Minimum Configuration

In the opened file, add the following section to define the FQDN (Fully Qualified Domain Name) for ingress routing:

global:   
  ingressRouter: <CUSTOM-FQDN>

🔁 Replace <CUSTOM-FQDN> with your actual domain, e.g., devops.example.com.

This is the minimum required customisation for the CX Helm chart to work.

Optional: Add Further Customisations

You can extend the same cx-channels-custom-values.yaml file with additional configurations as needed, including environment variables, replica counts, etc.

To view all available default configurations and decide what you want to override:

helm show values expertflow/channels --version 4.10.1

This command prints the full default values.yaml file used by the CX chart, which serves as a reference for all configurable parameters.

We recommend only overriding the values you need in your custom file to keep the configuration lean and maintainable.

Deploy the Channels helm chart by

helm upgrade --install --namespace expertflow  --set global.efCxReleaseName="ef-cx"   --debug   cx-channels --values  helm-values/cx-channels-custom-values.yaml  expertflow/channels --version 4.10.1

CX Reporting

Delete the existing configmap containing Reporting-Connector configurations

kubectl -n expertflow delete configmap ef-reporting-connector-conf

Reporting Connector Config-Map Setup

To upgrade the SQL DB, use the following upgrade script upgrade script 4.4.10 to 4.10.sql

Configure TLS connection for MySQL

Get the MySQL key-store (.jsk) & certificate(.cert) files from the customer. The .jsk file is required for configuration of the reporting connector, whereas the .cert file is required for Apache Superset SSL configuration. Skeleton Project (cim-solution) already contains the default .jks files in the keystore directory. Replace the mykeystore.jks file acquired from the customer in cim-solution/kubernetes/pre-deployment/reportingConnector/keystore/ directory.

Create keystore.jks used for MySQL TLS

kubectl create configmap -n expertflow ef-reporting-connector-keystore-cm --from-file=pre-deployment/reportingConnector/keystore/mykeystore.jks

Open the pre-deployment/reportingConnector/reporting-connector.conf and set the mysql_dbms_additional_params value as shown below.

mysql_dbms_additional_params=noDatetimeStringSync=true&useSSL=true&requireSSL=true&trustServerCertificate=true&clientCertificateKeyStoreUrl=file:///root/config/certs/mykeystore.jks&clientCertificateKeyStorePassword={KEYSTORE_PASSWORD}
 
# Replace the {KEYSTORE_PASSWORD} with your original keystore password. Use "changeit" in case of default password.

Reporting Connector Config-Map Setup

Update parameters below in the file pre-deployment/reportingConnector/reporting-connector.conf

Parameter	Requirement
fqdn	FQDN of the CX Solution
svc_name	k get svc -n expertflow \| grep historical http://ef-cx-historical-reports-svc.expertflow.svc:8081
browser_language	en or ar
connection_type	mysql or mssql
sql_dbms_server_ip	<IP>
sql_dbms_port	for MySQL 3306 / for MSSQL 1433
sql_dbms_username	<username>
sql_dbms_password	<password>
sql_database_name	<database name>

Apply configuration for Reporting-Connector

kubectl -n expertflow create configmap ef-reporting-connector-conf-cm --from-file=pre-deployment/reportingConnector/reporting-connector.conf

Create and Customise `cx-reporting-scheduler-custom-values.yaml`

Customise cx-reporting-scheduler-custom-values.yaml

You must first create custom values.yaml file to define your minimum required configurations.

Step 1: Create the Values File

Run the following command to create a new file:

vi helm-values/cx-reporting-scheduler-custom-values.yaml

Step 2: Add Required Minimum Configuration

In the opened file, add the following section to define the FQDN (Fully Qualified Domain Name) for ingress routing:

global:   
  ingressRouter: <CUSTOM-FQDN>

🔁 Replace <CUSTOM-FQDN> with your actual domain, e.g., devops.example.com.

This is the minimum required customisation for the CX Helm chart to work.

Optional: Add Further Customisations

You can extend the same cx-reporting-scheduler-custom-values.yaml file with additional configurations as needed, including environment variables, replica counts, etc.

To view all available default configurations and decide what you want to override:

helm show values expertflow/reporting --version 4.10.0

This command prints the full default values.yaml file used by the CX chart, which serves as a reference for all configurable parameters.

We recommend only overriding the values you need in your custom file to keep the configuration lean and maintainable.

Deploy the Reporting Scheduler

helm upgrade --install --namespace expertflow --set global.efCxReleaseName="ef-cx"   cx-reporting --debug --values helm-values/cx-reporting-scheduler-custom-values.yaml  expertflow/reporting  --version 4.10.0

To update the superset reports,

For Standard Reports:

Download the reports folder from kubernetes/post-deployment/supersetReporting/MySQL/all/dashboard_export_apache_superset_2.0_mysql reports

Compress to a Zip file, then update the file on EFBI (superset).

The guide to import the Reports Package into EFBI is here.

Upgrade permissions in Keycloak

Please create the following realm roles(if they do not exist) before importing the files mentioned below:
- conversation-studio-admin
- evaluator
- quality-manager
- customer
To upgrade Keycloak permissions, follow this guide
To migrate Keycloak Groups/Teams to CX Teams, follow this guide

Digital Channel Icons Bootstrapping

Proceed with icons bootstrapping.

kubectl apply -f scripts/minio-helper.yaml

kubectl -n ef-external --timeout=90s wait --for=condition=ready pod minio-helper

kubectl -n ef-external cp post-deployment/data/minio/bucket/default minio-helper:/tmp/

kubectl -n ef-external cp scripts/icon-helper.sh minio-helper:/tmp/

kubectl -n ef-external exec -it minio-helper -- /bin/sh /tmp/icon-helper.sh

kubectl delete -f scripts/minio-helper.yaml

Update Provider Webhooks

Update the channel Provider on the unified admin

Go to the channel provider menu and update all Provider Webhooks with the updated service name

for web channel, replace ef with ef-cx in the service name e.g.
from http://ef-web-channel-manager-svc:7000
To http://ef-cx-web-channel-manager-svc:7000
for any other channel deployed via cx-channels, replace ef with cx-channels in the service name e.g.
From http://ef-twilio-connector-svc:8085
To http://cx-channels-twilio-connector-svc:8085

Ingress Migration

Follow this guide to migrate Ingress from Traefik to NGINX.

API Authorisation (Optional)
By default, API Authentication and Authorisation are disabled; you can enable them by following this Guide

Expertflow ETL Deployment

Expertflow ETL

Upgrade External Components

Redis Upgrade

Copy TLS Cert to Expertflow Namespace

PostgreSQL and Keycloak Upgrade

Bring down the CX solution by following these steps

Custom Password Interpolations

Setup APISIX

ActiveMQ

CX Clamav

Setup Vault

Deploy Expertflow CX

Custom Configuration

CX Core

Create and Customise ef-cx-custom-values.yaml

Step 1: Create the Values File

Step 2: Add Required Minimum Configuration

Optional: Add Further Customisations

CX AgentDesk

For Supervisor Dashboard

For Agent Dashboard

Create and Customise cx-agent-desk-custom-values.yaml

Step 1: Create the Values File

Step 2: Add Required Minimum Configuration

Optional: Add Further Customisations

CX Channels (Optional)

Create and Customise cx-channels-custom-values.yaml

Step 1: Create the Values File

Step 2: Add Required Minimum Configuration

Optional: Add Further Customisations

CX Reporting

Configure TLS connection for MySQL

Reporting Connector Config-Map Setup

Create and Customise cx-reporting-scheduler-custom-values.yaml

Step 1: Create the Values File

Step 2: Add Required Minimum Configuration

Optional: Add Further Customisations

For Standard Reports:

Upgrade permissions in Keycloak

Digital Channel Icons Bootstrapping

Update Provider Webhooks

Ingress Migration

Copy TLS Cert to `Expertflow` Namespace

Create and Customise `ef-cx-custom-values.yaml`

Create and Customise `cx-agent-desk-custom-values.yaml`

Create and Customise `cx-channels-custom-values.yaml`

Create and Customise `cx-reporting-scheduler-custom-values.yaml`