Expertflow ETL Deployment

Requirements

A dedicated Fully Qualified Domain Name (FQDN) is required for CX Transflux.
Create the target database in the Database Management System by executing the scripts located at:
kubernetes/pre-deployment/reportingConnector/dbScripts/dbcreation/_historical_reports_db_creation_script_MySQL.sql.

Follow this guide to create the database: Guide For Setting Up or Upgrading MYSQL Database

Deployment

Clone CX-Transflux Repository

CODE

git clone https://efcx:RecRpsuH34yqp56YRFUb@gitlab.expertflow.com/cim/transflux.git -b 4.7

CODE

cd transflux

Add the Expertflow Helm charts repository.

CODE

helm repo add expertflow https://expertflow.github.io/charts

Update the charts repository

CODE

helm repo update expertflow

Create a database with name ‘airflow’ in MySQL

CODE

CREATE DATABASE airflow;

Create a folder to save the helm chart’s values

CODE

mkdir helm-values

Customize the deployment by fetching the values file and edit it as per requirements.

CODE

helm show values expertflow/transflux  > helm-values/cx-transflux-custom-values.yaml

Open the file helm-values/cx-transflux-custom-values.yaml and edit it according to the below given information which is required for the CX Transflux to work properly.

Value	Updated Value
tag	4.7_f-CIM-27922
AIRFLOW_DB_USER	Airflow DB Username in MySQL
AIRFLOW_DB_PASSWORD	Airflow DB Password in MySQL
AIRFLOW_DB_NAME	airflow
AIRFLOW__CORE__SQL_ALCHEMY_CONN	Hostname and port number must be updated
MYSQL_HOST	Upstream MySQL host IP
MYSQL_ROOT_PASSWORD	Password for MySQL user
MONGODB_PASSWORD	Update the local MongoDB password, when using non-default password

For additional parameters, refer to the extraEnvVars section in the helm-values/cx-transflux-custom-values.yaml file.

To configure CX-Transflux, update the configuration files with your specific values. Open the following files and ensure the required information is correctly set:

config/forms_data_pipeline_config.yaml
config/teams_data_pipeline_config.yaml

CODE

target:
  type: "mysql"
  db_url: "mysql+pymysql://<your-db-username>:<password>@<host>:<port>/<mysql-db-name>"
  
configdb:
  type: "mysql"
  db_url: "mysql+pymysql://<your-db-username>:<password>@<host>:<port>/<mysql-db-name>"

Create a directory for TLS certificates

CODE

mkdir -p certificates
Copy the mogoDB certs in certificates/

place the MySQL certs in the following directory

CODE

certificates/mysql_certs

Place all certificate files in the certificates/mysql_certs directory and create a ConfigMap for MySQL certificates to enable TLS encryption. The certificates should include the following files:

ca.pem
client-cert.pem
client-key.pem

CODE

kubectl -n expertflow create secret generic  ef-transflux-mysql-certs-secret  --from-file=certificates/mysql_certs

Create configuration ConfigMaps for CX-Transflux pipelines.

CODE

kubectl -n expertflow create configmap ef-transflux-config-cm --from-file=config

Finally, deploy CX-Transflux by replacing <FQDN> with your dedicated FQDN for CX-Transflux.

BASH

helm upgrade --install --namespace expertflow   --set global.efCxReleaseName="ef-cx" --set global.ingressRouter=<FQDN>  cx-transflux --debug --values helm-values/cx-transflux-custom-values.yaml  expertflow/transflux