Expertflow ETL Deployment

Requirements

A dedicated Fully Qualified Domain Name (FQDN) is required for CX Transflux.
Create the target database in the Database Management System by executing the scripts located at:
kubernetes/pre-deployment/reportingConnector/dbScripts/dbcreation/_historical_reports_db_creation_script_MySQL.sql.

Follow this guide to create database: Guide For Setting Up or Upgrading MySQL Database

Deployment

Clone CX-Transflux Repository

CODE

git clone https://efcx:RecRpsuH34yqp56YRFUb@gitlab.expertflow.com/cim/transflux.git -b 4.7

CODE

cd transflux

Add the Expertflow Helm charts repository.

CODE

helm repo add expertflow https://expertflow.github.io/charts

Update the charts repository

CODE

helm repo update expertflow

Create a folder to save the helm chart’s values

CODE

mkdir helm-values

Customize the deployment by fetching the values file and edit it as per requirements.

CODE

helm show values expertflow/transflux --version 4.8.0 > helm-values/cx-transflux-custom-values.yaml

Open the file helm-values/cx-transflux-custom-values.yaml and edit it according to the below given information which is required for the CX Transflux to work properly.

Value	Updated Value
ingressRouter	Dedicated Fully Qualified Domain Name (FQDN)
tag	`4.8`
MONGODB_PASSWORD	Update the local MongoDB password, when using non-default password
AIRFLOW__CORE__SQL_ALCHEMY_CONN	Update the local PostgreSQL password, when using non-default password as below postgresql+psycopg2://sa:<your-non-default-password>@ef-postgresql.ef-external.svc:5432/airflow?sslmode=verify-ca&sslrootcert=/postgresql/ca.crt

For additional parameters, refer to the extraEnvVars section in the helm-values/cx-transflux-custom-values.yaml file.

To configure CX-Transflux, update the configuration files with your specific values. Open the config files in /kubernetes/transflux/config directory and ensure the required information is correctly set:

In case of MSSQL Target database

CODE

target:
  type: "mssql"
  db_url: "mssql+pyodbc://<your-db-username>:<password>@<host>:<port>/<mssql-db-name>?driver=ODBC+Driver+17+for+SQL+Server"
  
configdb:
  type: "mssql"
  db_url: "mssql+pyodbc://<your-db-username>:<password>@<host>:<port>/<mssql-db-name>?driver=ODBC+Driver+17+for+SQL+Server"

In case of MySql Target database

CODE

target:
  type: "mysql"
  db_url: "mysql+pymysql://<your-db-username>:<password>@<host>:<port>/<mysql-db-name>"
  
configdb:
  type: "mysql"
  db_url: "mysql+pymysql://<your-db-username>:<password>@<host>:<port>/<mysql-db-name>"

place the MySQL certs in the following directory

CODE

certificates/mysql_certs

Place all certificate files in the certificates/mysql_certs directory and create a ConfigMap for MySQL certificates to enable TLS encryption. The certificates should include the following files:

ca.pem
client-cert.pem
client-key.pem

CODE

kubectl -n expertflow create secret generic  ef-transflux-mysql-certs-secret  --from-file=certificates/mysql_certs

Create a directory for TLS certificates

CODE

mkdir -p certificates
Copy the mogoDB certs in certificates/mongo_certs

Create configuration ConfigMaps for CX-Transflux pipelines.

CODE

kubectl -n expertflow create configmap ef-transflux-config-cm --from-file=config

Finally, deploy CX-Transflux.

BASH

helm upgrade --install --namespace expertflow   --set global.efCxReleaseName="ef-cx"  cx-transflux --debug --values helm-values/cx-transflux-custom-values.yaml  expertflow/transflux