Expertflow ETL Deployment

Requirements

A dedicated Fully Qualified Domain Name (FQDN) is required for CX Transflux.
Create the target database in the Database Management System by executing the scripts located at:
kubernetes/pre-deployment/reportingConnector/dbScripts/dbcreation/_historical_reports_db_creation_script_MySQL.sql.

Follow this guide to create database: Guide For Setting Up or Upgrading MySQL Database

Deployment

Clone CX-Transflux Repository

CODE

git clone https://efcx:RecRpsuH34yqp56YRFUb@gitlab.expertflow.com/cim/transflux.git -b 4.7

CODE

cd transflux

Add the Expertflow Helm charts repository.

CODE

helm repo add expertflow https://expertflow.github.io/charts

Update the charts repository

CODE

helm repo update expertflow

Create a folder to save the helm chart’s values

CODE

mkdir helm-values

Customize the deployment by fetching the values file and edit it as per requirements.

CODE

helm show values expertflow/transflux  > helm-values/cx-transflux-custom-values.yaml

Open the file helm-values/cx-transflux-custom-values.yaml and edit it according to the below given information which is required for the CX Transflux to work properly.

Value	Updated Value
ingressRouter	"transflux-rmt2.expertflow.com"
tag	`4.8`
MONGODB_PASSWORD	Update the local MongoDB password, when using non-default password
AIRFLOW__CORE__SQL_ALCHEMY_CONN	Update the local PostgreSQL password, when using non-default password as below postgresql+psycopg2://sa:<your-non-default-password>@ef-postgresql.ef-external.svc:5432/airflow?sslmode=verify-ca&sslrootcert=/postgresql/ca.crt

For additional parameters, refer to the extraEnvVars section in the helm-values/cx-transflux-custom-values.yaml file.

To configure CX-Transflux, update the configuration files with your specific values. Open the following config files in /kubernetes/transflux/config directory and ensure the required information is correctly set:

In case of MSSQL Target database

CODE

target:
  type: "mssql"
  db_url: "mssql+pyodbc://<your-db-username>:<password>@<host>:<port>/<mssql-db-name>?driver=ODBC+Driver+17+for+SQL+Server"
  
configdb:
  type: "mssql"
  db_url: "mssql+pyodbc://<your-db-username>:<password>@<host>:<port>/<mssql-db-name>?driver=ODBC+Driver+17+for+SQL+Server"

In case of MySql Target database

CODE

target:
  type: "mysql"
  db_url: "mysql+pymysql://<your-db-username>:<password>@<host>:<port>/<mysql-db-name>"
  
configdb:
  type: "mysql"
  db_url: "mysql+pymysql://<your-db-username>:<password>@<host>:<port>/<mysql-db-name>"

place the MySQL certs in the following directory

CODE

certificates/mysql_certs

Place all certificate files in the certificates/mysql_certs directory and create a ConfigMap for MySQL certificates to enable TLS encryption. The certificates should include the following files:

ca.pem
client-cert.pem
client-key.pem

CODE

kubectl -n expertflow create secret generic  ef-transflux-mysql-certs-secret  --from-file=certificates/mysql_certs

Create a directory for TLS certificates

CODE

mkdir -p certificates
Copy the mogoDB certs in certificates/mongo_certs

Create configuration ConfigMaps for CX-Transflux pipelines.

CODE

kubectl -n expertflow create configmap ef-transflux-config-cm --from-file=config

Finally, deploy CX-Transflux.

BASH

helm upgrade --install --namespace expertflow   --set global.efCxReleaseName="ef-cx"  cx-transflux --debug --values helm-values/cx-transflux-custom-values.yaml  expertflow/transflux