Expertflow ETL Deployment

Prerequisites for Expertflow ETL Deployment

Fully Qualified Domain Name (FQDN)
A dedicated FQDN is required for CX Transflux ( EF ETL ) to ensure proper routing and secure communication.
Database Setup
- Create the target database in your Database Management System by executing the following script:
  CODE
```
kubernetes/pre-deployment/reportingConnector/dbScripts/dbcreation/_historical_reports_db_creation_script_MySQL.sql 
```
- Ensure that the executing user has sufficient privileges to create databases and tables.

Follow this guide to create database

Resource Requirements:
- Minimum CPU: 2 cores
- Minimum Memory: 4 GB RAM
- These resources are essential for optimal performance of the CX Transflux ( EF ETL ) components during data processing and ETL operations.

Deployment

Clone CX-Transflux Repository

CODE

git clone https://efcx:RecRpsuH34yqp56YRFUb@gitlab.expertflow.com/cim/transflux.git -b 4.7

CODE

cd transflux

Add the Expertflow Helm charts repository.

CODE

helm repo add expertflow https://expertflow.github.io/charts

Update the charts repository

CODE

helm repo update expertflow

Create a folder to save the helm chart’s values

CODE

mkdir helm-values

Customize the deployment by fetching the values file and edit it as per requirements.

CODE

helm show values expertflow/transflux --version 4.9.0 > helm-values/cx-transflux-custom-values.yaml

Open the file helm-values/cx-transflux-custom-values.yaml and edit it according to the below given information which is required for the CX Transflux to work properly.

Value	Updated Value
ingressRouter	Dedicated Fully Qualified Domain Name (FQDN)
tag	`4.9`
MONGODB_PASSWORD	Update the local MongoDB password, when using non-default password
AIRFLOW__CORE__SQL_ALCHEMY_CONN	Update the local PostgreSQL password, when using non-default password as below postgresql+psycopg2://sa:<your-non-default-password>@ef-postgresql.ef-external.svc:5432/airflow?sslmode=verify-ca&sslrootcert=/postgresql/ca.crt

For additional parameters, refer to the extraEnvVars section in the helm-values/cx-transflux-custom-values.yaml file.

To configure CX-Transflux, update the configuration files with your specific values. Open the following config files in /kubernetes/transflux/config directory and ensure the required information is correctly set:

In case of MSSQL Target database

CODE

target:
  type: "mssql"
  db_url: "mssql+pyodbc://<your-db-username>:<password>@<host>:<port>/<mssql-db-name>?driver=ODBC+Driver+17+for+SQL+Server"
  
configdb:
  type: "mssql"
  db_url: "mssql+pyodbc://<your-db-username>:<password>@<host>:<port>/<mssql-db-name>?driver=ODBC+Driver+17+for+SQL+Server"

In case of MySql Target database

CODE

target:
  type: "mysql"
  db_url: "mysql+pymysql://<your-db-username>:<password>@<host>:<port>/<mysql-db-name>"
  
configdb:
  type: "mysql"
  db_url: "mysql+pymysql://<your-db-username>:<password>@<host>:<port>/<mysql-db-name>"

place the MySQL certs in the following directory

CODE

certificates/mysql_certs

Place all certificate files in the certificates/mysql_certs directory and create a ConfigMap for MySQL certificates to enable TLS encryption. The certificates should include the following files:

ca.pem
client-cert.pem
client-key.pem

CODE

kubectl -n expertflow create secret generic  ef-transflux-mysql-certs-secret  --from-file=certificates/mysql_certs

Create a directory for TLS certificates

CODE

mkdir -p certificates
Copy the mogoDB certs in certificates/mongo_certs

Create configuration ConfigMaps for CX-Transflux pipelines.

CODE

kubectl -n expertflow create configmap ef-transflux-config-cm --from-file=config

Finally, deploy CX-Transflux.

BASH

helm upgrade --install --namespace expertflow   --set global.efCxReleaseName="ef-cx"  cx-transflux --debug --values helm-values/cx-transflux-custom-values.yaml  expertflow/transflux --version 4.9.0