Breadcrumbs

Expertflow ETL Deployment

Prerequisites for Expertflow ETL Deployment

  1. Fully Qualified Domain Name (FQDN)
    A dedicated FQDN is required for CX Transflux ( EF ETL ) to ensure proper routing and secure communication.

  2. Database Setup

    • Create the target database in your Database Management System by executing the following script:

      kubernetes/pre-deployment/reportingConnector/dbScripts/dbcreation/_historical_reports_db_creation_script_MySQL.sql

    • Ensure that the executing user has sufficient privileges to create databases and tables.

Follow this guide to create database

  1. Resource Requirements:

    • Minimum CPU: 2 cores

    • Minimum Memory: 4 GB RAM

    • These resources are essential for optimal performance of the CX Transflux ( EF ETL ) components during data processing and ETL operations.

Deployment

Clone CX-Transflux Repository

git clone https://efcx:RecRpsuH34yqp56YRFUb@gitlab.expertflow.com/cim/transflux.git -b 4.7

cd transflux

Add the Expertflow Helm charts repository.

helm repo add expertflow https://expertflow.github.io/charts

Update the charts repository

helm repo update expertflow

Create a folder to save the helm chart’s values

mkdir helm-values

Customize the deployment by fetching the values file and edit it as per requirements.

helm show values expertflow/transflux --version 4.9.0 > helm-values/cx-transflux-custom-values.yaml

Open the file helm-values/cx-transflux-custom-values.yaml and edit it according to the below given information which is required for the CX Transflux to work properly.

Value

Updated Value

ingressRouter

Dedicated Fully Qualified Domain Name (FQDN)

tag

4.9

MONGODB_PASSWORD

Update the local MongoDB password, when using non-default password

AIRFLOW__CORE__SQL_ALCHEMY_CONN

Update the local PostgreSQL password, when using non-default password as below

postgresql+psycopg2://sa:<your-non-default-password>@ef-postgresql.ef-external.svc:5432/airflow?sslmode=verify-ca&sslrootcert=/postgresql/ca.crt

For additional parameters, refer to the extraEnvVars section in the helm-values/cx-transflux-custom-values.yaml file.

To configure CX-Transflux, update the configuration files with your specific values. Open the following config files in /kubernetes/transflux/config directory and ensure the required information is correctly set:

In case of MSSQL Target database

target:
  type: "mssql"
  db_url: "mssql+pyodbc://<your-db-username>:<password>@<host>:<port>/<mssql-db-name>?driver=ODBC+Driver+17+for+SQL+Server"
  
configdb:
  type: "mssql"
  db_url: "mssql+pyodbc://<your-db-username>:<password>@<host>:<port>/<mssql-db-name>?driver=ODBC+Driver+17+for+SQL+Server"

In case of MySql Target database

target:
  type: "mysql"
  db_url: "mysql+pymysql://<your-db-username>:<password>@<host>:<port>/<mysql-db-name>"
  
configdb:
  type: "mysql"
  db_url: "mysql+pymysql://<your-db-username>:<password>@<host>:<port>/<mysql-db-name>"

place the MySQL certs in the following directory

certificates/mysql_certs

Place all certificate files in the certificates/mysql_certs directory and create a ConfigMap for MySQL certificates to enable TLS encryption. The certificates should include the following files:

  • ca.pem

  • client-cert.pem

  • client-key.pem

kubectl -n expertflow create secret generic  ef-transflux-mysql-certs-secret  --from-file=certificates/mysql_certs

Create a directory for TLS certificates

mkdir -p certificates
Copy the mogoDB certs in certificates/mongo_certs

Create configuration ConfigMaps for CX-Transflux pipelines.

kubectl -n expertflow create configmap ef-transflux-config-cm --from-file=config

Finally, deploy CX-Transflux.

Bash
helm upgrade --install --namespace expertflow   --set global.efCxReleaseName="ef-cx"  cx-transflux --debug --values helm-values/cx-transflux-custom-values.yaml  expertflow/transflux --version 4.9.0