Expertflow ETL deployment - A1

Requirements

A dedicated Fully Qualified Domain Name (FQDN) is required for CX Transflux.

Deployment

Clone CX-Transflux Repository

CODE

git clone https://efcx:RecRpsuH34yqp56YRFUb@gitlab.expertflow.com/cim/transflux.git -b 4.6.1_f-BI-60

CODE

cd transflux

Add the Expertflow Helm charts repository.

CODE

helm repo add expertflow https://expertflow.github.io/charts

Update the charts repository

CODE

helm repo update expertflow

Create a database with name ‘airflow’ in MySQL.

CODE

CREATE DATABASE airflow;

Create folder to save the helm chart’s values

CODE

mkdir helm-values

Customize the deployment by fetching the values file and edit it as per requirements.

CODE

helm show values expertflow/transflux  > helm-values/cx-transflux-custom-values.yaml

Replace this file cx-transflux-custom-values.yaml with the already present file cx-transflux-custom-values.yaml
Verify the tag is 4.7_f-BI-60 in above file.

Open the file helm-values/cx-transflux-custom-values.yaml and edit it according to the below given information which is required for the CX Transflux to work properly.

Value	Updated Value
AIRFLOW_DB_USER	Airflow DB Username in MySQL
AIRFLOW_DB_PASSWORD	Airflow DB Password in MySQL
AIRFLOW_DB_NAME	Airflow DB Name
AIRFLOW__CORE__SQL_ALCHEMY_CONN	Hostname and port number must be updated
MYSQL_HOST	Upstream MySQL host IP
MYSQL_ROOT_PASSWORD	Password for MySQL user
MONGODB_PASSWORD	Update the local MongoDB password, when using non-default password

For additional parameters, refer to the extraEnvVars section in the helm-values/cx-transflux-custom-values.yaml file.

To configure CX-Transflux, update the configuration files with your specific values. Open the following files and ensure the required information is correctly set:

config/forms_data_pipeline_config.yaml
config/teams_data_pipeline_config.yaml

CODE

target:
  type: "mysql"
  db_url: "mysql+pymysql://<your-db-username>:<password>@<host>:<port>/<mysql-db-name>"
  
configdb:
  type: "mysql"
  db_url: "mysql+pymysql://<your-db-username>:<password>@<host>:<port>/<mysql-db-name>"

Create configuration ConfigMaps for CX-Transflux pipelines.

CODE

kubectl -n expertflow create configmap ef-transflux-config-cm --from-file=config

Finally, deploy CX-Transflux by replacing <FQDN> with your dedicated FQDN for CX-Transflux.

BASH

helm upgrade --install --namespace expertflow   --set global.efCxReleaseName="ef-cx" --set global.ingressRouter=<FQDN>  cx-transflux --debug --values helm-values/cx-transflux-custom-values.yaml  expertflow/transflux --version 4.7.0-alpha