Expertflow ETL Deployment

Requirements

A dedicated Fully Qualified Domain Name (FQDN) is required for EF ETL.
Create the target database in the Database Management System by executing the scripts located at:
kubernetes/pre-deployment/reportingConnector/dbScripts/dbcreation/_historical_reports_db_creation_script_MySQL.sql.

Follow this guide to create database: https://expertflow-docs.atlassian.net/wiki/x/QoGrGQ

Deployment

Clone EF ETL Repository

CODE

git clone https://efcx:RecRpsuH34yqp56YRFUb@gitlab.expertflow.com/cim/EF ETL.git -b 4.7

CODE

cd EF ETL

Add the Expertflow Helm charts repository.

CODE

helm repo add expertflow https://expertflow.github.io/charts

Update the charts repository

CODE

helm repo update expertflow

Create a database with name ‘airflow’ in MySQL.

CODE

CREATE DATABASE airflow;

Create folder to save the helm chart’s values

CODE

mkdir helm-values

Customize the deployment by fetching the values file and edit it as per requirements.

CODE

helm show values expertflow/EF ETL  > helm-values/cx-EF ETL-custom-values.yaml

Open the file helm-values/cx-EF ETL-custom-values.yaml and edit it according to the below given information which is required for the CX EF ETL to work properly.

Value	Updated Value
AIRFLOW_DB_USER	Airflow DB Username in MySQL
AIRFLOW_DB_PASSWORD	Airflow DB Password in MySQL
AIRFLOW_DB_NAME	airflow
AIRFLOW__CORE__SQL_ALCHEMY_CONN	Hostname and port number must be updated
MYSQL_HOST	Upstream MySQL host IP
MYSQL_ROOT_PASSWORD	Password for MySQL user
MONGODB_PASSWORD	Update the local MongoDB password, when using non-default password

For additional parameters, refer to the extraEnvVars section in the helm-values/cx-EF ETL-custom-values.yaml file.

To configure CX-EF ETL, update the configuration files with your specific values. Open the following files and ensure the required information is correctly set:

config/forms_data_pipeline_config.yaml
config/teams_data_pipeline_config.yaml

CODE

target:
  type: "mysql"
  db_url: "mysql+pymysql://<your-db-username>:<password>@<host>:<port>/<mysql-db-name>"
  
configdb:
  type: "mysql"
  db_url: "mysql+pymysql://<your-db-username>:<password>@<host>:<port>/<mysql-db-name>"

Create directory for upstream MySQL server TLS certificates

CODE

mkdir -p certificates/mysql_certs

Place all certificate files in the certificates/mysql_certs directory and create a ConfigMap for MySQL certificates to enable TLS encryption. The certificates should include the following files:

ca.pem
client-cert.pem
client-key.pem

CODE

kubectl -n expertflow create secret generic  ef-EF ETL-mysql-certs-secret  --from-file=certificates/mysql_certs

Create configuration ConfigMaps for CX-EF ETL pipelines.

CODE

kubectl -n expertflow create configmap ef-EF ETL-config-cm --from-file=config

Finally, deploy CX-EF ETL by replacing <FQDN> with your dedicated FQDN for CX-EF ETL.

BASH

helm upgrade --install --namespace expertflow   --set global.efCxReleaseName="ef-cx" --set global.ingressRouter=<FQDN>  cx-EF ETL --debug --values helm-values/cx-EF ETL-custom-values.yaml  expertflow/EF ETL