Data Platform Deployment Guide

Prerequisites for EF Data Platform Deployment

Deployment

Clone CX-Transflux Repository

git clone -b 5.1.0 https://efcx:RecRpsuH34yqp56YRFUb@gitlab.expertflow.com/cim/transflux.git transflux

cd transflux

Add the Expertflow Helm charts repository.

helm repo add expertflow https://expertflow.github.io/charts

Update the charts repository

helm repo update expertflow

Create a folder to save the Helm chart’s values

mkdir helm-values

Customise the deployment by creating the custom-values.yaml file and add the custom configurations as per the requirements.

vi helm-values/cx-transflux-custom-values.yaml 

Use the following command to see the default values.yaml

helm show values expertflow/transflux --version 5.1.0

Open the file helm-values/cx-transflux-custom-values.yaml and edit it according to the given information, which is required for the CX Transflux to work properly.

Below is the sample snippet after editing helm-values/cx-transflux-custom-values.yaml

global:
  ingressRouter: "transflux-rm-02.expertflow.com"
image:
  tag: "5.1.0"

mkdir /tmp/mongodb_certs
CERTFILES=($(kubectl get secret mongo-mongodb-ca -n ef-external -o go-template='{{range $k,$v := .data}}{{$k}}{{"\n"}}{{end}}'))
for f in ${CERTFILES[*]}; do   kubectl get secret mongo-mongodb-ca  -n ef-external -o go-template='{{range $k,$v := .data}}{{ if eq $k "'$f'"}}{{$v  | base64decode}}{{end}}{{end}}' > /tmp/mongodb_certs/${f} 2>/dev/null; done

Create a directory for TLS certificates

mkdir -p certificates/mongo_certs
mkdir -p certificates/mysql_certs

Copy all the certificate files present in /tmp/mongodb_certs directory and place them in the newly created directory certificates/mongo_certs in the transflux directory. These certificate files should include the following:

• mongodb-ca-key.pem

• mongodb-ca-cert.pem

• client-pem.pem

Place all certificate files in the certificates/mysql_certs directory and create a ConfigMap for MySQL certificates to enable TLS encryption. The certificates should include the following files:

• ca.pem

• client-cert.pem

• client-key.pem

kubectl -n expertflow create secret generic  ef-transflux-mysql-certs-secret  --from-file=certificates/mysql_certs

For On-Prem (Single tenant) Deployment

Edit the file transflux/config/tenants.yaml and configure the following

1. By default, the single tenant’s name is expertflow else it would be the user define defined <tenant-id>

2. FQDN_URL in api

3. TARGET_TYPE (mysql or mssql) as per the dedicated target database

4. Edit the existing transflux/config/tenants.yaml

   
    TARGET_TYPE: "mysql"
    MYSQL:
      TARGET_HOST: "192.168.2.18"
      TARGET_PORT: "3306"
      TARGET_USERNAME: "monty"
      TARGET_PASSWORD: "Expertflow#143"
      TARGET_SSL_ENABLED: false
      TARGET_DATABASE: "hold_db"
    MSSQL:
      TARGET_HOST: "192.168.1.77"
      TARGET_PORT: "1433"
      TARGET_USERNAME: "sa"
      TARGET_PASSWORD: "Expertflow464"
      TARGET_SSL_ENABLED: false
      TARGET_DATABASE: "hold_db"

   
    TARGET_TYPE: "mysql"
    TARGET_HOST: "192.168.2.18"
    TARGET_PORT: "3306"
    TARGET_USERNAME: "monty"
    TARGET_PASSWORD: "Expertflow#143"
    TARGET_SSL_ENABLED: false
    TARGET_DATABASE: "hold_db"

   
    TARGET_TYPE: "mssql"
    TARGET_HOST: "192.168.1.77"
    TARGET_PORT: "1433"
    TARGET_USERNAME: "sa"
    TARGET_PASSWORD: "Expertflow464"
    TARGET_SSL_ENABLED: false
    TARGET_DATABASE: "hold_db"

The new tenants.yaml file will look something like this for MYSQL.

tenants:
  expertflow: # or user defined <tenant-id>
    mongodb:
      SOURCE_HOST: "mongo-mongodb.ef-external.svc"
      SOURCE_PORT: "27017"
      SOURCE_USERNAME: "root"
      SOURCE_PASSWORD: "Expertflow123"
      SOURCE_TLS_ENABLED: true
      SOURCE_DATABASE: "expertflow" # or user defined <tenant-id> (database)
    postgre:
      SOURCE_HOST: "ef-postgresql.ef-external.svc"
      SOURCE_PORT: "5432"
      SOURCE_USERNAME: "sa"
      SOURCE_PASSWORD: "Expertflow123"
      SOURCE_DATABASE: "qm_db"
    api:
      FQDN_URL: "<FQDN URL>"
      REALM: "expertflow"       
    TARGET_TYPE: "mysql"
    TARGET_HOST: "192.168.2.18"
    TARGET_PORT: "3306"
    TARGET_USERNAME: "monty"
    TARGET_PASSWORD: "Expertflow#143"
    TARGET_SSL_ENABLED: false
    TARGET_DATABASE: "hold_db"

For Cloud (Multi-tenant) Deployment

1. In order to register new tenants, replicate the previous tenant configuration and append the new settings afterward, set the TARGET_TYPE as per the dedicated target database, and configure the database credentials accordingly. You can also update the existing tenant's information as per the following setup.
   The new tenants.yaml file will look something like this for MYSQL.

     <tenant_name>:
       mongodb:
         SOURCE_HOST: "mongo-mongodb.ef-external.svc"
         SOURCE_PORT: "27017"
         SOURCE_USERNAME: "root"
         SOURCE_PASSWORD: "Expertflow123"
         SOURCE_TLS_ENABLED: true
         SOURCE_DATABASE: "<tenant_name>"
       postgre:
         SOURCE_HOST: "ef-postgresql.ef-external.svc"
         SOURCE_PORT: "5432"
         SOURCE_USERNAME: "sa"
         SOURCE_PASSWORD: "Expertflow123"
         SOURCE_DATABASE: "<tenant_name>"
       api:
         FQDN_URL: "https://<tenant_name>.expertflow.com"
         REALM: "<tenant_name>"
       TARGET_TYPE: "mysql"
       TARGET_HOST: "82.208.20.221"
       TARGET_PORT: "30801"
       TARGET_USERNAME: "root"
       TARGET_PASSWORD: "Expertflow123$#"
       TARGET_SSL_ENABLED: false
       TARGET_DATABASE: "<tenant_name>"
   

For Cisco Sync Job Configuration

Edit the file transflux/config/qm_cisco_team_sync_config.yaml and update the endpoint with FQDN on which cisco sync job is configured

endpoint: "https://{FQDN}/cisco-sync-service/api/v1/sync"

Create configuration ConfigMaps for CX-Transflux pipelines with multi-tenancy configurations.

kubectl -n expertflow create configmap ef-transflux-config-cm --from-file=config

Create configuration ConfMaps for dbt_schema in transflux

kubectl -n expertflow create configmap ef-transflux-dbt-schema-cm --from-file=dbt_schema

Finally, deploy CX-Transflux.

helm upgrade --install --namespace expertflow   --set global.efCxReleaseName="ef-cx"  cx-transflux --debug --values helm-values/cx-transflux-custom-values.yaml  expertflow/transflux --version 5.1.0

Follow the User Manual to control pipelines from the Data Platform