Data Platform CX

Prerequisites for EF Data Platform Deployment

Deployment

Clone CX-Transflux Repository

git clone -b 5.0 https://efcx:RecRpsuH34yqp56YRFUb@gitlab.expertflow.com/cim/transflux.git transflux

cd transflux

Add the Expertflow Helm charts repository.

helm repo add expertflow https://expertflow.github.io/charts

Update the charts repository

helm repo update expertflow

Create a folder to save the Helm chart’s values

mkdir helm-values

Customise the deployment by creating the custom-values.yaml file and add the custom configurations as per the requirements.

vi helm-values/cx-transflux-custom-values.yaml 

Use the following command to see the default values.yaml

helm show values expertflow/transflux --version 5.0

Open the file helm-values/cx-transflux-custom-values.yaml and edit it according to the given information, which is required for the CX Transflux to work properly.

Below is the sample snippet after editing helm-values/cx-transflux-custom-values.yaml

global:
  ingressRouter: "transflux-rm-02.expertflow.com"
image:
  tag: "5.0"

mkdir /tmp/mongodb_certs
CERTFILES=($(kubectl get secret mongo-mongodb-ca -n ef-external -o go-template='{{range $k,$v := .data}}{{$k}}{{"\n"}}{{end}}'))
for f in ${CERTFILES[*]}; do   kubectl get secret mongo-mongodb-ca  -n ef-external -o go-template='{{range $k,$v := .data}}{{ if eq $k "'$f'"}}{{$v  | base64decode}}{{end}}{{end}}' > /tmp/mongodb_certs/${f} 2>/dev/null; done

Create a directory for TLS certificates

mkdir -p certificates/mongo_certs
mkdir -p certificates/mysql_certs

Copy all the certificate files present in /tmp/mongodb_certs directory and place them in the newly created directory certificates/mongo_certs in the transflux directory. These certificate files should include the following:

• mongodb-ca-key.pem

• mongodb-ca-cert.pem

• client-pem.pem

Place all certificate files in the certificates/mysql_certs directory and create a ConfigMap for MySQL certificates to enable TLS encryption. The certificates should include the following files:

• ca.pem

• client-cert.pem

• client-key.pem

kubectl -n expertflow create secret generic  ef-transflux-mysql-certs-secret  --from-file=certificates/mysql_certs

For On-Prem (Single tenant) Deployment

Edit the file transflux/config/tenants.yaml and configure the following

1. By default, the single tenant’s name is expertflow else it would be the user define defined <tenant-id>

2. FQDN_URL in api

3. TARGET_TYPE (mysql or mssql) as per the dedicated target database

4. For selected target type, update the dedicated MYSQL or MSSQL credentials

   tenants:
     expertflow: # or user defined <tenant-id>
       
       mongodb:
   
         SOURCE_HOST: "mongo-mongodb.ef-external.svc"
         SOURCE_PORT: "27017"
         SOURCE_USERNAME: "root"
         SOURCE_PASSWORD: "Expertflow123"
         SOURCE_TLS_ENABLED: true
         SOURCE_DATABASE: "expertflow" # or user defined <tenant-id> (database)
   
       postgre:
   
         SOURCE_HOST: "ef-postgresql.ef-external.svc"
         SOURCE_PORT: "5432"
         SOURCE_USERNAME: "sa"
         SOURCE_PASSWORD: "Expertflow123"
         SOURCE_DATABASE: "qm_db"
       
       api:
   
         FQDN_URL: "<FQDN URL>"
         REALM: "expertflow"       
   
       TARGET_TYPE: "mysql"
   
       MYSQL:
         TARGET_HOST: "192.168.2.18"
         TARGET_PORT: "3306"
         TARGET_USERNAME: "monty"
         TARGET_PASSWORD: "Expertflow#143"
         TARGET_SSL_ENABLED: false
         TARGET_DATABASE: "hold_db"
   
       MSSQL:
         TARGET_HOST: "192.168.1.77"
         TARGET_PORT: "1433"
         TARGET_USERNAME: "sa"
         TARGET_PASSWORD: "Expertflow464"
         TARGET_SSL_ENABLED: false
         TARGET_DATABASE: "hold_db"
   

For Cloud (Multi-tenant) Deployment

1. In order to register new tenants, replicate the previous tenant configuration and append the new settings afterward, set the TARGET_TYPE as per the dedicated target database, and configure the database credentials accordingly. You can also update the existing tenant's information as per the following setup

     <tenant_name>:
   
       mongodb:
       
         SOURCE_HOST: "mongo-mongodb.ef-external.svc"
         SOURCE_PORT: "27017"
         SOURCE_USERNAME: "root"
         SOURCE_PASSWORD: "Expertflow123"
         SOURCE_TLS_ENABLED: true
         SOURCE_DATABASE: "<tenant_name>"
   
       postgre:
   
         SOURCE_HOST: "ef-postgresql.ef-external.svc"
         SOURCE_PORT: "5432"
         SOURCE_USERNAME: "sa"
         SOURCE_PASSWORD: "Expertflow123"
         SOURCE_DATABASE: "<tenant_name>"
   
       api:
       
         FQDN_URL: "https://<tenant_name>.expertflow.com"
         REALM: "<tenant_name>"
   
       TARGET_TYPE: "mysql"
   
       MYSQL:
         TARGET_HOST: "82.208.20.221"
         TARGET_PORT: "30801"
         TARGET_USERNAME: "root"
         TARGET_PASSWORD: "Expertflow123$#"
         TARGET_SSL_ENABLED: false
         TARGET_DATABASE: "<tenant_name>"
   
       MSSQL:
         TARGET_HOST: "192.168.1.77"
         TARGET_PORT: "1433"
         TARGET_USERNAME: "sa"
         TARGET_PASSWORD: "Expertflow464"
         TARGET_SSL_ENABLED: false
         TARGET_DATABASE: "<tenant_name>"
   

For Cisco Sync Job Configuration

Edit the file transflux/config/qm_cisco_team_sync_config.yaml and update the endpoint with FQDN on which cisco sync job is configured

endpoint: "https://{FQDN}/cisco-sync-service/api/v1/sync"

Create configuration ConfigMaps for CX-Transflux pipelines with multi-tenancy configurations.

kubectl -n expertflow create configmap ef-transflux-config-cm --from-file=config

Create configuration ConfMaps for dbt_schema in transflux

kubectl -n expertflow create configmap ef-transflux-dbt-schema-cm --from-file=dbt_schema

Finally, deploy CX-Transflux.

helm upgrade --install --namespace expertflow   --set global.efCxReleaseName="ef-cx"  cx-transflux --debug --values helm-values/cx-transflux-custom-values.yaml  expertflow/transflux --version 5.0

Follow the User Manual to control pipelines from the Data Platform