Upgrade guide from CX-4.9.4 to CX-4.10 for Data Platform

Review the specific pipeline document to understand the configurations and how to run the pipeline.

1. Pause all the pipelines from the dedicated FQDN

2. Add the following new configuration files into the transflux/config/ folder within transflux directory and update them as per the dedicated source and target databases configurations. The steps to edit the configuration file and run the data pipelines are in the subsequent documents attached for each pipeline.

   1. qm_data_pipeline_config.yaml / https://expertflow-docs.atlassian.net/wiki/x/AwAPQg

   2. qm_cisco_team_sync_config.yaml / https://expertflow-docs.atlassian.net/wiki/x/tgC4QQ

   3. keyclock_users_data_pipeline_config.yaml / https://expertflow-docs.atlassian.net/wiki/x/XwDhQQ

   4. data_migration_config.yaml / Data Migration On-boarding on Expertflow Data Platform | Configurations

   5. Note: Use https for solution fqdn in bulk_upload_config.yaml /https://expertflow-docs.atlassian.net/wiki/x/MYCXRQ

   6. gold_15_minutes_queries_config.yaml / https://expertflow-docs.atlassian.net/wiki/x/7QBkR

   7. graphana_queries.yaml / https://expertflow-docs.atlassian.net/wiki/x/ewDXRg

3. Edit the previous/existing forms_data_pipeline_config.yaml configuration file as follows:

# Add the following query under the 'queries' section in transflux/config/forms_data_pipeline_config.yaml.
# Make sure to add this query after the 'survey_distributions' query.

    forms_schema:
      database: "adminPanel"
      collection_name: "forms"
      filter: {}
      replication_key: "updatedAt"
      transformation: "transform_forms_schema_data"
      num_batches: 50
      query_keys:

4. Edit the previous/existing data_migration_config.yaml configuration file and

   1. add these lines underbatch_processing section:

    agentStateChangeEvents-4.10:     # migration to run
      js_file: "corrupt-agent-mrd-data.js"
      start_date: "2025-07-01" # Should be updated according to data
      end_date: "2025-07-09" # Should be updated according to data
      interval: "720" ## minute-wise interval (0.5 day = 720)

b. add these lines undernon_batch_processingsection:

    RE_MRD-4.10:
      js_file: "re-mrd-id-fix.js"
    forms-schema-update-4.10:
      js_file: "forms-schema-update-4_10.js"

5. Re-create the config maps

   CODE

   # Delete config Maps
   
   k delete cm ef-transflux-config-cm -n expertflow
   
   # Create config Map from the transflux directory
   
   kubectl -n expertflow create configmap ef-transflux-config-cm --from-file=config

6. Update the tag for cx-transflux in transflux/helm-values/cx-transflux-custom-values.yaml and deploy it.

vi helm-values/cx-transflux-custom-values.yaml

tag: 4.10

#Re-deploy CX-Transflux
helm upgrade --install --namespace expertflow   --set global.efCxReleaseName="ef-cx"  cx-transflux --debug --values helm-values/cx-transflux-custom-values.yaml expertflow/transflux

5. Perform alembic migration for 4.10 mentioned in Alembic Schema Migration

6. Turn on the dedicated pipelines from the FQDN as per requirements

7. Once done, go to trasflux UI, login and run the pipelines Routing_Engine_mRD_Batch_Migration_4.10, Routing_Engine_mRD_no_Batch_Migration_4.10 and graphana_queries.