Data Migrations 4.4.10 to 4.10

Executive Summary

With the release of CX 4.10, incoming data will now adhere to the updated 4.10 schema, reflecting the schema modifications introduced in previous versions. However, legacy data stored in MongoDB remains in the format of earlier releases, necessitating a migration to align it with the updated schema.

To ensure compatibility between legacy data and the current release, data migration scripts have been developed. These scripts efficiently transform the data from the previous format to the 4.10 schema and are seamlessly integrated into the Experflow ETL Data Platform for automation and streamlined execution.

Migration Plan

We have developed migrating data pipeline. Each pipeline serves a distinct purpose and should be executed in a provided order to ensure the migration operation is completed successfully. The workflows are flexible, allowing pipelines to be executed in different intervals based on the requirements.

Before performing the activity, make sure that the collections conversations and CustomerTopicEvents have indexes in place within the mongodb
Indexes on conversations:

1. customer._id

2. endTime

3. agentParticipants.agentParticipant_id

4. agentParticipants.username

5. conversationDirection

6. creationTime

Indexes on CustomerTopicEvents

1. customerId

2. cimEvent.name

3. cimEvent.type

4. cimEvent.channelSession.customer._id

5. cimEvent.channelSession.customerSuggestions._id

6. cimEvent.channelSession.roomInfo.mode

7. timestamp

8. topicId

9. recordCreationTime

Pre-requisites

1. Mongo database with data available in 4.4.10 format within conversations-manager_db, adminPanel and routing-engine_db.

2. TLS certificates (mongo-ca-cert, client-pem) of the mongodb, if required.

Scope of Migration

• The source databases upon which this activity is to be perform are conversation-manager_db, adminPanel, and routing-engine_db.

• The configuration file is present is transflux/config/data_migration_config.yaml

Configurations

By default, the enabled flag for the below mentioned necessary migration pipelines is set to true

• connection_string: Refers to the MongoDB credentials.

• Within the batch_processing/conversation-manager-4.10, are the configuration for conversations migration, running in batches.

  • js_file: name of the JS file running in the backend (defined in codebase)

  • start_date: Start date for the data migration.

  • end_date: End date for the data migration.

  • interval: Minute-wise interval that filters the data to be processed within a specific timeline

  • enabled : Flag enabling (displaying) the migration pipeline onto Frontend UI

• Within the non_batch_processing/routing-engine-4.10, are the configurations for routing-engine_db and adminPanel data migration

• tls: TLS flag that determines if the mongo database supports only TLS verified connection

  • tls_ca_file: path for mongo-ca-cert file

  • tls_cert_key_file path for client-pem file

When changes are made, it is essential to delete the existing ConfigMap ef-transflux-config-cm and recreate it using the following command:
kubectl -n expertflow create configmap ef-transflux-config-cm --from-file=config

To follow along the demonstration of data migration:

1. (4.10)Migration Activity from Data Platform UI (4.4.10 to 4.10)