Migration Activity ( CX-4.7 to CX-4.8 )

How to Proceed

1. From the data_migration_config.yaml, set the following

   1. mongodb credentials (host, port, username, password, tls)

   2. start_date and end_date as per the duration you want to run migration (details mentioned in Configurations)

   3. For this demonstration, we have set the start_date and end_date to be “2025-01-08” to “2025-03-09”, and the interval to 360 (6 hours)

It is noted that whenever the start_date and end_date are updated in the yaml file, the config map ef-transflux-config-cm is re-created and the activity will resume from here

2. On airflow UI, within the Conversation_Migration -> Details, you should see the start and end date as set within the config from Figure 1, in the bottom. Make sure the pipeline is paused initially

3. If there are existing pipeline runs available (Green bars, to the right of figure 1.) and this is your first time running the pipeline after adjusting the configuration, then

   1. Click on Schedule (Refer to Figure 1), you will see all the instances ran, select all of them and delete from actions button (Refer to Figure 2.)

4. Go back to the Conversation_Migration pipeline, now you will see there will be no runs available (green bars) and the next RUN ID (next to the schedule in figure 1.) is set to the start_date as configured in the yaml file (Refresh the tab till the changes reflects)

5. Switch to Graph, next to Details tab (Refer to figure 1.) to see the graphical view, this will show you the interval wise task as set within the config file

6. Resume the pipeline and refresh the window,/tab you will see the tasks running in the graphical view section. Make sure to turn on the auto-refresh to keep seeing the most recent state.

7. As the pipeline runs, there are few things that need to be observed (Refer to Figure 3.)

   1. pipeline instance: To check whether the task instance of the specific date has run successfully (Left green bars), the square boxes highlights the task of that interval and vertical bar highlights the overall state of that pipeline instance of that date

   2. Run Date: The date on which pipeline instance has been launched and completed.

Rollback Plan

1. If for some reason, the data didn’t come for a specific interval, (lets say from 06:00:00 to 12:00:00 of 2025-01-09). and we want to execute that interval again,

   1. Click on the small green square (Refer to figure 3.) of that interval and select “Clear task”. This will open a new dialogue, select the “Recursive” option and then Clear (Refer to figure 4). This will clear up that task at that interval and queue it up for re-run
      

      

2. If we want to re-run for the entire day (lets say for 2025-01-08),

   1. Click on the vertical bar at that time, then Clear → Clear existing tasks, this will re-run the entire instance on that date (Refer to figure 5.), refresh the tab if necessary.

3. It is recommended to pause the pipeline while clearing states to synchronize the schedule with the latest state.