Upgrade Guide from Legacy Architecture to New Architecture

Upgrade from 3.10.2 & later to 3.19.0

Upgrade Procedure

To upgrade follow the steps below:

1. Take the backup of MongoDB and MySql database data. For backup follow this guide HC Backup and Restore.
2. Stop HC completely.

   1. For HC 3.10.2 & HC 3.11, stop NGINX service & disable auto-start of NGINX service using the following:

      BASH

      # Stop the service
      sudo systemctl stop nginx.service
       
      # Disable auto-restart of NGINX service upon host reboot
      sudo systemctl disable nginx.service

   2. For HC 3.13.x & later, remove efutils symbolic link

      Remove Symbolic Link

      BASH

      # the following command can be used to remove efutils symbolic link, make sure your have remove privilidges
      # press y when this command prompts for removing the symlink, press enter to confirm
      [root@hc-host ~] rm -i $(which efutils)
      rm: remove symbolic link '/usr/bin/efutils'? y

3. Upgrade the reporting database by executing the following scripts:
   

   

   Executing these scripts using SQL CMD is recommended because it ensures only one connection is made to the database for query execution. However, SSMS can be used to execute these scripts but sometimes it can lead to an issue when a database remains stuck in the single-user mode after a script is executed. If you face this issue, execute this script to set the database to multi-user mode. After the database is set to multi-user mode, execute the script again which originally caused this issue.

   
   

   	Source Release	Target Release	Scripts	Notes
   1	3.10.2	3.18.0	upgrade-from-3.10.2-to-3.11.sql update-from-3.11-to-3.12.1.sql upgrade-from-3.12.1-to-3.13.0.sql upgrade-from-3.13.x-to-3.14.0.sql remove-uq-botname.sql upgrade-from-3.16.x-to-3.18.0.sql	Make sure that the Reporting Connector is stopped before executing these SQL scripts Make sure that the HC database adheres to SQL database naming rules. If it doesn't, rename the database before scripts execution. Execute the scripts in the same order as listed in the Scripts column Update database name in the upgrade-from-3.10.2-to-3.11.sql script before execution Update HC FQDN in the upgrade-from-3.13.x-to-3.14.0.sql script before execution
   2	3.11	3.18.0	update-from-3.11-to-3.12.1.sql upgrade-from-3.12.1-to-3.13.0.sql upgrade-from-3.13.x-to-3.14.0.sql remove-uq-botname.sql upgrade-from-3.16.x-to-3.18.0.sql	Make sure that the Reporting Connector is stopped before executing these SQL scripts Make sure that the HC database adheres to SQL database naming rules. If it doesn't, rename the database before scripts execution. Execute the scripts in the same order as listed in the Scripts column Update HC FQDN in the upgrade-from-3.13.x-to-3.14.0.sql script before execution
   3	3.12.1	3.18.0	upgrade-from-3.12.1-to-3.13.0.sql upgrade-from-3.13.x-to-3.14.0.sql remove-uq-botname.sql upgrade-from-3.16.x-to-3.18.0.sql	Make sure that the Reporting Connector is stopped before executing these SQL scripts Make sure that the HC database adheres to SQL database naming rules. If it doesn't, rename the database before scripts execution. Execute the scripts in the same order as listed in the Scripts column Update HC FQDN in the upgrade-from-3.13.x-to-3.14.0.sql script before execution
   4	3.13.0, 3.13.1	3.18.0	upgrade-from-3.13.x-to-3.14.0.sql remove-uq-botname.sql upgrade-from-3.16.x-to-3.18.0.sql	Make sure that the Reporting Connector is stopped before executing these SQL scripts Make sure that the HC database adheres to SQL database naming rules. If it doesn't, rename the database before scripts execution. Execute the scripts in the same order as listed in the Scripts column Update HC FQDN in the upgrade-from-3.13.x-to-3.14.0.sql script before execution
   5	3.14.0	3.18.0	remove-uq-botname.sql upgrade-from-3.16.x-to-3.18.0.sql	Make sure that the Reporting Connector is stopped before executing these SQL scripts Make sure that the HC database adheres to SQL database naming rules. If it doesn't, rename the database before scripts execution. Execute the scripts in the same order as listed in the Scripts column
   6	3.15.0, 3.15.1	3.18.0	upgrade-from-3.16.x-to-3.18.0.sql	Make sure that the Reporting Connector is stopped before executing these SQL scripts Make sure that the HC database adheres to SQL database naming rules. If it doesn't, rename the database before scripts execution. Execute the scripts in the same order as listed in the Scripts column
   7	3.16.0	3.18.0	upgrade-from-3.16.x-to-3.18.0.sql	Make sure that the Reporting Connector is stopped before executing these SQL scripts Make sure that the HC database adheres to SQL database naming rules. If it doesn't, rename the database before scripts execution. Execute the scripts in the same order as listed in the Scripts column

   The reporting database migration is complete. For the updated schema of this database, refer to the Hybrid Chat Reporting Database Schema. For updated reports, refer to the Reports.

4. Deploy HC 3.19.0 in a different directory using the Docker-based deployment guide. Use the same reporting database server and name that was being used with the older version of HC when prompted by the install script.

5. Stop HC 3.19.0 after deployment using the efutils utility. This utility is available from any location in the host OS after HC 3.19.0 deployment. Turn down HC 3.19.0 using

   Turn Down HC 3.18.0

   BASH

   # try efutils help to get familiar with efutils e.g.
   # efutils help
   # the following command is used to turn the HC 3.19.0 down
   $ efutils all down

6. Migrate the environment variables using the following procedure:

   1. Take a backup of the HC 3.19.0 Internal environment variables

      Take Backup of HC 3.18.0 Env Variables

      BASH

      $ mv <hc-3.19.0-install-dir>/current/docker/environment-variables <hc-3.19.0-install-dir>/current/docker/environment-variables.backup

   2. Take a backup of the HC 3.19.0 External environment variables

      Take Backup of HC 3.18.0 Env Variables

      BASH

      $ mv <hc-3.19.0-install-dir>/external/docker/environment-variables <hc-3.19.0-install-dir>/external/docker/environment-variables.backup

   3. Create an environment variables directory for internal updated environment variable files in HC 3.19.0 installation

      Make directory for environment variables

      BASH

      $ mkdir -p <hc-3.19.0-install-dir>/current/docker/environment-variables

   4. Create an environment variables directory for external updated environment variable files in HC 3.19.0 installation

      Make directory for environment variables

      BASH

      $ mkdir -p <hc-3.19.0-install-dir>/external/docker/environment-variables

   5. The following script will merge the older HC and HC 3.19.0 environment variables in the following way:
      
      1. It will read environment variable files one by one from older HC deployment.
      2. Will add these environment variables in a new file with the same name in <hc-3.19.0-install-dir>/current/docker/environment-variables directory.
      3. Will add newly added environment variables from HC 3.19.0 environment variables backup directory and add them in the file created in step ii.

      4. It yields a new file that contains environment variables from older HC with their values and added environment variables with the default values from HC 3.19.0 release
         The usage details are added below

         Merge Environment Variables

         BASH

         ################################################################################
         # Usage Guide:                                                                 #
         # 1. Copy this script and either edit it using any text editor or create       #
         #    an sh file & paste the script in it                                       #
         # 2. Edit the value of hc_older_baseDir variable in this script, enter         #
         #    absolute path to older HC deployment directory, do not put a forward      #
         #    slash at the end of this value.                                           #
         # 3. Edit the value of hc_3_18_0_baseDir variable in this script, enter        #
         #    absolute path to HC 3.157.0 deployment directory, do not put a forward     #
         #    slash at the end of this value.                                           #
         # 4. Copy this script, paste in your shell and execute. If you have created an #
         #    sh file in step # 1, execute that file.                                   #
         ################################################################################
         
         ################################################################################
         # The following script makes an assumption that backup of HC 3.19.0 env vars.  #
         # is located at <hc3.19.0-base-dir>/docker/environment-variables.backup, if    #
         # this is not the case with your deployment, update the following path in this #
         # script:                                                                      #
         # $hc_3_19_0_baseDir/docker/environment-variables.backup/$filename             #
         ################################################################################
         
         
         
         # Change these variables with the solution paths for source and target releases.
         
         -----------------------COPY FROM HERE ----------------------------------------
         
         
         src_release="/root/chat-solution"
         target_release="/var/expertflow"
         
         
         
         
         # Merge the internal Variables
         
         
         
         for file in ${src_release}/docker/environment-variables/*; do
                 [ -e "$file" ] || continue
                 filename=${file##*/}
                 if  [[ ! -e "${target_release}/current/docker/environment-variables.backup/${filename}"   ]]
         		then
         			echo "Skipping ${filename}"
         		else
         			echo "Merging ${filename}"
         	        awk -F= '!a[$1]++' ${src_release}/docker/environment-variables/${filename} ${target_release}/current/docker/environment-variables.backup/${filename} > ${target_release}/current/docker/environment-variables/${filename}
         		fi
         done
         

         To merge external variables copy this script in your CLI.

         CODE

         #Merge the External Variables
         
         
         
         -----------------------COPY  FROM HERE ----------------------------------------
         src_release="/root/chat-solution"
         target_release="/var/expertflow"
         
         
         for file in $src_release/docker/environment-variables/*; do
                 [ -e "$file" ] || continue
                 filename=${file##*/}
                 if  [[ ! -e "${target_release}/external/docker/environment-variables.backup/${filename}"   ]]
         		then
         			echo "Skipping ${filename}"
         		else
         			echo "Merging ${filename}"
         	        awk -F= '!a[$1]++' ${src_release}/docker/environment-variables/$filename ${target_release}/external/docker/environment-variables.backup/${filename} > ${target_release}/external/docker/environment-variables/${filename}
         		fi
         done
         
         -------------------------------------------------------------------------------

         

         The script above merges files but cannot assure environment variables sequence/order in a file. It can lead to leaving some comments in environment variable files out of sequence. So it is recommended to refer to the documentation of environment variable files in  <hc-3.19.0-install-dir>/current/docker/environment-variables.backup a directory to review environment variables comments/explanations. The recommended way is to refer to Environment Configurations for env variables details.

7. For merging translation files, follow these steps: 

   CODE

   mv /var/expertflow/current/scripts/merg.py /var/expertflow/current/docker/

8. take a backup of default translation files in HC 3.19.0

   CODE

   $ mv <hc-3.19.0-install-dir>/current/docker/translations <hc-3.19.0-install-dir>/current/docker/translations-backup

    copy translations files from older hc to hc 3.19.0

   CODE

   $ cp -r <older-hc-install-dir>/docker/translations <hc-3.19.0-install-dir>/current/docker/translations

9. For Agent-Gadget Translations Files:

   CODE

   #navigate to hc 3.19.0 docker directory
   $ cd <hc-3.19.0-install-dir>/current/docker/
   

   

   Install python-3 on your machines if it is not already installed, before executing the below steps:

   $ yum install python3 -y

   CODE

   $ python3 merg.py <hc-3.19.0-install-dir>/current/docker/translations/agent-gadget/en.json <hc-3.19.0-install-dir>/current/docker/translations-backup/agent-gadget/en.json <hc-3.19.0-install-dir>/current/docker/translations/agent-gadget/local.en.json
   $ python3 merg.py <hc-3.19.0-install-dir>/current/docker/translations/agent-gadget/fr.json <hc-3.19.0-install-dir>/current/docker/translations-backup/agent-gadget/fr.json <hc-3.19.0-install-dir>/current/docker/translations/agent-gadget/local.fr.json
   $ python3 merg.py <hc-3.19.0-install-dir>/current/docker/translations/agent-gadget/ar.json <hc-3.19.0-install-dir>/current/docker/translations-backup/agent-gadget/ar.json <hc-3.19.0-install-dir>/current/docker/translations/agent-gadget/local.ar.json
     

10. rename the local merged files to their original names

    CODE

    $ mv <hc-3.19.0-install-dir>/current/docker/translations/agent-gadget/local.en.json <hc-3.19.0-install-dir>/current/docker/translations/agent-gadget/en.json
    $ mv <hc-3.19.0-install-dir>/current/docker/translations/agent-gadget/local.fr.json <hc-3.19.0-install-dir>/current/docker/translations/agent-gadget/fr.json
    $ mv <hc-3.19.0-install-dir>/current/docker/translations/agent-gadget/local.ar.json <hc-3.19.0-install-dir>/current/docker/translations/agent-gadget/ar.json

    CODE

    # Do you want to overwrite the existing translations file? y

11. For Customer Gadget Translation Files:

    CODE

    $ python3 merg.py <hc-3.19.0-install-dir>/current/docker/translations/customer-gadget/en.json <hc-3.19.0-install-dir>/current/docker/translations-backup/customer-gadget/en.json    <hc-3.19.0-install-dir>/current/docker/translations/customer-gadget/local.en.json
    $ python3 merg.py <hc-3.19.0-install-dir>/current/docker/translations/customer-gadget/fr.json <hc-3.19.0-install-dir>/current/docker/translations-backup/customer-gadget/fr.json    <hc-3.19.0-install-dir>/current/docker/translations/customer-gadget/local.fr.json
    $ python3 merg.py <hc-3.19.0-install-dir>/current/docker/translations/customer-gadget/ar.json <hc-3.19.0-install-dir>/current/docker/translations-backup/customer-gadget/ar.json    <hc-3.19.0-install-dir>/current/docker/translations/customer-gadget/local.ar.json

    restore the original file names

    CODE

    $ mv <hc-3.19.0-install-dir>/current/docker/translations/customer-gadget/local.en.json <hc-3.19.0-install-dir>/current/docker/translations/customer-gadget/en.json
    $ mv <hc-3.19.0-install-dir>/current/docker/translations/customer-gadget/local.fr.json <hc-3.19.0-install-dir>/current/docker/translations/customer-gadget/fr.json
    $ mv <hc-3.19.0-install-dir>/current/docker/translations/customer-gadget/local.ar.json <hc-3.19.0-install-dir>/current/docker/translations/customer-gadget/ar.json

    CODE

    # Do you want to overwrite the existing translations file? y

12. For Knowledgebase UI translation files, there is no change since HC 3.10.2. So the older translation files can be used to update kb-ui translations which are already copied to the HC 3.19.0 release in the above-given commands.

13. Migrate key stores and trust stores if you have added custom client certificates in your HC deployment

    CODE

    # take backup of current certificates
    $ mv <hc-3.19.0-install-dir>/current/docker/certificates <hc-3.19.0-install-dir>/current/docker/certificates.backup
    
    # copy certificates from older HC deployment
    $ \cp -rp <older-hc-install-dir>/docker/certificates <hc-3.19.0-install-dir>/current/docker/certificates

    

    `\` in the above cp command is necessary to override the aliasing in the bash shell.

14. Migrate your HC public certificate by replacing the localhost.crt and localhost.key files in <hc-3.19.0-install-dir>/docker/nginx/certs 
15. Deploy Customer Web Widget from the master branch. Update Queues & customer channel data configurations in the new Web Widget deployment as per your previous configuration. Update config.json file accordingly.
16. If upgrading from HC 3.10.2, HC 3.11 or HC 3.12.1, remove LD_LIBRARY_PATH environment variable from <hc-3.19.0-install-dir>/current/docker/environment-variables/umm-variables.env file.
17. Migrate Minio data as follows:

    1. Copy Minio mount data from older HC to 3.19.0 release

       Copy Minio Mount Point to HC 3.18.0

       BASH

       \cp -rp <older-hc-install-dir>/docker/data/minio/data/default  $(docker volume  inspect --format "{{.Mountpoint}}/" expertflow_minio)

       

       for large data set size of minio container, this command may take longer than expected to copy the data. Please don't interrupt the operation.

18. Copy the Mongo backup to the xdata volume 

    1. Navigate to /root/db_backup

       CODE

       $ cp -rp mongo-backup $(docker volume  inspect --format "{{.Mountpoint}}/" expertflow_xdata)

    2. bring the mongo service profile up by executing

       CODE

       $ efutils profile external-mongo up

    3. login to the mongo service container using

       CODE

       $ efutils login mongo

    4. import the dump using ( singleton )

       CODE

       $ mongorestore --gzip --verbose --drop /xdata/mongo-backup

    5. import the dump using(For HA)

       CODE

       $ mongorestore -u "root" -p "expertflow123" --gzip --verbose --drop /xdata/mongo-backup

19. For mongo restore from 3.10.x or earlier, please use this procedure. ( make sure the mongo service profile is down during these steps) 

    1. take a backup of existing data from the current release using 

       CODE

       $ \cp -rp $(docker volume  inspect --format "{{.Mountpoint}}/data/db" expertflow_mongo) $(docker volume  inspect --format "{{.Mountpoint}}/data/db-orig" expertflow_mongo)

    2. remove the original contents using 

       CODE

       $ rm -fr $(docker volume  inspect --format "{{.Mountpoint}}/data/db/*" expertflow_mongo)

    3. copy the data from old deployment directly to the mongo volume 

       CODE

       $ \cp -rp  3.10.x/docker/data/mongo/data/*  $(docker volume  inspect --format "{{.Mountpoint}}/data/db/" expertflow_mongo)

20. Start HC 3.19.0.
21. Restore MySQL backup. For restore follow this guide HC Backup and Restore.

22. Download this script and then copy it  inside mongo container using the following command

    CODE

    docker cp ./mongo-script.js expertflow_mongo_1:/

     Access MongoDB container by the following command

    CODE

    docker exec -it expertflow_mongo_1 sh 

     Execute the following code inside mongo container.

    CODE

    mongo localhost:27017/chatsolution mongo-script.js

23. If Communication Server stays in an unhealthy state after the upgrade, execute the following:
    1. Stop Comm Server using efutils

    2. Execute the following script to clear real-time events for Agents

       CODE

       # For Singleton
       docker exec -it expertflow_mongo_1 mongo
       use chatsolution
       db.agents.deleteMany({});
       db.tasks.deleteMany({});
       exit
       
       # For HA
       docker exec -it expertflow_mongo_1 mongo -u root -p
       use chatsolution
       db.agents.deleteMany({});
       db.tasks.deleteMany({});
       exit

    3. Start Comm Server again

24. If CCM stays in an unhealthy state after the upgrade, change the following environment variables:

    	Env Variable	Old Value	Updated Value	Notes
    1	AMQ_PRIMARY	activemq:61617 or <ip>:61617	activemq:61617?verifyHostName=false or <ip>:61617?verifyHostName=false	Do not change activemq or the IP address is written in this environment variable value, just add ?verifyHostName=false at the end of the value.
    2	AMQ_SECONDARY	activemq:61617 or <ip>:61617	activemq:61617?verifyHostName=false or <ip>:61617?verifyHostName=false	Do not change activemq or the IP address is written in this environment variable value, just add ?verifyHostName=false at the end of the value.

    Perform up operation for CCM service using efutils 

25. MRE queues in HC 3.10.2 are not compatible with HC 3.19.0, so when upgrading from HC 3.10.2 to HC 3.19.0, execute the following steps:
    1. Delete queues from MRE UI
    2. Recreate queues in MRE UI
    3. Restart MRE

If HC 3.10.2 was deployed in high availability mode, perform the upgrade operations for Primary and Secondary host machines. If you are upgrading source release from between HC 3.10.2 and HC 3.15.1 to 3.19.1, turn down the Arbiter HC and redeploy the Arbiter node. No data migration is needed for the Arbiter node. After upgrade, execute the renounce scripts in the secondary host machine.