HA deployment

Solution Prerequisites

Following are the solution setup prerequisites.

For HA deployment, we will be using two VMs, each machine in the cluster should have the following hardware specifications. The twoVMs will be referred by VM1 and VM2 in this guide.


Minimum requirement

CPU

4 cores on each VM

RAM

4 GB on each VM

Disk

300 GB on VM

NICs

1 NIC per VM

Software requirements


Minimum requirement

OS (2)

CentOS 7

MS SQL Server (2)

2016 

Docker CE

18+

Docker compose

1.21

On this page


Installation Steps

Internet should be available on the machine where the application is being installed and connections on port 9242 should be allowed in the network firewall to carry out the installation steps. All the commands start with a # indicating that root user privileges are required to execute these commands. Trailing # is not a part of the command.

Allow ports in the firewall

To start the firewall on CentOS (if it isn't started already), execute the following commands:  

# systemctl enable firewalld
# systemctl start firewalld

To allow the ports on CentOS firewall, you can execute the following commands. You'll have to execute these commands on all the cluster machines. 

# firewall-cmd --add-port=2376/tcp --permanent
# firewall-cmd --add-port=2377/tcp --permanent
# firewall-cmd --add-port=7946/tcp --permanent
# firewall-cmd --add-port=7946/udp --permanent
# firewall-cmd --add-port=4789/udp --permanent
# firewall-cmd --add-port=80/tcp --permanent
# firewall-cmd --add-port=443/tcp --permanent

# firewall-cmd --reload

On VM1 and VM2, execute below additional commands,

# firewall-cmd --add-port=5060/tcp --permanent 
# firewall-cmd --add-port=16386-32768/udp --permanent 
# firewall-cmd --add-port=9092/tcp --permanent 
# firewall-cmd --reload



Configure Log Rotation


Add the following lines in 
/etc/docker/daemon.json file (create the file if not there already) and restart the docker daemon using systemctl restart docker. Perform this step on all the machines in the cluster..

{  
    "log-driver": "json-file"
    "log-opts": {
        "max-size": "50m",
        "max-file": "3"
    
}

Installation Steps

  1. Download the deployment script deployment.sh and place it in the user home or any desired directory. This script will:
    1. delete the recording-solution directory if it exists.
    2. clone the required files for deployment
  2. To execute the script, give it the execute permissions and execute it. 

    $ chmod 755 deployment.sh
    $ ./deployment.sh
  3. Change to newly created directory with name recording-solution. This directory contains all the required files.

  4. Run SQL script in MySQL to create database and tables.  (recording-solution/db_schema.sql).
  5. Update environment variables  in the following files inside /root/recording-solution/docker/environment_variables folder.

    1. general-environment.env

      Name

      Description

      DB_URL

      VRS database connection URL

      jdbc:jtds:sqlserver://192.168.1.92:1433/vrs;user=sa;password=Expertflow464

      DB_DRIVER

      JDBC driver

      net.sourceforge.jtds.jdbcx.JtdsDataSource


      CC_TYPECisco Contact center type (UCCX or UCCE)
      TZTimezone (Asia/Karachi)
      ENCRYPTION_ENABLED

      To enable/disable recorded file encryption

      true = enabled

      false = disabled

      AMQ_PRIMARYPrimary ActiveMQ URL, (VRS machine IP)
      tcp://192.168.1.242:61616
      AMQ_SECONDARYSecondary ActiveMQ URL, Keep it same as primary if ActiveMQ not available in HA
      AMQ_TIMEOUT3000, keep it same
      AMQ_RANDOMIZEfalse, keep it same
      AMQ_PRIORITY_BACKUPtrue, keep it same
      LOCAL_MACHINE_IPVRS machine IP
      CUCM IPCisco Call Manager IP
      CUCM_APPLICATION_USER_NAMECUCM Application user's username created in step 6.
      CUCM_APPLICATION_USER_PASSWORDCUCM Application user's password created in step 6.
      TIME_CUSHIONThe number of seconds to add to the start and end time of call when calling API from CIM. There are few seconds difference between CIM interaction's start and end time and recording solution start and end time since CIM fetch interactions from Finesse while recording solution gets time from CUCM 
      MAX_RING_TIMEMaximum Call ring time on agent desktop, default is 30 seconds
      CALL_TIMEOUTSocket timeout for recording rtp packets, set it to 10
      THREAD_TIMEInterval in seconds between two jobs that clears completed calls, set it 10
      FILE_EXTENSIONExtension that archival file will look for file to archive. Set it to "wav"
      DIRECTORY_PATH_TO_MONITORThis and following 9 variables are used for the archival process. This variable will hold the path of the recording
      ARCHIVED_MEDIA_FILES_EXTENSIONThe archival process will archive recordings with this extension, set it to "wav"
      NO_OF_DAYSThe number of days to keep recordings in the primary server. Recordings older than this value days will be archived
      SFTP_HOSTSFTP hostname or IP
      SFTP_PORTSFTP port
      SFTP_USERNAMESFTP username
      SFTP_PASSWORDSFTP password
      ARCHIVAL_JOB_INTERVALArchival process will run every this values seconds and archive any pending archival recordings
      RETRY_LIMITNumber of retries on pending archival recording folders
      ARCHIVE_PATHThe shared path on the archival server where archival process archive recordings
      ARCHIVE_PATH_USERArchive path's machine user
      ARCHIVE_PATH_PASSArchive path's machine password
      ARCHIVAL_PROCESS_NODE

      This variable value should be "active" on once machine and "passive" on second machine in HA.

      "active" machine archival process will sent files to SFTP server and then delete. "passive" machine 

      process will only delete local file.

      UCCE_DB_URL

      UCCE awdb database connection URL, used for UCCE deployment only.

      jdbc:jtds:sqlserver://192.168.1.87:1433/ucce_awdb;user=sa;password=Expertflow464
      Used for UCCX deployment only...
      UCCX_URLUCCX URL, used for fetching agent details, 
      https://192.168.1.101
      UCCX_USERNAMEUCCX user,  should have privileges to fetch agents
      UCCX_PASSWORDUCCX user password
      ui-environment.env
      DB_IPVRS database machine IP
      DB_USERVRS database user
      DB_PASSWORDDatabase password
      DB_NAMEDatabase name


  6. Having environment configurations done, copy the recording-solution directory on VM2 in/root directory using the following command.

    # scp -r /root/recording-solution root@<vm-ip>:/root/
  7. Execute the following commands inside /root/recording-solution directory. 

    # chmod 755 install.sh
    # ./install.sh
  8. Run the following command to ensure that all the components are up and running. 

    # docker ps

    This will show services status as shown below image 

  9.  Now go to VM2, update LOCAL_MACHINE_IP  variable to VM2 IP in root/recording/solution/docker/environment variables/recorder-environment.env file  and run below command inside /root/recording-solution to start recorder and activemq services. The two activemq services on VM1 and VM2 will now act as master/slave to provide HA. The two recorder services on VM1 and VM2 will be configured in Cisco Call Manager (CUCM) to provide HA.  

    # chmod 755 install.sh
    # ./install.sh
  10. The directory "/root/recording-solution/recordings/wav"  should also be mounted on network shared file system on both VMs or they should be synchronized with each other . In this way, all services on two VMs will have a shared directory for recording files reading or writing. Follow next step if network shared and synchronized folder is not provided
  11. Recording folder synchronization, follow below steps;
    1. Install lyncd utility on one machine, run below commands. 

      root@host # yum -y install epel-release
      root@host # yum -y install lsyncd
    2. Generate SSH Keys on same. Run below command to generate a key. Use default by pressing enter every time it prompts 

      root@host # ssh-keygen -t rsa
    3. Transfer the SSH key to the other other machine by running below commands, enter other machine root password when prompted 

      ssh-copy-id root@other-machine-ip
    4. vi ~/.ssh/config

      enter below text in config file, replace the Hostname with other machine IP

      Host dest_host
       Hostname 172.16.144.32
       User root
       IdentityFile ~/.ssh/id_rsa
    5. settings {
              logfile = "/var/log/lsyncd/lsyncd.log",
              statusFile = "/var/log/lsyncd/lsyncd-status.log",
              statusInterval = 1
      }
      
      sync {
              default.rsync,
              source="/root/recording-solution/recordings",
              target="192.168.1.125:/root/recording-solution/recordings",
              delete = false,
              rsync={
                     compress = true,
      acls = true,
      verbose = true,
      owner = true,
      group = true,
      perms = true,
      rsh = "/usr/bin/ssh -p 22 -o StrictHostKeyChecking=no"}
      }
    6. Follow above steps for the other machine

  12. Repeat the following steps on both machines.

    1. Download keepalived.sh script and place it in any directory.
    2. Give execute permission and execute the script. This will create a keep-alived directory.

      # chmod +x keepalived.sh
      # ./keepalived.sh

    3. Configure keep.env file inside keep-alived directory

      Name

      Description

      Name

      Description

      KEEPALIVED_UNICAST_PEERS

      IPs of the machines in the cluster. On each machine, this variable should have a list of IPs of all the other machines in the cluster. The format of the list is as below: 

      192.168.1.80

      KEEPALIVED_VIRTUAL_IPSVirtual IP of the cluster. It should be available in the LAN. For example: 192.168.1.245
      KEEPALIVED_PRIORITYPriority of the node. Instance with lower number will have a higher priority. It can take any value from 1-255. 
      KEEPALIVED_INTERFACEName of the network interface with which your machine is connected to the network. On CentOS, ifconfig or ip addr sh will show all the network interfaces and assigned addresses. 
      CLEARANCE_TIMEOUTCorresponds to the initial startup time of the application in seconds which is being monitored by keepalived. A nominal value of 60-120 is good enough
      KEEPALIVED_ROUTER_IDDo not change this value.
      SCRIPT_VAR

      This script is continuously polled after 2 seconds. Keepalived relinquishes control if this shell script returns a non-zero response. It could be either umm or ECM backend API.

      pidof dockerd && wget -O index.html https://localhost:443/

    4. Give the execute permission and execute the script: 

      # chmod +x keep-command.sh
      # ./keep-command.sh

Troubleshooting