Rsync configuration steps for unidirectional syncing

This document contains the steps to set up and configure Rsync for unidirectional directory synchronization from both Recorder servers to the Replay server.

Prerequisites

Root or sudo privileges on all VMs.
Make sure that the mounting point of rsync is /var/vrs/recordings on all three VMs; two VRS, and a Replay Server.
Grant full permission to this /var/vrs/recordings directory and its subdirectories created within it.

Steps to Set Up Synchronization

Step 1: Install the Required Package

This step needs to be done on all Servers ( Both the recorder servers and the Replay Server)

Update the package list:
CODE
```
sudo apt update
```
Install rsync:
CODE
```
sudo apt install -y rsync
```

Step 2: Configure SSH Key-Based Authentication

This step needs to be done on both the recorder servers one by one.

Generate an SSH key pair by running the following command
CODE
```
ssh-keygen -t rsa
```

Press Enter to accept the default file location (~/.ssh/id_rsa).

Again press Enter to provide the empty passphrase for automated access, and press ENTER again to confirm.

Copy the public key to the replay server. Run the following command by replacing <replay-server-ip> with a real IP address of the replay server:
CODE
```
ssh-copy-id root@<replay-server-ip>
```

Type yes and press ENTER to continue connecting, enter the password for the root user when prompted, and press ENTER key to copy it at the default location on the Replay server.

Now test passwordless SSH access to ensure SSH Key-Based Authentication. Run the following command.
CODE
```
ssh root@<replay-server-ip>
```

If the Replay server is accessed via SSH without a password, then this configuration will be successful.

Step 3: Create a script for the Rsync operation

This step needs to be done on both the recorder servers one by one.

Make a directory for the script.
CODE
```
mkdir /root/vrs_scripts
```
Create the script
CODE
```
nano /root/vrs_scripts/vrs_rsync.sh
```

Now paste the code below into it and save it.

Ensure that the real target VM IP is provided in line 4.
The SOURCE_DIR and the DEST_DIR are set to /var/vrs/recordings/.. if you are using other directories for vrs recordings set those in line 5 and 6.
No need to create the LOG_FILE, it will be created automatically.

CODE

#!/bin/bash

LOG_FILE="/var/log/vrs_rsync.log"
TARGET_VM="192.168.1.200"
SOURCE_DIR="/var/vrs/recordings/cucmRecording/sessions/"
DEST_DIR="root@$TARGET_VM:/var/vrs/recordings/cucmRecording/sessions/"

echo "----- $(date '+%Y-%m-%d %H:%M:%S') Checking target VM status -----" >> "$LOG_FILE"

# Check if rsync is already running
if pgrep -x rsync > /dev/null; then
    LOG_MESSAGE="$(date '+%Y-%m-%d %H:%M:%S') ERROR: An rsync process is already running. Aborting new instance!"
    echo "$LOG_MESSAGE" | tee -a "$LOG_FILE" 
    exit 1
fi

# Check if the target VM is reachable before rsync
if ping -c 3 "$TARGET_VM" > /dev/null 2>&1; then 
    echo "$(date '+%Y-%m-%d %H:%M:%S') Target VM is UP. Starting rsync..." >> "$LOG_FILE"

    # Run rsync and capture its exit status with timestamps in the error log
    rsync -avp --info=progress2 "$SOURCE_DIR" "$DEST_DIR" 2>> >(while read line; do echo "$(date '+%Y-%m-%d %H:%M:%S') $line"; done >> "$LOG_FILE") | 
    awk '/^sending / {next} /to-chk/ {print strftime("%Y-%m-%d %H:%M:%S"), "Transferring:", prev} {prev=$0}' >> "$LOG_FILE"
    
    RSYNC_EXIT_CODE=$?

    if [ $RSYNC_EXIT_CODE -ne 0 ]; then 
        echo "$(date '+%Y-%m-%d %H:%M:%S') ERROR: rsync failed or was interrupted! Exit code: $RSYNC_EXIT_CODE" >> "$LOG_FILE"

        # Check if VM is still reachable after failure
        if ! ping -c 3 "$TARGET_VM" > /dev/null 2>&1; then
            echo "$(date '+%Y-%m-%d %H:%M:%S') ERROR: Target VM went DOWN during rsync!" >> "$LOG_FILE"
        fi
    else
        if ! ping -c 3 "$TARGET_VM" > /dev/null 2>&1; then
            echo "$(date '+%Y-%m-%d %H:%M:%S') ERROR: Target VM went DOWN during rsync!" >> "$LOG_FILE"
        else
            echo "$(date '+%Y-%m-%d %H:%M:%S') rsync completed successfully!" >> "$LOG_FILE"
        fi    
    fi
else 
    echo "$(date '+%Y-%m-%d %H:%M:%S') ERROR: Target VM is DOWN. rsync aborted!" >> "$LOG_FILE"
fi

Give this script executive permissions

CODE

chmod +x /root/vrs_scripts/vrs_rsync.sh

Step 4: Add the cronjobs

The log files that are referred to below, will be created by themselves.

Before adding cronjobs, we need to make a directory to save the files temporarily. Run the command
CODE
```
sudo mkdir /var/vrs/recordings/cucmRecording/old_sessions/
```
This directory is created assuming that the vrs recordings directory is /var/vrs/recordings/. If you are using any other directory make this old_sessions/ directory there.
Now to add cronjobs run the following command
CODE
```
crontab -e 
```

Now paste these cronjobs at the end.

CODE

*/5 * * * * /root/vrs_scripts/vrs_rsync.sh

0 0 * * * find /var/vrs/recordings/cucmRecording/sessions/ -type f -mmin +2880 -exec mv {} /var/vrs/recordings/cucmRecording/old_sessions/ \; -exec echo "$(date '+\%Y-\%m-\%d \%H:\%M:\%S') Moved: {}" >> /var/log/vrs_moved_files.log \;

53 0 * * *  find /var/vrs/recordings/cucmRecording/streams/ -type f -mmin +2880 -exec rm -f {} \; -exec echo "$(date '+\%Y-\%m-\%d \%H:\%M:\%S') Deleted: {}" >> /var/log/vrs_streams_dlt.log \;

Now add this cronjob as well after updating <replay-server-ip> with the replay server IP

CODE

33 0 * * * { echo "----- $(/usr/bin/date '+\%Y-\%m-\%d \%H:\%M:\%S') House Keeping and final rsync on files -----"; rsync -avz --remove-source-files /var/vrs/recordings/cucmRecording/old_sessions/ root@<replay-server-ip>:/var/vrs/recordings/cucmRecording/sessions/; } >> /var/log/vrs_removed_files.log 2>&1

There are 4 cronjobs, and their purposes are

The first cron job executes the vrs_rsync.sh script. This script first checks whether the target VM or replay server is pingable. If it is pingable, the script starts the rsync process to transfer the necessary files to the target. This cron job runs every 5 minutes; you can increase or decrease this interval by replacing */5 * * * * with */10 * * * * or */3 * * * * respectively.
The second cron job runs every day at 00:00. It first finds the files in the directory /var/vrs/recordings/cucmRecordings/sessions/ that are older than 2880 minutes, i.e., 2 days, and then moves these files to /var/vrs/recordings/cucmRecordings/old_sessions. You can change the time of this cron job from 00:00 to 12:00 or 18:30 by replacing "0 0 * * *" with "0 12 * * *" or "30 18 * * *” respectively. Also, you can configure this cron job to find and move the files that are older than 3 days by changing “-mmin +2880” to “-mmin +4320” or files that are older than 1 day by making it “-mmin +1440”. By setting this we can configure how old session files we want to remove.
The third cronjob runs every day at 00:53. It first finds the files in the directory /var/vrs/recordings/cucmRecordings/streams/ that are older than 2880 minutes, i.e., 2 days, and then deletes these files. You can change the time of this cron job from 00:53 to 01:53 or 18:30 by replacing "0 53 * * *" with "53 1 * * *" or "30 18 * * *” respectively. Also, you can configure this cron job to find and delete the files that are older than 3 days by changing “-mmin +2880” to “-mmin +4320” or files that are older than 1 day by making it “-mmin +1440”. By setting this we can configure how old stream files we want to delete.
The last cron job runs every day at 00:33. It starts the rsync process on the files in directory /var/vrs/recordings/cucmRecordings/old_sessions/ and transfers these to the target VM. It will remove the files from old_sessions/ directory after transferring these. You can change the time of this cron job from 00:33 to 12:33 or 18:33 by replacing "0 0 * * *" with "33 12 * * *" or "33 18 * * *” respectively.