Kubernetes Backup/Restore using Velero

Requirements

Please go though the Backup and Restore Recommendations before implementing the solution available at link Backup and Restore Strategies and Recommendations for Expertflow CX

Backups using Velero requires at least one of the storage location available where backup can be saved. Below is the list of these options which are needed and must be catered before proceeding for backup solution.

Backup Type	Deployment Requirement	Distributed	DR
S3 Compatible Cloud Storage	any kubernetes deployment with persistent storage enabled helm for kubernetes based deployment alternatively docker-compose based deployments can also be used if kubernetes is not possible at DR site with docker volume used to persist backups. minio can be deployed without kubernetes or docker-compose and requires advanced expertise to mainain it. All minio should be deployed at the DR sites. deploying at the same location is not recommended	Yes	Yes
hostpath	native to kubernetes. Backups will be available on the host file-system customer should take regular backups off this file-system to a safer location	No	No

Deploy Velero

velero is the application used for managing all the backups and should be deployed on the cluster where CIM solution is deployed.

For an HA based deployments, the velero should be deployed on any of the Control-Plane nodes

Download the compatible version of velero

CODE

wget https://github.com/vmware-tanzu/velero/releases/download/v1.9.4/velero-v1.9.4-linux-amd64.tar.gz

untar + uncompress it

CODE

tar zxvf velero-v1.9.4-linux-amd64.tar.gz

and copy the velero binary to /usr/local/bin

CODE

cp velero-v1.9.4-linux-amd64/velero /usr/local/bin/

make it executable by running

CODE

chmod 777 /usr/local/bin/velero

S3 Compatible Storage

If the customer can provide S3, compatible storage, please use this link to configure Velero accordingly. However, if this is not possible, deploy the minio as S3 storage endpoint and direct all backups via velero to the minio storage. Minio can be deployed in different regions to make a complete DR fault tollerant backup/restore solution.

Kubernetes based deployment

If minio is to be deployed on any of kubernetes cluster, please use this helm command to deploy the minio

add the helm report

CODE

helm repo add bitnami https://charts.bitnami.com/bitnami

update the repo

CODE

helm repo update

deploy the default backup location

CODE

helm install minio-default     \
--namespace velero     \
--create-namespace     \
--set resources.requests.memory=1Gi     \
--set persistence.size=32Gi \
--set auth.rootUser=minio \
--set auth.rootPassword=minio123  \
--set service.type=NodePort  \
--set service.nodePorts.api=30100  \
--set service.nodePorts.console=30200  \
--set provisioning.enabled=true \
--set provisioning.buckets[0].name=velero    \
--set provisioning.buckets[0].policy=public \
--set provisioning.buckets[0].purge=true  \
bitnami/minio

If more than 1 locations are needed, you can install multiple instances of minio in different locations (Optional )

CODE

helm install minio-primary     \
--namespace velero     \
--create-namespace     \
--set resources.requests.memory=1Gi     \
--set persistence.size=32Gi \
 --set auth.rootUser=minio \
--set auth.rootPassword=minio123  \
--set service.type=NodePort  \
--set service.nodePorts.api=30101  \
--set service.nodePorts.console=30201  \
--set provisioning.enabled=true   \
--set provisioning.buckets[0].name=velero    \
--set provisioning.buckets[0].policy=public \
--set provisioning.buckets[0].purge=true \
bitnami/minio

CODE

helm install minio-secondary     \
--namespace velero     \
--create-namespace     \
--set resources.requests.memory=1Gi     \
--set persistence.size=32Gi \
 --set auth.rootUser=minio \
--set auth.rootPassword=minio123  \
--set service.type=NodePort  \
--set service.nodePorts.api=30102  \
--set service.nodePorts.console=30202  \
--set provisioning.enabled=true \
--set provisioning.buckets[0].name=velero    \
--set provisioning.buckets[0].policy=public  \
--set provisioning.buckets[0].purge=true  \
bitnami/minio

Host-Path as Backup Destination

In order to use host-Path of the host for backup location. This section expects that you have completed the velero deployment in step 1. Please follow these steps.

for a hostpath based backups, this should be considered only for single-node deployment

Create fake velero entry so that we can add local-path plugin and intialize it

CODE

velero install --use-restic --use-volume-snapshots=false --namespace velero --plugins velero/velero-plugin-for-aws:v1.2.0 --no-default-backup-location --no-secret

Install the velero host-path plugin

CODE

velero plugin add replicated/local-volume-provider:v0.3.3

confirm the host-path plugin is deployed successfully by running

CODE

velero plugin get

Sample output will look like

CODE

# velero plugin get
NAME                                        KIND
velero.io/crd-remap-version                 BackupItemAction
velero.io/pod                               BackupItemAction
velero.io/pv                                BackupItemAction
velero.io/service-account                   BackupItemAction
replicated.com/hostpath                     ObjectStore
replicated.com/nfs                          ObjectStore
replicated.com/pvc                          ObjectStore
velero.io/aws                               ObjectStore
velero.io/add-pv-from-pvc                   RestoreItemAction
velero.io/add-pvc-from-pod                  RestoreItemAction
velero.io/admission-webhook-configuration   RestoreItemAction
velero.io/apiservice                        RestoreItemAction
velero.io/change-pvc-node-selector          RestoreItemAction
velero.io/change-storage-class              RestoreItemAction
velero.io/cluster-role-bindings             RestoreItemAction
velero.io/crd-preserve-fields               RestoreItemAction
velero.io/init-restore-hook                 RestoreItemAction
velero.io/job                               RestoreItemAction
velero.io/pod                               RestoreItemAction
velero.io/restic                            RestoreItemAction
velero.io/role-bindings                     RestoreItemAction
velero.io/service                           RestoreItemAction
velero.io/service-account                   RestoreItemAction
velero.io/aws                               VolumeSnapshotter

Once the plugin is fully deployed ( it may take a while to deploy the plugin ) create the destination directory on the host using.

CODE

mkdir /var/my-host-path-to-snaps

and change its permission to 777 so that anyone can write to it.

CODE

chmod 777 /var/my-host-path-to-snaps

Add the backup location in velero

CODE

velero backup-location create default --default --bucket my-hostpath-snaps --provider replicated.com/hostpath --config path=/var/my-host-path-to-snaps,resticRepoPrefix=/var/velero-local-volume-provider/my-hostpath-snaps/restic

make sure the backup location is added and made available by looking though logs

CODE

kubectl logs deployment/velero -n velero

get the backup locations where backups will be saved

CODE

velero backup-location get

Commission the backup Destination

Create Credentials for Backup

CODE

cat <<EOF > credentials-velero
[default]
aws_access_key_id = minio
aws_secret_access_key = minio123
EOF

run the velero install to configure the backup destination

CODE

velero install \
    --secret-file=./credentials-velero \
    --provider=aws \
    --bucket=velero \
    --backup-location-config region=minio-default,s3ForcePathStyle=true,s3Url=http://minio-default.velero.svc.cluster.local:9000,publicUrl=http://localhost:30100 region=minio-default \
    --plugins=velero/velero-plugin-for-aws:v1.4.0 \
    --use-volume-snapshots=true \
    --use-restic=true \
    --snapshot-location-config region=minio-default \
    --wait

and wait for the confirmation message from above command to finish the deployment on your production cluster.

Add Default backup location for Velero

This the where all your backups will be saved in case you dont provide any specific location

Add other location (Optional )

minio-primary

CODE

velero backup-location create primary \
    --provider aws \
    --bucket velero \
    --config region=minio-primary,s3ForcePathStyle=true,s3Url=http://minio-primary.velero.svc.cluster.local:9000,publicUrl=http://localhost:30201

minio-secondary

CODE

velero backup-location create secondary \
    --provider aws \
    --bucket velero \
    --config region=minio-secondary,s3ForcePathStyle=true,s3Url=http://minio-secondary.velero.svc.cluster.local:9000,publicUrl=http://localhost:30202

List all Backup-Locations

velero get backup-locations

the output will be similar to below given

CODE

# velero get backup-locations
NAME        PROVIDER   BUCKET/PREFIX   PHASE       LAST VALIDATED                  ACCESS MODE   DEFAULT
default     aws        velero          Available   2023-01-03 16:52:49 +0500 PKT   ReadWrite     true
primary     aws        velero          Available   2023-01-03 16:52:41 +0500 PKT   ReadWrite
secondary   aws        velero          Available   2023-01-03 16:53:09 +0500 PKT   ReadWrite

the default location is the one which will be used as target backup-location when none is specified.

Create Backup

You can create backup for

cluster
namespace
deployment
POD
label

Cluster Level Backup

For now we can configure velero to take backup of the ef-external namespace. This will always go to default backup-location

CODE

velero backup create ef-external-backup --include-namespaces=ef-external

if you want to specify a particular location of the backup you can use

CODE

velero backup create expertflow-backup --include-namespaces=expertflow --storage-location primary
OR
velero backup create expertflow-backup --include-namespaces=expertflow --storage-location secondary

Application Level Backup

Mongo

CODE

velero backup create mongo-backup --include-resources pvc,pv --selector release=mongodb

PostgreSQL

CODE

velero backup create mongo-backup --include-resources pvc,pv --selector release=postgresql -n ef-external

Schedule backups

Velero backups can be scheduled using iteration and frequency. like

CODE

velero create schedule system-critical --include-namespaces=expertflow --schedule="@every 24h"

Or

CODE

velero create schedule thrice-daily --schedule="@every 8h"

List all Backups in Velero

CODE

velero get backups

list all schedules

CODE

velero get schedules

Delete a specific scheduled backup

CODE

velero delete schedule velero-default

Delete backup

CODE

velero delete backups <BACKUP-NAME>

Status of backup

CODE

velero backup describe ex-backup

Restore

To restore from a particular backup

CODE

velero restore create --from-backup <BACKUP-NAME>

this will restore the components to same state when backup was taken.