RKE2 Deployment in High Availability With Kube-VIP

Purpose

The purpose of this document is to describe steps to deploy the RKE2 Kubernetes Distribution in High Availability with Kube-VIP.

Pre-requisites 

The prerequisites and cluster topologies are describe in the RKE2 High Availability Pre-Deployment & Installation Guide. Please review the document before proceeding with installation in High Availability mode.

Preparing for Deployment

Decide what level of availability is needed and prepare your nodes according to the structure for the solution as given below.

CIM Prerequisites 

Kube-VIP Requirements

A VIP is a virtual IP Address that remains available  and traverses between all the Control-Plane nodes seamlessly with 1 Control-Plane node active to Kube-VIP. Kube-VIP works exactly as keepalive except that it has some additional flexibilities to configure depending upon the environment for example Kube-VIP can work using 

• ARP – When using ARP or Layer 2 it will use leader election.
• BGP – BGP is a mechanism so that networks that rely on routing (layer 3) can ensure that new addresses are advertised to the routing infrastructure 
• Routing Tables –  The Routing Table mode is to allow additional routing technologies such as ECMP etc. The table mode enables kube-vip to manage the addition/deletion of addresses to the routing tables of these nodes so that they can recieve the correct traffic.

For RKE2 cluster setup, we will be using ARP mode, which requires

• the VIP is transferable between any of the Control-Plane Nodes i-e same subnet as of the control-Plane node. ARP will typically broadcast updates to the entire network that update the IP to Hardware (MAC) mapping this ensures traffic is sent to the correct physical or virtual network nic.
• ARP traffic is open in the network

Kube-VIP deployment is dependent on 1 working RKE2 Control Plane node before we can deploy other node ( both CP and Workers ) . 

RKE2 Cluster WorkLoad Topologies

A production cluster can be run with a mixture of workload options.

1. For a lighter loaded cluster Control-Plane Nodes can also be part of the work-load along with their Worker nodes. 
2. For a highly busy workload, it is recommended to off-load the Control-Plane Nodes from the work load so that Control-Plane nodes are not effected by the heavy usage of the cluster resources and only use Worker nodes for processing the business logic.

Control-Plane Nodes (without Workload)

If workload is enabled for Control-Plane Nodes, please enhance these figures to maximum available like 16GiB RAM, 8 vCPUS and 250 GIB of storage.

Worker Nodes

Worker nodes requirements are more energetic to entertain the workload.

RASA-X Prerequisites

In a Multi-node cluster, Rasa-X can be deployed in different tiers:

1. In a multi-node HA cluster, add another node and deploy the Rasa-X using Node-Affinity in such a way that allocates 1 worker node to RASA-X only. This method is preferred in a HA cluster. Read more at Node Affinity and Node Selector.
2. However, if this is not a do-able option, RKE2 for Single-Node deployment should be used for standalone RASA-X and then configure the CIM accordingly.

Superset Prerequisites

For BI Reporting, Superset must be deployed separately from the main CIM Solution.

In a Multi-node cluster, Superset can be deployed in different tiers

1. In a multi-node HA cluster, add another node and deploy the Superset using Node-Affinity in such a way that allocates 1 worker node specifically to superset only. This method is preferred in a HA cluster. Read Node Affinity and Node Selector
2. However, if this is not a do-able option, RKE2 for Single Node Deployment should be used for standalone Superset and then configure the CIM accordingly.

¹ RKE2 detailed requirements are also available at this.

² Kubernetes detailed requirements can be seen here.

This deployment model requires that your connection to the system is stable and consistent. You can use any virtual terminal like 'screen' or 'tmux' which gives you the ability to resume your session even if the network gets disconnected.

FQDN 

An FQDN must be mapped to an IP address

Iptables​

If you are running iptables in nftables mode instead of legacy you might encounter issues. We recommend utilizing newer iptables (such as 1.6.1+) to avoid issues.

Additionally, versions 1.8.0-1.8.4 have known issues that can cause RKE2 to fail. See Additional OS Preparations for workarounds.

Prepare all the Nodes in the cluster

Disable Services

Disable firewall and nm-cloud-setup on all nodes.

Environment Preparation

Before starting with RKE2 installation, following are the optional steps and the checklist to make sure that environment is prepared for the installation:

systemctl disable apparmor.service
systemctl disable firewalld.service
systemctl stop apparmor.service
systemctl stop firewalld.service

subscription-manager release --set=8.4 ;
yum clean all;
subscription-manager release --show;
rm -rf /var/cache/dnf

systemctl disable swap.target
swapoff -a

yum update -y

Checklist

Before proceeding with the deployment of HA cluster for RKE2, go through the checklist:

1. Air-Gapped deployment is also possible, check RKE2 web-site for more details at Air-Gapped install of K3s
2.  if any of the nodes is not running iscsid.service, the stateful workload will fail and may result in data loss .

Corporate HTTP/S Proxy Requirement

If the environment has strict HTTP or HTTPS proxy set, we must exclude the environment from the proxy controls. 

The NO_PROXY variable must include your  cluster pod and service IP ranges.

HTTP_PROXY=http://your-proxy.example.com:8888
HTTPS_PROXY=http://your-proxy.example.com:8888
NO_PROXY=127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,.svc,.cluster.local

If you want to configure the proxy settings for container without affecting K3s and the Kubelet, you can prefix the variables with CONTAINERD_:

CONTAINERD_HTTP_PROXY=http://your-proxy.example.com:8888
CONTAINERD_HTTPS_PROXY=http://your-proxy.example.com:8888
CONTAINERD_NO_PROXY=127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,.svc,.cluster.local

Customize the RKE2 Deployment for your Environment EXTRA OPTION

Environment Customization Steps

Click here to see customization steps.....

Below given options can also be used for customized environment setup:

Option	Switch	Default	Description
Default Deployment Directory of K3s	--data-dir value, -d value	/var/lib/rancher/rke2	Folder to hold state
Default POD IP Assignment Range	--cluster-cidr value	"10.42.0.0/16"	IPv4/IPv6 network CIDRs to use for pod IPs
Default Service IP Assignment Range	--service-cidr value	"10.43.0.0/16"	IPv4/IPv6 network CIDRs to use for service IPs

If any of the above option is required, add it in the next step.

cluster-cidr and service-cidr are independently evaluated. Decide wisely well before the the cluster deployment. This option is not configurable once the cluster is deployed and workload is running.

Installation Steps

Step 1: Prepare First Control Plane

1. Run the below commands to create required directories for RKE2 configurations.

mkdir -p /etc/rancher/rke2/
mkdir -p  /var/lib/rancher/rke2/server/manifests/

2. Create a deployment manifest called config.yaml for RKE2 Cluster and replace the IP addresses and corresponding FQDNS according.( add any other fields from the Extra Options sections in config.yaml  at this point ) 

cat<<EOF|tee /etc/rancher/rke2/config.yaml
tls-san:
  - devops67.ef.com
  - 10.192.168.67
  - devops61.ef.com
  - 10.192.168.61
  - devops62.ef.com
  - 10.192.168.62
  - devops63.ef.com
  - 10.192.168.63
write-kubeconfig-mode: "0600"
etcd-expose-metrics: true
cni:
  - canal

EOF

In above mentioned template manifest,

• 10.192.168.67 is the Kube-VIP  IP
• devops67.ef.com is the Kube-VIP FQDN
• remaining IPs and FQDN are for all 3 Control Planes
  

Step 2: Ingress-Nginx config for RKE2

1. By default RKE-2 based ingress controller doesn't allow additional snippet information in ingress manifests. Create this config before starting the deployment of RKE2.

cat<<EOF| tee /var/lib/rancher/rke2/server/manifests/rke2-ingress-nginx-config.yaml
---
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: rke2-ingress-nginx
  namespace: kube-system
spec:
  valuesContent: |-
    controller:
      metrics:
        service:
          annotations:
            prometheus.io/scrape: "true"
            prometheus.io/port: "10254"
      config:
        use-forwarded-headers: "true"
      allowSnippetAnnotations: "true"
EOF

2. Begin the RKE2 Deployment

curl -sfL https://get.rke2.io | INSTALL_RKE2_TYPE=server sh -

3. Start the RKE2 service. Starting the Service will take approx. 10-15 minutes based on the network connection.

systemctl start rke2-server

Step 3: Enable the RKE2 Service

1. Enable RKE2 service.

systemctl enable rke2-server

2. By default RKE2 deploys all the binaries in /var/lib/rancher/rke2/bin path. Add this path to system's default PATH for kubectl utility to work appropriately.

export PATH=$PATH:/var/lib/rancher/rke2/bin
export KUBECONFIG=/etc/rancher/rke2/rke2.yaml

3. Also, append these lines into current user's .bashrc  file.

echo "export PATH=$PATH:/var/lib/rancher/rke2/bin" >> $HOME/.bashrc
echo "export KUBECONFIG=/etc/rancher/rke2/rke2.yaml"  >> $HOME/.bashrc 

4. Get the token for joining other Control-Plane Nodes.

cat /var/lib/rancher/rke2/server/node-token

Step 4: Deploy Kube-VIP

1. Decide the IP and the interface on all nodes for Kube-VIP and setup these as environment variables. This step must be completed before deploying any other node in the cluster (both CP and Workers).

export VIP=10.192.168.67
export INTERFACE=enp1s0

2. Import the RBAC manifest for Kube-VIP

curl https://kube-vip.io/manifests/rbac.yaml > /var/lib/rancher/rke2/server/manifests/kube-vip-rbac.yaml

3. Fetch the kube-vip image 

/var/lib/rancher/rke2/bin/crictl -r "unix:///run/k3s/containerd/containerd.sock"  pull ghcr.io/kube-vip/kube-vip:latest

4. Deploy the Kube-VIP 

CONTAINERD_ADDRESS=/run/k3s/containerd/containerd.sock  ctr -n k8s.io run \
--rm \
--net-host \
ghcr.io/kube-vip/kube-vip:latest vip /kube-vip manifest daemonset --arp --interface $INTERFACE --address $VIP --controlplane  --leaderElection --taint --services --inCluster | tee /var/lib/rancher/rke2/server/manifests/kube-vip.yaml

5. Wait for the kube-vip to complete bootstrapping 

kubectl rollout status daemonset   kube-vip-ds    -n kube-system   --timeout=650s

6. Once the condition is met, you can check the daemonset by kube-vip is running 1 pod 

kubectl  get ds -n kube-system  kube-vip-ds

Once the cluster has more control-plane nodes added, the count will be equal to the total number of CP nodes.

Step 5: Remaining Control-Plane Nodes

Perform these steps on remaining control-plane nodes.

1. Create required directories for RKE2 configurations.

mkdir -p /etc/rancher/rke2/
mkdir -p  /var/lib/rancher/rke2/server/manifests/

2. Create a deployment manifest called config.yaml  for RKE2 Cluster  and replace the IP addresses and corresponding FQDNS according (add any other fields from the Extra Options sections in config.yaml  at this point).

cat<<EOF|tee /etc/rancher/rke2/config.yaml
server: https://10.192.168.67:9345
token: [token from /var/lib/rancher/rke2/server/node-token on server node 1]
write-kubeconfig-mode: "0644" tls-san:
  - devops67.ef.com
  - 10.192.168.67
  - devops61.ef.com
  - 10.192.168.61
  - devops62.ef.com
  - 10.192.168.62
  - devops63.ef.com
  - 10.192.168.63
write-kubeconfig-mode: "0644"
etcd-expose-metrics: true
cni:
  - canal

EOF

In above mentioned template manifest,

• 10.192.168.67 is theKube-VIP  IP
• devops67.ef.com is the Kube-VIP  FQDN
• remaining IPs and FQDN are for all 3 Control Planes

Ingress-Nginx config for RKE2

By default RKE-2 based ingress controller doesn't allow additional  snippet information in ingress manifests, create this config before starting the deployment of RKE2 

cat<<EOF| tee /var/lib/rancher/rke2/server/manifests/rke2-ingress-nginx-config.yaml
---
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: rke2-ingress-nginx
  namespace: kube-system
spec:
  valuesContent: |-
    controller:
      metrics:
        service:
          annotations:
            prometheus.io/scrape: "true"
            prometheus.io/port: "10254"
      config:
        use-forwarded-headers: "true"
      allowSnippetAnnotations: "true"
EOF

Step 6: Begin the RKE2 Deployment

1. Begin the RKE2 Deployment

curl -sfL https://get.rke2.io | INSTALL_RKE2_TYPE=server sh -

2. Start the RKE2 service. Starting the Service will take approx. 10-15 minutes based on the network connection

systemctl start rke2-server

3. Enable the RKE2 Service

systemctl enable rke2-server

4. By default, RKE2 deploys all the binaries in /var/lib/rancher/rke2/bin  path. Add this path to system's default PATH for kubectl utility to work appropriately.

export PATH=$PATH:/var/lib/rancher/rke2/bin
export KUBECONFIG=/etc/rancher/rke2/rke2.yaml

5. Also, append these lines into current user's .bashrc  file

echo "export PATH=$PATH:/var/lib/rancher/rke2/bin" >> $HOME/.bashrc
echo "export KUBECONFIG=/etc/rancher/rke2/rke2.yaml"  >> $HOME/.bashrc 

Step 7: Deploy Worker Nodes

1. Create the RKE2 directory

mkdir -p /etc/rancher/rke2/

 2. Create the config.yaml  

cat<<EOF|tee /etc/rancher/rke2/config.yaml
server: https://10.192.168.67:9345
token: [token from /var/lib/rancher/rke2/server/node-token on server node 1]
write-kubeconfig-mode: \"0644\"

EOF

3. Initiate the deployment of RKE2

curl -sfL https://get.rke2.io | INSTALL_RKE2_TYPE=agent sh -

4. Start the RKE2 Agent Service

systemctl start rke2-agent.service

5. Enable the RKE2 Agent Service to start at the boot time

systemctl enable  rke2-agent.service

6. Repeat these steps for all worker nodes

Step 7: Bash Completion for kubectl

1. Install bash-completion package

yum install bash-completion -y

2. Setup autocomplete in bash into the current shell, bash-completion  package should be installed first.

source <(kubectl completion bash) 
echo "source <(kubectl completion bash)" >> ~/.bashrc 

3. Also, add alias for short notation of kubectl

echo "alias k=kubectl"  >> ~/.bashrc 
echo "complete -o default -F __start_kubectl k"  >> ~/.bashrc 

4. Source your ~/.bashrc  

source ~/.bashrc

Step 8: Install Helm

1. Helm is a nifty tool to deploy external components. To install helm on cluster, execute the following command.

curl -fsSL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3|bash

2. Enable bash completion for Helm

helm completion bash > /etc/bash_completion.d/helm

3. and relogin to enable the bash completion  or do `su - ` if running as `root ` user

4. List the cluster nodes' details. You can get the details of all nodes using the following command:

kubectl get nodes -o wide

You may install helm on only one of any master nodes.

Storage

RKE2 Kubernetes requires that at least 1 of the storage-class is available for storing data on the cluster, This is a mandatory step and requires the operator to decide well before deploying the production workload. Details provided are self-explanatory and should be considered according to the cluster usage.

Longhorn for Replicated Storage

Longhorn deployment is available at Longhorn Deployment Guide. This deployment model is for lighter scale cluster workloads and should be used with cautions that longhorn will require additional hardware specs for a production cluster. If this is only option, consider deploying the Longhorn on dedicated only in the cluster using node-affinity.

OpenEBS for Local Storage

Deploying OpenEBS enables localhost storage as target devices and can only be used in below given scenarios. 

• Deployment of StatefulSets using nodeSelectors. In this deployment model, each statefulset is confined to a particular node so that it always be running on the same node. However, this inverses the High Availability of the statefulset services in such a way that when 1 worker node goes down, all services will not be available until the node recovers.
• Deploy StatefulSets in High-level replication and use local disks on each node. this deployment model gives the flexibility of having at least 3 nodes available with completes services.

Details on OpenEBS can be read here.

Expertflow CX Deployment on Kubernetes

Please follow the steps in the document, Expertflow CX Deployment on Kubernetes to deploy Expertflow CX Solution.