Monitoring Solution Deployment

Requirements

This document covers the process of deploying the Monitoring solution stack for CX. This stack consists of the following components.

1. Prometheus

2. Grafana

3. Alertmanager

4. Node Exporter

The Prometheus operator includes the following features:

• Kubernetes Custom Resources: Use Kubernetes custom resources to deploy and manage Prometheus, Alertmanager, and related components.

• Simplified Deployment Configuration: Configure the fundamentals of Prometheus like versions, persistence, retention policies, and replicas from a native Kubernetes resource.

• Prometheus Target Configuration: Automatically generate monitoring target configurations based on familiar Kubernetes label queries; no need to learn a Prometheus-specific configuration language.

Considerations

Monitoring stack helm chart is customized to accommodate the monitoring requirements of the CX Solution monitoring.

add the helm repository

helm repo add expertflow https://expertflow.github.io/charts

update the repository

helm repo update expertflow

make the directory

mkdir helm-values

customize the parameters by cloning the values file

helm show values expertflow/kube-prometheus-stack > helm-values/kube-prometheus-stack-values.yaml

• create monitoring namespace

  CODE

  kubectl create namespace monitoring

• Alert Manager Ingress (Optional)

  • alert-manager ingress is disabled by default as it can be explored publicly by anyone without any authentication, exposing potentially critical information . However if this is required, please follow these steps

    • Create an apache2 pod by running

      CODE

       kubectl run apache2 --image=bitnami/apache2 

    • Create secret using. Change the password placeholder <CHANGE_ME> with the required value.

      CODE

      kubectl -n monitornig create secret generic  basic-auth  --from-literal=auth="$(kubectl exec -ti apache2 -- bash -c 'echo <CHANGE_ME>|htpasswd -i -n monitoring')" 

    • Delete the apache2 pod

      CODE

      kubectl delete pod apache2

• Prometheus Ingress (Optional)

  • Prometheus ingress is disabled by default as it can be explored publicly by anyone without any authentication, exposing potentially critical information . However if this is required ( skip these steps if already completed for alert-manager as both ingresses use same auth token and credentials as kubernetes secret.)

    • Create an apache2 pod by running

      CODE

       kubectl run apache2 --image=bitnami/apache2 

    • Create secret using. Change the password placeholder <CHANGE_ME> with the required value.

      CODE

      kubectl -n monitornig create secret generic  basic-auth  --from-literal=auth="$(kubectl exec -ti apache2 -- bash -c 'echo <CHANGE_ME>|htpasswd -i -n monitoring')" 

    • Delete the apache2 pod

      CODE

      kubectl delete pod apache2

Considerations to cater for when changing values file helm-values/kube-prometheus-stack-values.yaml

• Replace all occurrences of devops.ef.com to FQDN. You can use sed -i -e 's/devops.ef.com/<FQDN>/g' helm-values/kube-prometheus-stack-values.yaml

• change the default password for grafana admin user under grafana section. optionally command sed -i -e 's/Expertflow123/<CUSTOM_PASSWORD>/g' helm-values/kube-prometheus-stack-values.yaml can also be used

• change the retention period for prometheus. default 30 days. command to substitute this parameter can be sed -i -e 's/^retention: 30d/retention: <NUMBER_OF_DAYS>d' helm-values/kube-prometheus-stack-values.yaml

• If the monitoring solution is to be deployed on a specific node in cluster, update the nodeSelector term for Alert-Manager, Grafana and Prometheus or use appropriate nodeAffinity terms for proper placement plan of monitoring solution.

• Create secret containing TLS certificates in monitoring namespace. All ingress resources use ef-ingress-tls-secret as secret name for TLS encryption.

  • For Self Signed please use this guide Create self-signed certificates for ingress

  • For commercial SSL/TLS certificates, please import them as tls.crt and tls.key and create secret with the name of ef-ingress-tls-secret in both ef-external and expertflow namespaces

  • For LetsEncrypt based SSL/TLS Certificates please consult LetsEncrypt SSL for EF-CX

• update the Grafana password (adminPassword) . The default password is Expertflow123

Deploy the monitoring Solution stack helm chart.

helm upgrade --namespace monitoring --install=true kube-stack-prometheus --values=helm-values/kube-prometheus-stack-values.yaml  expertflow/kube-prometheus-stack

after all the pods created successfully, you can access the monitoring solution using https://FQDN>/monitoring with these credentials

Username: admin

Password: Password set in helm-values/kube-prometheus-stack-values.yaml Default Password is Expertflow123