Log retention on Kubernetes

Logs retention sometimes called logs aggregation as well, performs collection of logs from all kubernetes engine sources e-g pods, system services, or nodes and correlate them based on the filtering algorithm that they can be digested appropriately and easily at the target service. Below are some of the examples we can implement on the kubernetes cluster to route logs from the Kubernetes to different targets.

EFK to Elasticsearch/Kibana

Requirements

For logs retention on a kubernetes cluster, below given Hardware specs are required for the solution. Please consult official documentation for the cluster size.

Component	CPU	RAM(GB)	Disk( GB)
elasticsearch	4	16	250
Kibana	2	4	N/A
fluent-bit	0.5	.5

A dedicated worker node will be required with above given specs.

Introduction

This document proposes a solution to retain logs for the solution deployed on any Kubernetes engine. These logs are then served in a web administration pane of ElasticSearch and are stored in its own backend storage supported by the respective storage class configured on the cluster. Ideally, when configuring the logs retention using this document, kindly consider reserving enough storage for the volume to hold as much logs as required or mandated in the customer environment.

For details on how Kubernetes maintains logs on the node level, please consult this Kubernetes Logging Mechanism

We are using Fluentbit for preprocessing the logs before sending them to Elasticsearch. Fluentbit collects data and logs from different sources, unifies them, and sends them to multiple destinations. It has a minimal footprint ~450KB as compared to its predeceases Fluentd which has 40 MB of footprint which makes Fluentbit a super fast, lightweight, and highly scalable logging and metrics processor and forwarder.

First, need to clone the repository.

CODE

git clone -b <branch>   https://gitlab.expertflow.com/cim/cim-solution.git

you should insert the name of the release branch name here <branch> i.e. CIM-1.0-Beta-SR9

After cloning the repository ,run the following command:

CODE

cd kubernetes/logging/EFK
kubectl create ns logging

Generate self-signed certificates. You can skip the following step if you already have valid certificates.

CODE

openssl req -x509 \
-newkey rsa:4096 \
-sha256 \
-days 3650 \
-nodes \
-keyout devops238.ef.com.key \
-out devops238.ef.com.crt \
-subj "/CN=devops238.ef.com" \
-addext "subjectAltName=DNS:devops238.ef.com,DNS:devops238.ef.com"

Replace the --key and --cert field with the name of your own key and cert files.

CODE

 kubectl create secret tls elastic-certificates -n logging \
 --key devops238.ef.com.key \
 --cert devops238.ef.com.crt \
 --dry-run=client -o yaml > elastic-certificates-secret.yml

Apply login credentials and certificates secrets.

CODE

kubectl apply -f elastic-certificates-secret.yml
kubectl apply -f elastic-credentials-secret.yml

Install Elastic search

Before Installing ElasticSearch, please confirm the storage class in values.yaml file matches your storage class.

CODE

helm upgrade --install=true elasticsearch \
--namespace logging \
--values elasticsearch/values-small.yaml elasticsearch/

for the validation of the Elasticsearch deployment, type:
curl -sk -u "elastic:vXVWD81ms2s6B56KVGQO" https://devops238.ef.com:9200/ | jq

Install Kibana

CODE

helm upgrade --install=true kibana \
  kibana/ \
  --namespace logging \
  --values kibana/values-large.yaml

Install Fluentbit

CODE

cd fluent-bit

kubectl apply  -f fb-role.yaml \
-f fb-rolebind.yaml \
-f fb-service.yaml \
-f fb-configmap.yaml \
-f fb-ds.yaml

You can access Kibana on the following link:

CODE

https://devops238.ef.com:5601/

username: elastic

password: vXVWD81ms2s6B56KVGQO

ELK to Syslog

Introduction

This document proposes a solution to retain logs for the solution deployed on any Kubernetes engine. We use filebeat to preprocess logs before delivering them to logstash.Filebeat collects and combines data and logs from many sources before sending them to logstash.For log routing, logstash is utilised.Syslog is used for both storage and presentation.

First, need to clone the repository.

CODE

git clone -b CIM-1.0-Beta-SR15  https://gitlab.expertflow.com/cim/cim-solution.git

you should insert the name of the release branch name here <release_branch> i.e. CIM-1.0-Beta-SR13

After cloning the repository, run the following command:

CODE

cd kubernetes/logging/syslog-ELK

Create the logging namespace

CODE

kubectl create ns logging

Apply the Kubernetes Manifests:

CODE

kubectl apply -f filebeat.yaml

Edit the logstash-values.yaml file to reflect your environment and run the helm command to deploy the logstash

CODE

helm upgrade --install --namespace=logging --values=logstash-values.yaml logstash logstash

wait for sometime to let both filebeat and logstash synchronize and know each other's presence.

You can explore the flow of logs by tailing logs of logstash pod.

CODE

kubectl -n logging logs -f logstash-pod-name

CODE

helm upgrade --install=true kibana \
  kibana/ \
  --namespace logging \
  --values kibana/values-small.yaml