Log retention on Kubernetes

Logs retention sometimes called logs aggregation as well, performs collection of logs from all  kubernetes engine sources e-g pods, system services, or nodes and correlate them based on the filtering algorithm that they can be digested appropriately and easily at the target service. Below are some of the examples we can implement on the kubernetes cluster to route logs from the Kubernetes to different targets.  

EFK to Elasticsearch/Kibana 

Requirements

For logs retention on a kubernetes cluster,  below given Hardware specs are required for the solution. Please consult official documentation for the cluster size.

A dedicated worker node will be required with above given specs.

Introduction

This document proposes a solution to retain logs for the solution deployed on any Kubernetes engine. These logs are then served in a web administration pane of ElasticSearch and are stored in its own backend storage supported by the respective storage class configured on the cluster. Ideally, when configuring the logs retention using this document, kindly consider reserving enough storage for the volume to hold as much logs as required or mandated in the customer environment. 

For details on how Kubernetes maintains logs on the node level, please consult this Kubernetes Logging Mechanism

We are using Fluentbit for preprocessing the logs before sending them to Elasticsearch. Fluentbit collects data and logs from different sources, unifies them, and sends them to multiple destinations. It has a minimal footprint ~450KB as compared to its predeceases Fluentd which has 40 MB of footprint which makes  Fluentbit a super fast, lightweight, and highly scalable logging and metrics processor and forwarder.

First, need to clone the repository.

git clone -b <branch>   https://gitlab.expertflow.com/cim/cim-solution.git 

After cloning the repository ,run the following command:

cd kubernetes/logging/EFK
kubectl create ns logging

Generate self-signed certificates. You can skip the following step if you already have valid certificates.

openssl req -x509 \
-newkey rsa:4096 \
-sha256 \
-days 3650 \
-nodes \
-keyout devops238.ef.com.key \
-out devops238.ef.com.crt \
-subj "/CN=devops238.ef.com" \
-addext "subjectAltName=DNS:devops238.ef.com,DNS:devops238.ef.com"

Replace the --key and --cert field with the name of your own key and cert files.

 kubectl create secret tls elastic-certificates -n logging \
 --key devops238.ef.com.key \
 --cert devops238.ef.com.crt \
 --dry-run=client -o yaml > elastic-certificates-secret.yml

Apply login credentials and certificates secrets.

kubectl apply -f elastic-certificates-secret.yml
kubectl apply -f elastic-credentials-secret.yml

Install Elastic search

helm upgrade --install=true elasticsearch \
--namespace logging \
--values elasticsearch/values-small.yaml elasticsearch/

Install Kibana

helm upgrade --install=true kibana \
  kibana/ \
  --namespace logging \
  --values kibana/values-large.yaml

Install Fluentbit

cd fluent-bit

kubectl apply  -f fb-role.yaml \
-f fb-rolebind.yaml \
-f fb-service.yaml \
-f fb-configmap.yaml \
-f fb-ds.yaml

You can access Kibana on the following link:

https://devops238.ef.com:5601/

ELK to Syslog

Introduction

This document proposes a solution to retain logs for the solution deployed on any Kubernetes engine. We use filebeat to preprocess logs before delivering them to logstash.Filebeat collects and combines data and logs from many sources before sending them to logstash.For log routing, logstash is utilised.Syslog is used for both storage and presentation.

First, need to clone the repository.

git clone -b CIM-1.0-Beta-SR15  https://gitlab.expertflow.com/cim/cim-solution.git  

After cloning the repository, run the following command:

cd kubernetes/logging/syslog-ELK 

Create the logging namespace

kubectl create ns logging

Apply the Kubernetes Manifests:

kubectl apply -f filebeat.yaml

Edit the logstash-values.yaml  file to reflect your environment and run the helm command to deploy the logstash

helm upgrade --install --namespace=logging --values=logstash-values.yaml logstash logstash

wait for sometime to let both filebeat and logstash synchronize and know each other's presence. 

You can explore the flow of logs by tailing logs of logstash  pod.

kubectl -n logging logs -f logstash-pod-name 

helm upgrade --install=true kibana \
  kibana/ \
  --namespace logging \
  --values kibana/values-small.yaml