Skip to main content
Skip table of contents

Log retention on Kubernetes

Logs retention sometimes called logs aggregation as well, performs collection of logs from all  kubernetes engine sources e-g pods, system services, or nodes and correlate them based on the filtering algorithm that they can be digested appropriately and easily at the target service. Below are some of the examples we can implement on the kubernetes cluster to route logs from the Kubernetes to different targets.  

EFK to Elasticsearch/Kibana 


For logs retention on a kubernetes cluster,  below given Hardware specs are required for the solution. Please consult official documentation for the cluster size.

ComponentCPURAM(GB)Disk( GB)

A dedicated worker node will be required with above given specs.


This document proposes a solution to retain logs for the solution deployed on any Kubernetes engine. These logs are then served in a web administration pane of ElasticSearch and are stored in its own backend storage supported by the respective storage class configured on the cluster. Ideally, when configuring the logs retention using this document, kindly consider reserving enough storage for the volume to hold as much logs as required or mandated in the customer environment. 

For details on how Kubernetes maintains logs on the node level, please consult this Kubernetes Logging Mechanism

We are using Fluentbit for preprocessing the logs before sending them to Elasticsearch. Fluentbit collects data and logs from different sources, unifies them, and sends them to multiple destinations. It has a minimal footprint ~450KB as compared to its predeceases Fluentd which has 40 MB of footprint which makes  Fluentbit a super fast, lightweight, and highly scalable logging and metrics processor and forwarder.

First, need to clone the repository.

git clone -b <branch> 

you should insert the name of the release branch name here <branch> i.e. CIM-1.0-Beta-SR9

After cloning the repository ,run the following command:

cd kubernetes/logging/EFK
kubectl create ns logging

Generate self-signed certificates. You can skip the following step if you already have valid certificates.

openssl req -x509 \
-newkey rsa:4096 \
-sha256 \
-days 3650 \
-nodes \
-keyout \
-out \
-subj "/" \
-addext ","

Replace the --key and --cert field with the name of your own key and cert files.

 kubectl create secret tls elastic-certificates -n logging \
 --key \
 --cert \
 --dry-run=client -o yaml > elastic-certificates-secret.yml

Apply login credentials and certificates secrets.

kubectl apply -f elastic-certificates-secret.yml
kubectl apply -f elastic-credentials-secret.yml

Install Elastic search

Before Installing ElasticSearch, please confirm the storage class in values.yaml file matches your  storage class.

helm upgrade --install=true elasticsearch \
--namespace logging \
--values elasticsearch/values-small.yaml elasticsearch/

for the validation of the Elasticsearch deployment, type: 
curl -sk -u "elastic:vXVWD81ms2s6B56KVGQO" | jq

Install Kibana

helm upgrade --install=true kibana \
  kibana/ \
  --namespace logging \
  --values kibana/values-large.yaml

Install Fluentbit

cd fluent-bit

kubectl apply  -f fb-role.yaml \
-f fb-rolebind.yaml \
-f fb-service.yaml \
-f fb-configmap.yaml \
-f fb-ds.yaml

You can access Kibana on the following link:


username: elastic

password: vXVWD81ms2s6B56KVGQO

ELK to Syslog


This document proposes a solution to retain logs for the solution deployed on any Kubernetes engine. We use filebeat to preprocess logs before delivering them to logstash.Filebeat collects and combines data and logs from many sources before sending them to logstash.For log routing, logstash is utilised.Syslog is used for both storage and presentation.

First, need to clone the repository.

git clone -b CIM-1.0-Beta-SR15  

you should insert the name of the release branch name here <release_branch> i.e. CIM-1.0-Beta-SR13

After cloning the repository, run the following command:

cd kubernetes/logging/syslog-ELK 

Create the logging namespace

kubectl create ns logging

Apply the Kubernetes Manifests:

kubectl apply -f filebeat.yaml

Edit the logstash-values.yaml  file to reflect your environment and run the helm command to deploy the logstash

helm upgrade --install --namespace=logging --values=logstash-values.yaml logstash logstash

wait for sometime to let both filebeat and logstash synchronize and know each other's presence. 

You can explore the flow of logs by tailing logs of logstash  pod.

kubectl -n logging logs -f logstash-pod-name 

helm upgrade --install=true kibana \
  kibana/ \
  --namespace logging \
  --values kibana/values-small.yaml

JavaScript errors detected

Please note, these errors can depend on your browser setup.

If this problem persists, please contact our support.