Skip to main content
Skip table of contents

Log retention on Kubernetes

Logs retention sometimes called logs aggregation as well, performs collection of logs from all  kubernetes engine sources e-g pods, system services, or nodes and correlate them based on the filtering algorithm that they can be digested appropriately and easily at the target service. Below are some of the examples we can implement on the kubernetes cluster to route logs from the Kubernetes to different targets.  

EFK to Elasticsearch/Kibana 


Requirements


For logs retention on a kubernetes cluster,  below given Hardware specs are required for the solution. Please consult official documentation for the cluster size.

ComponentCPURAM(GB)Disk( GB)
elasticsearch416250
Kibana24N/A
fluent-bit0.5.5

A dedicated worker node will be required with above given specs.



Introduction

This document proposes a solution to retain logs for the solution deployed on any Kubernetes engine. These logs are then served in a web administration pane of ElasticSearch and are stored in its own backend storage supported by the respective storage class configured on the cluster. Ideally, when configuring the logs retention using this document, kindly consider reserving enough storage for the volume to hold as much logs as required or mandated in the customer environment. 

For details on how Kubernetes maintains logs on the node level, please consult this Kubernetes Logging Mechanism


We are using Fluentbit for preprocessing the logs before sending them to Elasticsearch. Fluentbit collects data and logs from different sources, unifies them, and sends them to multiple destinations. It has a minimal footprint ~450KB as compared to its predeceases Fluentd which has 40 MB of footprint which makes  Fluentbit a super fast, lightweight, and highly scalable logging and metrics processor and forwarder.

First, need to clone the repository.

CODE
git clone -b <branch>   https://gitlab.expertflow.com/cim/cim-solution.git 

you should insert the name of the release branch name here <branch> i.e. CIM-1.0-Beta-SR9

After cloning the repository ,run the following command:

CODE
cd kubernetes/logging/EFK
kubectl create ns logging

Generate self-signed certificates. You can skip the following step if you already have valid certificates.

CODE
openssl req -x509 \
-newkey rsa:4096 \
-sha256 \
-days 3650 \
-nodes \
-keyout devops238.ef.com.key \
-out devops238.ef.com.crt \
-subj "/CN=devops238.ef.com" \
-addext "subjectAltName=DNS:devops238.ef.com,DNS:devops238.ef.com"

Replace the --key and --cert field with the name of your own key and cert files.

CODE
 kubectl create secret tls elastic-certificates -n logging \
 --key devops238.ef.com.key \
 --cert devops238.ef.com.crt \
 --dry-run=client -o yaml > elastic-certificates-secret.yml

Apply login credentials and certificates secrets.

CODE
kubectl apply -f elastic-certificates-secret.yml
kubectl apply -f elastic-credentials-secret.yml

Install Elastic search

Before Installing ElasticSearch, please confirm the storage class in values.yaml file matches your  storage class.

CODE
helm upgrade --install=true elasticsearch \
--namespace logging \
--values elasticsearch/values-small.yaml elasticsearch/

for the validation of the Elasticsearch deployment, type: 
curl -sk -u "elastic:vXVWD81ms2s6B56KVGQO" https://devops238.ef.com:9200/ | jq

Install Kibana

CODE
helm upgrade --install=true kibana \
  kibana/ \
  --namespace logging \
  --values kibana/values-large.yaml

Install Fluentbit

CODE
cd fluent-bit

kubectl apply  -f fb-role.yaml \
-f fb-rolebind.yaml \
-f fb-service.yaml \
-f fb-configmap.yaml \
-f fb-ds.yaml

You can access Kibana on the following link:

CODE
https://devops238.ef.com:5601/

username: elastic

password: vXVWD81ms2s6B56KVGQO




ELK to Syslog

Introduction

This document proposes a solution to retain logs for the solution deployed on any Kubernetes engine. We use filebeat to preprocess logs before delivering them to logstash.Filebeat collects and combines data and logs from many sources before sending them to logstash.For log routing, logstash is utilised.Syslog is used for both storage and presentation.

First, need to clone the repository.

CODE
git clone -b CIM-1.0-Beta-SR15  https://gitlab.expertflow.com/cim/cim-solution.git  

you should insert the name of the release branch name here <release_branch> i.e. CIM-1.0-Beta-SR13

After cloning the repository, run the following command:

CODE
cd kubernetes/logging/syslog-ELK 

Create the logging namespace

CODE
kubectl create ns logging

Apply the Kubernetes Manifests:

CODE
kubectl apply -f filebeat.yaml

Edit the logstash-values.yaml  file to reflect your environment and run the helm command to deploy the logstash

CODE
helm upgrade --install --namespace=logging --values=logstash-values.yaml logstash logstash

wait for sometime to let both filebeat and logstash synchronize and know each other's presence. 

You can explore the flow of logs by tailing logs of logstash  pod.

CODE
kubectl -n logging logs -f logstash-pod-name 


CODE
helm upgrade --install=true kibana \
  kibana/ \
  --namespace logging \
  --values kibana/values-small.yaml



JavaScript errors detected

Please note, these errors can depend on your browser setup.

If this problem persists, please contact our support.