Banzai Cloud Logging Operator Deployment in Kubernetes

Introduction

This document illustrates the procedure and steps to deploy BanzaiCloud Logging Operator in Kubernetes. The BanzaiCloud Logging Operator is designed to simplify and automate log collection in Kubernetes clusters. The Logging Operator leverages popular open-source technologies such as Fluentd, Fluent Bit,to aggregate, process, and store logs generated by applications running in Kubernetes. It streamlines the deployment and configuration of these components, making it easier to set up a scalable and efficient logging infrastructure.

Logging Hierarchy

Benzai Cloud Logging works as an operator in kubernetes which means the instances are created as per the requirement for a said objective. In Benzai Cloud Logging Operator, these are mainly 

• fluentd
• fluentbit

In previous step #3 we have already created an instance for the logging to routed to respective Upstream. Below given diagram represents the flow in best way

Feature highlights

 Namespace isolation

 Native Kubernetes label selectors

 Secure communication (TLS)

 Configuration validation

 Multiple flow support (multiply logs for different transformations)

 Multiple output support (store the same logs in multiple storage: S3, GCS, ES, Loki and more...)

 Multiple logging system support (multiple fluentd, fluent-bit deployment on the same cluster)

Architecture

The Logging operator manages the log collectors and log forwarders of your logging infrastructure, and the routing rules that specify where you want to send your different log messages.

The log collectors are endpoint agents that collect the logs of your Kubernetes nodes and send them to the log forwarders. Logging operator currently uses Fluent Bit as log collector agents.

The log forwarder instance receives, filters, and transforms the incoming the logs, and transfers them to one or more destination outputs. The Logging operator supports Fluentd and syslog-ng as log forwarders. Which log forwarder is best for you depends on your logging requirements.

You can filter and process the incoming log messages using the flow custom resource of the log forwarder to route them to the appropriate output. The outputs are the destinations where you want to send your log messages, for example, Elasticsearch, or an Amazon S3 bucket. You can also define cluster-wide outputs and flows, for example, to use a centralized output that namespaced users can reference but cannot modify. Note that flows and outputs are specific to the type of log forwarder you use (Fluentd or syslog-ng).

You can configure the Logging operator using the following Custom Resource Definitions.

• Logging - The Logging resource defines the logging infrastructure (the log collectors and forwarders) for your cluster that collects and transports your log messages. It also contains configurations for Fluent Bit, Fluentd, and syslog-ng.
• CRDs for Fluentd:
  • Output - Defines a Fluentd Output for a logging flow, where the log messages are sent using Fluentd. This is a namespaced resource. See also ClusterOutput. To configure syslog-ng outputs, see SyslogNGOutput.
  • Flow - Defines a Fluentd logging flow using filters and outputs. Basically, the flow routes the selected log messages to the specified outputs. This is a namespaced resource. See also ClusterFlow. To configure syslog-ng flows, see SyslogNGFlow.
  • ClusterOutput - Defines a Fluentd output that is available from all flows and clusterflows. The operator evaluates clusteroutputs in the controlNamespace only unless allowClusterResourcesFromAllNamespaces is set to true.
  • ClusterFlow - Defines a Fluentd logging flow that collects logs from all namespaces by default. The operator evaluates clusterflows in the controlNamespace only unless allowClusterResourcesFromAllNamespaces is set to true. To configure syslog-ng clusterflows, see SyslogNGClusterFlow.

In a summary, the logs from any range of objects like pods or containers to namespace and namespaces to whole cluster can be selected in Flow which is destined to route these collected logs to multiple Outputs which can be any or multiple of the destinations Loki, ELK, Amazon S3, kafka, splunk etc )

Why Benzai Cloud logging Operator

The Banzai Cloud Logging Operator helps Kubernetes users overcome the complexities of log collection and management in distributed environments. It provides a streamlined solution for centralizing and processing logs, enabling better observability and troubleshooting capabilities for your applications.

Installation Steps

1. Run the following command to add the repo.

helm repo add banzaicloud-stable https://kubernetes-charts.banzaicloud.com

2.Create ef-logging namespace

kubectl create namespace logging 

3. The Helm command for deploying Operator is as follows:

helm upgrade --install --wait  --namespace logging logging-operator banzaicloud-stable/logging-operator

The logging operator will be deployed after executing the above command.

Which log forwarder to use

The Logging operator supports Fluentd and syslog-ng (via the AxoSyslog syslog-ng distribution) as log forwarders. The log forwarder instance receives, filters, and transforms the incoming the logs, and transfers them to one or more destination outputs. Which one to use depends on your logging requirements.

The following points help you decide which forwarder to use.

• The forwarders support different outputs. If the output you want to use is supported only by one forwarder, use that.
• If the volume of incoming log messages is high, use syslog-ng, as its multithreaded processing provides higher performance.
• If you have lots of logging flows or need complex routing or log message processing, use syslog-ng.

Note: Depending on which log forwarder you use, some of the CRDs you have to create and configure are different.

syslog-ng is supported only in Logging operator 4.0 or newer.

Using Fluentd as Log Forwarder

Create this logging object to define the logging infrastructure using fluentbit  and fluentd 

cat<<EOF| tee /root/loging-config.yaml
------
apiVersion: logging.banzaicloud.io/v1beta1
kind: Logging
metadata:
  name: default-logging-simple
  namespace: logging
spec:
  fluentd: {}
  fluentbit: {}
  controlNamespace: ef-logging
EOF

Using syslogNG as the log forwarder

Below given manifest can be use to route logs using syslogNG

apiVersion: logging.banzaicloud.io/v1beta1
kind: Logging
metadata:
  name: syslog-ng-logging
  namespace: logging
spec:
  enableRecreateWorkloadOnImmutableFieldChange: true
  fluentbit:
    bufferStorage: {}
    bufferStorageVolume:
      hostPath:
        path: ""
    bufferVolumeImage: {}
    filterKubernetes: {}
    image: {}
    inputTail:
      storage.type: filesystem
    positiondb:
      hostPath:
        path: ""
    resources: {}
    updateStrategy: {}
  syslogNG:
#    jsonKeyDelim: "~"
    nodeSelector:
	   - 
    statefulSet:
      spec:
        template:
          spec:
            containers:
            - name: syslog-ng
              volumeMounts:
              - mountPath: /buffers
                name: buffer
        volumeClaimTemplates:
        - metadata:
            name: buffer
          spec:
            accessModes:
            - ReadWriteOnce
            resources:
              requests:
                storage: 20Gi
  controlNamespace: logging