Breadcrumbs

Deploy Fluentbit

Fluent Bit is an open-source and lightweight log processor and forwarder designed for cloud-native environments. It is part of the Fluent ecosystem and is specifically optimized for performance and memory usage. Fluent Bit enables you to collect, process, and ship logs from various sources to multiple destinations, including log storage and analysis systems such as OpenSearch, Elasticsearch, and various cloud services.

Deployment Steps

To deploy Fluent Bit using Helm, you first need to add the ExpertFlow Helm repository.

helm repo add expertflow https://expertflow.github.io/charts/

Ensure you have the latest charts from the ExpertFlow repository.

helm repo update expertflow

You can deploy Fluent Bit by using a custom values file for configuration. First, create the custom values file based on the default configuration provided by the chart.

helm show values expertflow/fluent-bit > helm-values/cx-fluent-bit-custom-values.yaml

Now, use the following command to deploy Fluent Bit with the custom values.

helm upgrade --install --namespace ef-external --set global.efCxReleaseName="ef-cx"  cx-fluent-bit --debug --values helm-values/cx-fluent-bit-custom-values.yaml expertflow/fluent-bit

After deploying Fluent Bit, it is essential to verify that it is running correctly and processing logs as expected.

  • Ensure that the Fluent Bit pod is running without errors.

kubectl get pods -n ef-external

  • To view the logs of the Fluent Bit pod, use the following command:

kubectl logs -f <fluent-bit-pod-name> -n ef-external

Verify Fluent Bit is Running

Check that the pod is running without errors:

kubectl get pods -n ef-external | grep fluent

Expected output:

cx-fluent-bit-xxxxx   1/1   Running   0   2m

If the pod is in CrashLoopBackOff or Error state, see the Troubleshooting section.

Check the Fluent Bit logs to confirm it is connecting to OpenSearch successfully:

kubectl logs -f <fluent-bit-pod-name> -n ef-external | grep -i "opensearch\|error\|retry"

What to look for:

  • A successful connection shows output referencing the opensearch plugin starting up with no errors.

  • connection refused means the Host or Port is wrong.

  • unauthorized means the HTTP_User or HTTP_Passwd is incorrect.

  • TLS-related errors mean the tls and tls.verify settings need adjustment.

Uninstall Fluentbit

If you need to disable or remove the Fluent Bit deployment, use the following command:

helm uninstall -n ef-external cx-fluent-bit