Logging and Tracing Integration with OpenSearch

• Kubernetes cluster is running and you have kubectl access

• Helm is installed (version 3.x or later)

This guide, intended for the operations team, walks you through integrating your audit logging system with OpenSearch.

Before You Begin

Confirm all of the following before starting the setup sequence:

• Kubernetes cluster is running and you have kubectl access

• Helm is installed (version 3.x or later)

• The expertflow namespace exists and application pods are running

• The ef-external namespace exists or you have permissions to create it

• You have an OpenSearch instance accessible at a known host or IP address

• You have OpenSearch admin credentials (username and password)

• You have permissions to create Kubernetes namespaces and deploy Helm charts

• CX solution is running; all pods are up and healthy

• Audit logs for keycloak is enabled Enablement Guide For Keycloak Audit Logging and Events Tracing

Architecture

Application Pods (namespace: expertflow)
        │
        │  JSON logs written to pod stdout
        ▼
Fluent Bit DaemonSet (namespace: ef-external)
        │
        │  Filters logs by type: audit_logging
        │  Forwards over HTTP/HTTPS
        ▼
OpenSearch (port 9200)
        │
        │  Indexes into audit_log_index
        ▼
OpenSearch Dashboards (port 5601)
        │
        │  Visualized via index pattern audit_log_index*
        ▼
Operations Team

Each component runs independently. If OpenSearch is unavailable, Fluent Bit buffers logs and retries. If Fluent Bit is down, logs remain in the pod output until it recovers.

For detailed architecture and log formats, refer to this document: Logging and Tracing Architecture

Setup Sequence

Complete these steps in order. Do not proceed to the next step until you have verified the outputs of the current one.

1. Deploy OpenSearch

Set up OpenSearch cluster in your Kubernetes environment using the following guide.

Opensearch and Opensearch Dashboard Deployment Guide

After deployment, verify the cluster is healthy before proceeding:

curl -X GET "http://<opensearch-host>:9200/_cluster/health?pretty" \
  -u admin:<password>

Expected response:

{
  "cluster_name": "opensearch",
  "status": "green",
  ...
}

A status of green or yellow is acceptable. yellow is normal for single-node deployments. If you see red, do not proceed — the cluster has a problem that must be resolved first.

Also verify that OpenSearch Dashboards is accessible by opening http://<opensearch-host>:5601 in a browser and logging in with your admin credentials.

Key Outputs

• OpenSearch cluster URL: http://{{baseURL}}:9200

• OpenSearch Dashboards URL: http://{{baseURL}}:5601

• Admin credentials configured

• Cluster health: green or yellow

2. Configure OpenSearch and Opensearch dashboard

Follow this guide to create the audit logging index and dashboards in OpenSearch Dashboard

Index Configuration, Opensearch and Opensearch Dashboard Setup Guide

Key Outputs:

• Index created: audit_log_index

• Mappings configured

• Index status: ACTIVE

• Index pattern created: audit-logs-*

• Time field configured: timestamp

• Visualization created

• Fields discoverable in Dashboards

• Search functionality enabled

3. Configure Fluent Bit Output

Deploy fluentBit using this guide: Deploy Fluentbit

Connect Fluent Bit to send audit logs to OpenSearch.

• Configure Fluent Bit output plugin

• Set OpenSearch endpoint URL

• Configure authentication (if needed)

• Enable retry and buffering policies

update the OUTPUT section in helm-values/cx-fluent-bit-custom-values.yaml

[OUTPUT]
      Name  opensearch
      Match audit.admin
      Host  [opensearch host]
      Port  9200
      Index audit_log_index
      HTTP_User [opensearch user]
      HTTP_Passwd [opensearch password]
      Logstash_Format Off
      Replace_Dots On
      Suppress_Type_Name On
      Retry_Limit 5
      tls [opensearch tls verification]
      tls.verify Off

Field reference

Upgrade fluentBit:

helm upgrade --install --namespace ef-external --set global.efCxReleaseName="ef-cx"  cx-fluent-bit --debug --values helm-values/cx-fluent-bit-custom-values.yaml expertflow/fluent-bit

Key Outputs:

• Fluent Bit connected to OpenSearch

• Logs being received successfully

• Index naming pattern working

• No connection errors in Fluent Bit logs

4. Verify the Pipeline

Prerequisite: Steps 1 through 3 complete.

This step confirms that the full pipeline is working — from a user action in Expertflow CX through to a visible record in OpenSearch Dashboards.

4.1 Check That the Index Is Receiving Data

curl -X GET "http://<opensearch-host>:9200/audit_log_index/_count?pretty" \
  -u admin:<password>

Expected response:

{
  "count": 42,
  "_shards": { ... }
}

If count is greater than 0, logs are flowing into the index. If count is 0, wait two minutes and check again — Fluent Bit batches logs before forwarding. If count remains 0 after five minutes, proceed to the Troubleshooting section.

4.2 Verify Data Is Visible in Dashboards

1. Open http://<opensearch-host>:5601

2. Navigate to Discover

3. Select the audit_log_index* index pattern

4. Set the time range to Last 1 hour

You should see log entries appearing in the results panel. Each row represents one audit event. Click any row to expand the full field list.

4.3 Generate a Test Event

Perform any of the following actions in Expertflow CX using an admin account:

• Create a new team

• Update a reason code

• Delete a configuration record

Return to Dashboards and search for your username or the resource you modified. The event should appear within one to two minutes.

4.4 Confirm Field Mapping Is Correct

In Discover, expand any log entry and verify the following fields are present and populated:

• timestamp — shows a valid date

• user_id — shows a UUID

• action — shows CREATE, UPDATE, or DELETE

• resource — shows the entity name

• source_ip_address — shows an IP address

If any of these fields are missing or showing as unmapped, the index mapping was not applied correctly. Delete the index, re-run the creation script from Step 3.1, and re-ingest logs.

Troubleshooting

Work through these checks from top to bottom. Each level assumes the level above it is healthy.

Level 1 — Is OpenSearch healthy?

curl -X GET "http://<opensearch-host>:9200/_cluster/health?pretty" \
  -u admin:<password>

If status is red, OpenSearch has a problem. Do not investigate Fluent Bit or the index until the cluster is healthy.

Level 2 — Does the index exist?

curl -X GET "http://<opensearch-host>:9200/audit_log_index?pretty" \
  -u admin:<password>

A 404 response means the index creation script failed or was skipped in step 2.

Level 3 — Is Fluent Bit running?

kubectl get pods -n ef-external | grep fluent

If the pod is not in Running state, check its logs:

kubectl logs <fluent-bit-pod-name> -n ef-external --previous

Level 4 — Is Fluent Bit connecting to OpenSearch?

kubectl logs <fluent-bit-pod-name> -n ef-external | grep -i "opensearch\|error\|unauthorized\|refused"

Level 5 — Are the application pods emitting logs?

kubectl logs <app-pod-name> -n expertflow | grep "audit_logging"

If no results appear, the application is not emitting audit logs. This is an application-level issue, not a pipeline issue. Contact the development team.

Level 6 — Is Fluent Bit reading from the right pods?

kubectl logs <fluent-bit-pod-name> -n ef-external | grep "unified-admin"

If nothing appears, the Path in the Fluent Bit INPUT config may not match the actual pod log file path. The path pattern must match the pod name prefix in the expertflow namespace.