Accessing Logs from Kubernetes Pods

This guide is intended for Operations Managers and Webmasters. It explains how to access and view OTLP-formatted logs generated by applications running in Kubernetes pods. These logs can be viewed directly through the pod console. This is useful during initial setup, when OpenSearch is unavailable, or when investigating a specific pod in isolation.

Pod logs and OpenSearch are not separate systems. Fluent Bit reads from pod stdout and forwards to OpenSearch. Viewing pod logs directly gives you the raw source before any processing.

Prerequisites

Access to the Kubernetes cluster where the application is deployed.
Permissions to view pod logs

Steps

Identify the pod running the application.
```
kubectl get pods -n expertflow
```
View logs for a specific pod:
```
kubectl logs <pod-name> -n expertflow
```
Replace <pod-name> with the pod you want to inspect.

Filter logs (optional)

audit logs

kubectl logs <pod-name> -n <namespace> | grep "audit_logging"

tracing logs

kubectl logs <pod-name> -n <namespace>  | grep "tracing"

What a Valid Audit Log Entry Looks Like

{
  "timestamp": "2024-10-28T09:00:00Z",
  "user_id": "c7a904cc-262f-41f3-988a-351f6326e004",
  "user_name": "admin",
  "action": "UPDATE",
  "resource": "teams",
  "resource_id": "3e0b50a2-64fa-4051-8d16-3db6408fddec",
  "source_ip_address": "192.168.1.100",
  "attributes": {
    "service": "unified_admin",
    "tenantId": "expertflow",
    "updated_data": { "description": "updated value" }
  },
  "type": "audit_logging",
  "level": "info"
}

If you see entries matching this structure, the application is emitting logs correctly and Fluent Bit should be picking them up.