Multi-Tenant Observability on Amazon EKS with Fluent Bit & OpenTelemetry Collector

Kubernetes clusters in modern enterprises often run applications for multiple teams or tenants. This creates a key challenge: How can we build an observability pipeline that keeps data isolated per tenant while staying scalable, cost-effective, and secure?

This article explains how to design a multi-tenant observability pipeline on Amazon EKS using Fluent Bit for collecting logs and the OpenTelemetry Collector for metrics and traces. We’ll also cover security best practices, performance tuning, and cost optimization with production-ready examples.

Overview

In a multi-tenant EKS environment, observability data (logs, metrics, traces) needs to be:

1. Isolated per tenant for compliance, security, and privacy.
2. Processed efficiently to manage costs, especially when dealing with high-cardinality metrics.
3. Consistently structured across environments to enable reliable monitoring and alerting.

Without careful design, these requirements can lead to operational complexity. Let’s look at how to build a robust solution.

Architecture Overview

Key Components:

• Fluent Bit: Collect logs and add tenant metadata.
• OpenTelemetry Collector: Collect metrics and traces; run both on each node (agent mode) and centrally (gateway mode).
• Storage: OpenSearch, Amazon Managed Prometheus (AMP), AWS X-Ray, Grafana Mimir.

Tenant Identification

You can identify tenants via:

• Pod Labels: like tenant_id or team.
• Namespace names.
• Workload annotations.
• SPIFFE IDs issued by SPIRE.

Example Fluent Bit config to parse Kubernetes metadata:

[FILTER]
    Name kubernetes
    Match kube.*
    Labels On
    Annotations On

Security Best Practices

Certificate Rotation for mTLS

To automate mTLS certificate renewal, integrate cert-manager:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: metric-collector-cert
spec:
  secretName: metric-collector-tls
  issuerRef:
    name: spire-ca-issuer
    kind: ClusterIssuer

Identity with SPIFFE/SPIRE

SPIRE can issue SPIFFE IDs for workload identity. Though OTel Collector lacks direct SPIFFE support, you can use:

• Sidecar proxies (e.g., Envoy) with mTLS.
• Mutating Webhooks to enforce telemetry traffic identity.

Example SPIFFE ID:

spiffe://example.org/ns/tenant-a/pod/my-app

Rate Limiting and Resource Quotas

While OTel Collector lacks a native rate limiter, you can control usage via:

• tail_sampling: Reduce trace volume.
• memory_limiter + batch processors: Manage memory and batch size.
• Kubernetes ResourceQuota: To limit resources per tenant.

Example ResourceQuota:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: tenant-a-quota
  namespace: tenant-a
spec:
  hard:
    cpu: "4"
    memory: 8Gi
    pods: "10"

Network Policies for Isolation

kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: tenant-a-isolation
  namespace: tenant-a
spec:
  podSelector: {}
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          tenant: tenant-a

Performance Tuning

Fluent Bit Settings

[INPUT]
    Name tail
    Path /var/log/containers/*.log
    Buffer_Chunk_Size 2M
    Buffer_Max_Size 8M
    Refresh_Interval 5

Scaling the OTel Collector

Example HPA config:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
spec:
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70

Sampling Processor Example

processors:
  tail_sampling:
    decision_wait: 10s
    policies:
      - name: tenantA_policy
        spans_per_second: 50

Bonus: Parny for Incident Management

For managing incidents in a multi-tenant environment, Parny can be integrated into this observability pipeline. Parny collects alerts from systems like Prometheus Alertmanager, OpenSearch alerting, and traces errors flagged via OTel.

Integration Highlights:

• Alerts are tagged with tenant context (tenant_id or namespace).
• Parny applies escalation policies, routing incidents to the correct team.
• Supports integrations with PagerDuty, Slack, and Microsoft Teams.
• Helps track SLAs per tenant.

By integrating Parny, the pipeline not only monitors but also automates the response process tailored to each tenant’s requirements. For more details: https://parny.io/

Conclusion

By combining AWS native services, Fluent Bit, and OpenTelemetry Collector, enhanced with SPIFFE/SPIRE, certificate rotation, and Kubernetes controls, you can build a secure, scalable, and cost-effective multi-tenant observability pipeline.