End-to-End Observability for Kubernetes Microservices with OpenTelemetry Protocol (OTLP): A Practical Guide

A complete walkthrough — from the basics of OTLP to production-grade implementation, including advanced patterns like tail sampling, mTLS, and multi-cluster federation.

🚨 HIRING: Tech Talent 💰 $50–$120/hr | 🔥 Multiple Roles

Frontend • Backend • Full Stack • Mobile • AI/ML • DevOps 👉 **Apply Here**

Table of Contents

1. What is OpenTelemetry?
2. Understanding OTLP — The Protocol Behind OpenTelemetry
3. Core Use Cases
4. Architecture: How the Pieces Fit Together
5. Practical Implementation on Kubernetes (Step-by-Step)
6. Application Instrumentation
7. Wiring Up Backends (Jaeger, Prometheus, Loki, Tempo)
8. Advanced Topics
9. Best Practices and Common Pitfalls
10. Conclusion

1. Why Observability Matters in Microservices

A monolithic application is relatively easy to debug — you’ve got one log file, one process, one stack trace. The moment you decompose that monolith into 40 microservices spread across a Kubernetes cluster, a single user request can hop through a dozen pods, traverse two service meshes, hit three databases, and emit logs in five different formats.

When something goes wrong (and it will), the question becomes: where exactly did the request slow down or fail, and why?

Observability — the ability to ask arbitrary questions about your system’s behavior without having to ship new code — rests on three pillars:

• Metrics — numeric measurements aggregated over time (request rate, CPU usage, p99 latency).
• Logs — discrete, timestamped event records.
• Traces — the lifecycle of a single request as it propagates through services, broken into spans.

Historically, each pillar had its own protocol, agent, and vendor lock-in. You’d run a Prometheus exporter for metrics, Fluent Bit for logs, and Jaeger or Zipkin clients for traces. Three agents, three configs, three sets of bugs.

OpenTelemetry was created to fix exactly this fragmentation.

2. What is OpenTelemetry?

OpenTelemetry (OTel) is a CNCF project (graduated for tracing, stable for metrics and logs) that provides:

• A specification for telemetry data — what a span, metric, or log looks like semantically.
• SDKs and APIs in every major language (Go, Java, Python, .NET, Node.js, Rust, etc.) for emitting telemetry.
• The OpenTelemetry Collector — a vendor-agnostic agent/gateway that receives, processes, and exports telemetry.
• OTLP — a single wire protocol that carries traces, metrics, and logs.

The big idea: instrument once, export anywhere. Your application doesn’t care whether telemetry ends up in Datadog, Honeycomb, Grafana Cloud, or a self-hosted Tempo + Prometheus + Loki stack. Switching backends becomes a Collector config change.

3. Understanding OTLP — The Protocol Behind OpenTelemetry

OTLP (OpenTelemetry Protocol) is the native protocol of OpenTelemetry. It defines how telemetry data is encoded and transmitted between OTel components.

Key Characteristics

• Single protocol for all three signals — traces, metrics, and logs all flow over OTLP.
• Protocol Buffers (protobuf) for efficient binary encoding.
• Two transports:

OTLP/gRPC on port 4317 — high performance, streaming, recommended for production.

OTLP/HTTP on port 4318 — easier to debug, works through restrictive proxies, supports both protobuf and JSON payloads.

Why OTLP Matters

Before OTLP, each backend defined its own ingestion format. Switching from Jaeger to Tempo meant re-instrumenting or running format-translation proxies. With OTLP, the entire pipeline — SDK → Collector → backend — speaks the same language. Most observability vendors now accept OTLP natively.

OTLP Data Model (Simplified)

Resource
  ├── service.name = "checkout-api"
  ├── k8s.pod.name = "checkout-api-7d9c..."
  └── deployment.environment = "prod"
       │
       ├── Traces → ResourceSpans → ScopeSpans → Span (trace_id, span_id, attributes...)
       ├── Metrics → ResourceMetrics → ScopeMetrics → Metric (counter, gauge, histogram...)
       └── Logs → ResourceLogs → ScopeLogs → LogRecord (severity, body, attributes...)

Every piece of telemetry is tagged with a Resource — a set of attributes describing where it came from (service, pod, region, version). This is what makes correlation possible: a trace, a log, and a metric from the same pod all share the same resource attributes, so you can pivot between them in a UI.

4. Core Use Cases

a. Distributed Tracing Across Services

Track a request from the ingress controller, through the API gateway, into the order service, payment service, and database — all stitched together by a propagated trace context (traceparent header).

b. SLO Monitoring

Emit RED metrics (Rate, Errors, Duration) automatically from instrumentation libraries and compare against your service level objectives.

c. Root Cause Analysis

When latency spikes, jump from a metric anomaly → the slow traces → the exact spans → the correlated logs — all without leaving the same UI.

d. Vendor-Neutral Pipelines

Send the same data to multiple backends simultaneously: Prometheus for short-term metrics, a long-term store (Thanos/Mimir) for retention, and a vendor like Honeycomb for advanced analysis.

e. Cost Optimization

Sample aggressively at the Collector level. Drop unnecessary attributes. Aggregate metrics before they hit your paid backend.

f. Security and Compliance

Strip PII at the Collector layer using attribute processors before telemetry leaves your cluster.

5. Architecture: How the Pieces Fit Together

A production-grade OTel pipeline on Kubernetes typically looks like this:

┌────────────────────────────────────────────────────────────────┐
│                    Kubernetes Cluster                          │
│                                                                │
│  ┌──────────────┐   ┌──────────────┐   ┌──────────────┐        │
│  │  App Pod 1   │   │  App Pod 2   │   │  App Pod N   │        │
│  │ (OTel SDK)   │   │ (OTel SDK)   │   │ (OTel SDK)   │        │
│  └──────┬───────┘   └──────┬───────┘   └──────┬───────┘        │
│         │ OTLP/gRPC        │                  │                │
│         └──────────────────┼──────────────────┘                │
│                            │                                   │
│                  ┌─────────▼──────────┐                        │
│                  │  Agent Collector   │  (DaemonSet)           │
│                  │  - receives        │                        │
│                  │  - batches         │                        │
│                  │  - adds k8s attrs  │                        │
│                  └─────────┬──────────┘                        │
│                            │ OTLP                              │
│                  ┌─────────▼──────────┐                        │
│                  │ Gateway Collector  │  (Deployment + HPA)    │
│                  │  - tail sampling   │                        │
│                  │  - PII scrubbing   │                        │
│                  │  - routing         │                        │
│                  └─────────┬──────────┘                        │
└────────────────────────────┼───────────────────────────────────┘
                             │
            ┌────────────────┼────────────────┐
            │                │                │
       ┌────▼─────┐    ┌─────▼─────┐    ┌─────▼─────┐
       │  Tempo   │    │Prometheus │    │   Loki    │
       │ (traces) │    │ (metrics) │    │  (logs)   │
       └──────────┘    └───────────┘    └───────────┘
                             │
                       ┌─────▼─────┐
                       │  Grafana  │
                       └───────────┘

Why Two Layers of Collectors?

• Agent (DaemonSet) — runs on every node, close to the workloads. Handles fast local ingestion, enriches data with node/pod metadata, and offloads the application as quickly as possible.
• Gateway (Deployment) — a smaller fleet of pods that aggregates data from all agents. This is where you do expensive work like tail-based sampling (which needs to see all spans of a trace) and routing to multiple backends.

For small clusters, a single Collector tier is fine. For production at scale, the two-tier pattern is the standard.

6. Practical Implementation on Kubernetes (Step-by-Step)

Let’s build this from scratch. I’ll assume you have a working Kubernetes cluster (kubectl access) and helm installed.

Step 1: Create a Dedicated Namespace

kubectl create namespace observability

Step 2: Install cert-manager (Prerequisite for the OTel Operator)

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.14.0/cert-manager.yaml
kubectl wait --for=condition=Available --timeout=300s \
  deployment --all -n cert-manager

Step 3: Install the OpenTelemetry Operator

The Operator manages Collectors and auto-instrumentation declaratively via CRDs.

kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml
kubectl wait --for=condition=Available --timeout=300s \
  deployment/opentelemetry-operator-controller-manager -n opentelemetry-operator-system

Step 4: Deploy the Agent Collector (DaemonSet)

Create otel-agent.yaml:

apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  name: otel-agent
  namespace: observability
spec:
  mode: daemonset
  hostNetwork: false
  serviceAccount: otel-agent
  env:
    - name: K8S_NODE_NAME
      valueFrom:
        fieldRef:
          fieldPath: spec.nodeName
    - name: K8S_POD_IP
      valueFrom:
        fieldRef:
          fieldPath: status.podIP
  config:
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
          http:
            endpoint: 0.0.0.0:4318
      hostmetrics:
        collection_interval: 30s
        scrapers:
          cpu: {}
          memory: {}
          disk: {}
          filesystem: {}
          network: {}
      kubeletstats:
        collection_interval: 30s
        auth_type: serviceAccount
        endpoint: "${env:K8S_NODE_NAME}:10250"
        insecure_skip_verify: true

    processors:
      batch:
        send_batch_size: 8192
        timeout: 5s
      memory_limiter:
        check_interval: 1s
        limit_percentage: 80
        spike_limit_percentage: 25
      k8sattributes:
        auth_type: serviceAccount
        passthrough: false
        extract:
          metadata:
            - k8s.pod.name
            - k8s.pod.uid
            - k8s.deployment.name
            - k8s.namespace.name
            - k8s.node.name
            - k8s.cluster.uid
        pod_association:
          - sources:
              - from: resource_attribute
                name: k8s.pod.ip
          - sources:
              - from: connection
      resourcedetection:
        detectors: [env, system, k8snode]
        timeout: 5s

    exporters:
      otlp:
        endpoint: otel-gateway-collector.observability.svc.cluster.local:4317
        tls:
          insecure: true
        sending_queue:
          enabled: true
          num_consumers: 4
          queue_size: 1000
        retry_on_failure:
          enabled: true
          initial_interval: 5s
          max_interval: 30s

    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: [memory_limiter, k8sattributes, resourcedetection, batch]
          exporters: [otlp]
        metrics:
          receivers: [otlp, hostmetrics, kubeletstats]
          processors: [memory_limiter, k8sattributes, resourcedetection, batch]
          exporters: [otlp]
        logs:
          receivers: [otlp]
          processors: [memory_limiter, k8sattributes, resourcedetection, batch]
          exporters: [otlp]

You’ll also need an RBAC-enabled ServiceAccount so the agent can read pod metadata:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: otel-agent
  namespace: observability
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: otel-agent
rules:
  - apiGroups: [""]
    resources: ["pods", "namespaces", "nodes", "nodes/stats", "nodes/proxy"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["apps"]
    resources: ["replicasets", "deployments", "daemonsets", "statefulsets"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: otel-agent
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: otel-agent
subjects:
  - kind: ServiceAccount
    name: otel-agent
    namespace: observability

Apply both:

kubectl apply -f rbac.yaml
kubectl apply -f otel-agent.yaml

Step 5: Deploy the Gateway Collector (Deployment)

Create otel-gateway.yaml:

apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  name: otel-gateway
  namespace: observability
spec:
  mode: deployment
  replicas: 3
  resources:
    limits:
      cpu: 1000m
      memory: 2Gi
    requests:
      cpu: 200m
      memory: 512Mi
  config:
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
          http:
            endpoint: 0.0.0.0:4318

    processors:
      batch:
        send_batch_size: 8192
        timeout: 5s
      memory_limiter:
        check_interval: 1s
        limit_percentage: 80
        spike_limit_percentage: 25
      # Tail sampling: keep all errors + slow traces, sample the rest
      tail_sampling:
        decision_wait: 10s
        num_traces: 100000
        expected_new_traces_per_sec: 1000
        policies:
          - name: errors-policy
            type: status_code
            status_code: { status_codes: [ERROR] }
          - name: slow-traces-policy
            type: latency
            latency: { threshold_ms: 500 }
          - name: probabilistic-policy
            type: probabilistic
            probabilistic: { sampling_percentage: 10 }
      # Scrub PII
      attributes/scrub:
        actions:
          - key: user.email
            action: delete
          - key: http.request.header.authorization
            action: delete

    exporters:
      otlp/tempo:
        endpoint: tempo.observability.svc.cluster.local:4317
        tls: { insecure: true }
      prometheusremotewrite:
        endpoint: http://prometheus.observability.svc.cluster.local:9090/api/v1/write
        resource_to_telemetry_conversion: { enabled: true }
      otlphttp/loki:
        endpoint: http://loki.observability.svc.cluster.local:3100/otlp

    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: [memory_limiter, attributes/scrub, tail_sampling, batch]
          exporters: [otlp/tempo]
        metrics:
          receivers: [otlp]
          processors: [memory_limiter, batch]
          exporters: [prometheusremotewrite]
        logs:
          receivers: [otlp]
          processors: [memory_limiter, attributes/scrub, batch]
          exporters: [otlphttp/loki]

Apply:

kubectl apply -f otel-gateway.yaml

Step 6: Verify the Pipeline

kubectl get pods -n observability
kubectl logs -n observability -l app.kubernetes.io/name=otel-gateway-collector

You should see the Collector reporting which receivers, processors, and exporters are running.

7. Application Instrumentation

You have two paths: auto-instrumentation (zero code changes for supported languages) and manual instrumentation (full control).

Option A: Auto-Instrumentation via the OTel Operator

Create an Instrumentation custom resource that targets your apps:

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: default-instrumentation
  namespace: my-app
spec:
  exporter:
    endpoint: http://otel-agent-collector.observability.svc.cluster.local:4317
  propagators:
    - tracecontext
    - baggage
    - b3
  sampler:
    type: parentbased_traceidratio
    argument: "1.0"
  java:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:latest
  python:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-python:latest
  nodejs:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-nodejs:latest
  dotnet:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-dotnet:latest
  go:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-go:latest

Then add an annotation to your application Pod template:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout-api
  namespace: my-app
spec:
  template:
    metadata:
      annotations:
        instrumentation.opentelemetry.io/inject-java: "true"
        # or inject-python, inject-nodejs, inject-dotnet, inject-go
    spec:
      containers:
        - name: app
          image: my-registry/checkout-api:1.2.0
          env:
            - name: OTEL_SERVICE_NAME
              value: "checkout-api"
            - name: OTEL_RESOURCE_ATTRIBUTES
              value: "deployment.environment=production,service.version=1.2.0"

The Operator injects an init container that adds the language agent and sets the right env vars (OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_TRACES_SAMPLER, etc.). No application code changes required.

Option B: Manual Instrumentation (Example in Python)

from opentelemetry import trace, metrics
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.flask import FlaskInstrumentor
from opentelemetry.instrumentation.requests import RequestsInstrumentor

resource = Resource.create({
    "service.name": "checkout-api",
    "service.version": "1.2.0",
    "deployment.environment": "production",
})

provider = TracerProvider(resource=resource)
exporter = OTLPSpanExporter(endpoint="otel-agent-collector.observability:4317", insecure=True)
provider.add_span_processor(BatchSpanProcessor(exporter))
trace.set_tracer_provider(provider)

# Auto-instrument popular libraries
FlaskInstrumentor().instrument()
RequestsInstrumentor().instrument()

tracer = trace.get_tracer(__name__)

@app.route("/checkout")
def checkout():
    with tracer.start_as_current_span("validate_cart") as span:
        span.set_attribute("cart.size", len(cart))
        # business logic
    return "ok"

The same pattern works in every language SDK — only the imports change.

Context Propagation

For distributed tracing to work, the traceparent header (W3C Trace Context) must be propagated across service calls. Auto-instrumentation handles this. If you use raw HTTP clients or custom queue producers, you need to inject and extract context manually.

8. Wiring Up Backends

A practical, fully open-source stack:

• Traces → Grafana Tempo
• Metrics → Prometheus (or Mimir for horizontal scale)
• Logs → Grafana Loki
• Visualization → Grafana

Install with Helm:

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

helm install tempo grafana/tempo -n observability \
  --set tempo.receivers.otlp.protocols.grpc.endpoint=0.0.0.0:4317

helm install loki grafana/loki -n observability \
  --set loki.auth_enabled=false

helm install prometheus prometheus-community/prometheus -n observability \
  --set server.extraFlags={web.enable-remote-write-receiver}

helm install grafana grafana/grafana -n observability \
  --set adminPassword=admin

In Grafana, add three data sources pointing at tempo:3200, prometheus-server:80, and loki:3100. Enable trace-to-logs and trace-to-metrics correlation in the Tempo data source — this is the magic that lets you click a slow span and instantly see the logs from that pod at that exact moment.

9. Advanced Topics

9.1 Sampling Strategies

Head-based sampling decides at the start of a trace whether to keep it. Cheap, but you might miss the very errors you care about.

Tail-based sampling waits until the trace is complete, then decides based on the full picture: “keep all traces with errors, all traces above 500ms, and 10% of the rest.” This is what the gateway Collector config above does. The tradeoff: the gateway has to buffer spans long enough to see the whole trace, which costs memory.

A common production policy:

policies:
  - name: errors
    type: status_code
    status_code: { status_codes: [ERROR] }
  - name: slow
    type: latency
    latency: { threshold_ms: 1000 }
  - name: high-value-endpoint
    type: string_attribute
    string_attribute:
      key: http.route
      values: [/api/checkout, /api/payment]
  - name: baseline
    type: probabilistic
    probabilistic: { sampling_percentage: 5 }

9.2 Securing the Pipeline with mTLS

In a multi-tenant or regulated environment, OTLP traffic between agents and the gateway should be mutually authenticated. Use cert-manager to issue certificates and configure the Collector:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
        tls:
          cert_file: /certs/tls.crt
          key_file: /certs/tls.key
          client_ca_file: /certs/ca.crt
exporters:
  otlp:
    endpoint: otel-gateway.observability:4317
    tls:
      cert_file: /certs/tls.crt
      key_file: /certs/tls.key
      ca_file: /certs/ca.crt

Mount the certs as a Kubernetes Secret.

9.3 Multi-Cluster and Multi-Region Federation

Pattern: each cluster runs its own agent + local gateway. A central, regional gateway aggregates from multiple clusters and forwards to the backend. Use the loadbalancing exporter on the upstream gateway so that all spans of a trace land on the same downstream collector — essential for tail sampling to work correctly across clusters.

exporters:
  loadbalancing:
    routing_key: traceID
    protocol:
      otlp: { tls: { insecure: false } }
    resolver:
      dns:
        hostname: otel-gateway-tail.observability.svc.cluster.local

9.4 Cardinality Management

The number-one cost driver in observability is high-cardinality metrics — labels like user_id, request_id, or pod_uid that explode time-series counts. Use the transform and filter processors to drop or hash dangerous labels before export:

processors:
  transform/metrics:
    metric_statements:
      - context: datapoint
        statements:
          - delete_key(attributes, "user_id")
          - replace_pattern(attributes["url.path"], "/users/\\d+", "/users/:id")

9.5 Logs as a First-Class Signal

OTel’s logs support is now stable. The trick is correlation: when you log within a span, the SDK auto-attaches trace_id and span_id to the log record. In Grafana, this means a single click moves you from a span to the exact log lines emitted during it. Configure your logging framework (Logback, Winston, structlog, Serilog) to use the OTel log bridge.

9.6 Profiling — the Emerging Fourth Pillar

OpenTelemetry is actively standardizing continuous profiling as a fourth signal. Pyroscope and Parca already accept OTel-compatible profiles. Expect this to mature into a first-class part of OTLP in the next year or two.

9.7 Performance Tuning the Collector

• Use batch processor — never export individual spans.
• Tune send_batch_size and timeout based on observed throughput.
• Always include memory_limiter as the first processor in every pipeline; it's your circuit breaker.
• Run the Collector with GOMEMLIMIT set just under the container memory limit.
• Use Horizontal Pod Autoscaler on the gateway tier based on CPU and queue depth.
• For very high throughput, enable the file_storage extension so retries survive Collector restarts.

10. Best Practices and Common Pitfalls

Do:

• Follow OTel Semantic Conventions religiously. Use http.request.method, not http_method. Use service.name, not app. This is what makes dashboards portable across teams and tools.
• Always set service.name, service.version, and deployment.environment on every workload.
• Treat the Collector config as production code — version it, review it, test it in staging.
• Sample early at the SDK for cost, then tail-sample at the gateway for fidelity.
• Set resource requests and limits on Collectors based on real measurements, not guesses.

Don’t:

• Don’t put high-cardinality data (user IDs, full URLs with IDs, request bodies) into span attributes or metric labels without thinking through the cost.
• Don’t send telemetry directly from apps to vendor endpoints — always go through a Collector. This decouples your apps from your backend choice.
• Don’t enable every auto-instrumentation library blindly; some create thousands of low-value spans per request.
• Don’t forget to monitor the Collector itself. Scrape its :8888/metrics endpoint and alert on dropped spans, queue size, and refused data points.

11. Conclusion

OpenTelemetry — and OTLP as its lingua franca — has done for observability what Kubernetes did for orchestration: it turned a fragmented, vendor-locked landscape into a standardized, portable foundation. You instrument your apps once. You run a Collector pipeline that’s the same in every cluster. You swap backends without touching application code.

For a microservices platform on Kubernetes, the payoff is concrete:

• A single agent on every node instead of three.
• Traces, metrics, and logs correlated by shared resource attributes.
• A pipeline you can extend with sampling, scrubbing, and routing without redeploying apps.
• Freedom to use Tempo today, switch to a commercial vendor tomorrow, and run both in parallel during the transition.

Start small — one cluster, one Collector, one auto-instrumented service. Get the trace-to-log correlation working in Grafana. From there, scale out to the two-tier pattern, layer in tail sampling, and harden the pipeline with mTLS and cardinality controls.

The investment compounds. Every new service you ship arrives in production already observable. Every incident gets shorter because you can answer “where did it slow down?” in seconds instead of hours. That’s the real value of OTLP — not the protocol itself, but everything it makes possible on top of it.

If you found this useful, feel free to share and connect. Happy to discuss specific implementation challenges in the comments.

Tags: #OpenTelemetry #Kubernetes #Observability #DevOps #SRE #Microservices #CloudNative #Monitoring #OTLP #CNCF

Thank you for being a part of the community

Before you go:

👉 Be sure to clap and follow the writer ️👏️️

👉 Follow us: **Linkedin| [Medium](https://medium.com/codetodeploy)**

👉 CodeToDeploy Tech Community is live on Discord — **Join now!**

Disclosure: This post includes affiliate and partnership links.