Observability for LLM Applications: What to Log, What to Monitor, and Why

Your LLM application is failing silently right now. You just don’t know it yet. Here’s what you need to see before it’s too late.

Photo by Damian Zaleski on Unsplash

The email came in at 9:47 PM on a Wednesday.

A customer’s support team reported that their AI chatbot had started giving completely wrong answers. Not hallucinating in the obvious “confidently making things up” way. Just… subtly wrong. Answers that sounded reasonable but contained factual errors. Products listed that don’t exist. Pricing that was off by thousands.

The customer had been deploying the chatbot to production for two weeks. For two weeks, these errors were being served to users. For two weeks, nobody noticed because nobody was watching.

Here’s what made it worse: the team had logs. They had basic application logs. They had the timestamps of every API call. They could see that requests came in and responses went out. What they didn’t have was the data that mattered. They couldn’t correlate the subtle errors to changes in prompt structure. They couldn’t see which model version was deployed during which time window. They couldn’t tell if it was the retrieval that broke or the generation or the post-processing logic.

They had observability infrastructure. They didn’t have observability for LLMs.

This distinction matters because LLM applications fail differently than traditional software. Your API endpoint returns a 500 error and your monitoring catches it instantly. Your LLM endpoint returns a 200 with a plausible-sounding but completely wrong answer and you need to be watching very specifically to notice.

That’s what we’re going to talk about.

Why LLM Observability Is Not Just Logging

Before we get into what to log, let’s be clear about what the problem actually is.

Traditional application monitoring tracks whether things worked. HTTP 200 vs 500. Response time under 100ms or over 1000ms. Error rates. Availability. These are binary or easily measurable categories.

LLM applications don’t work that way. Your chatbot returns a 200 in 342ms. Response time is excellent. The API worked perfectly. But the response contains incorrect information. It’s factually wrong. It’s not what the user asked for. It violated a safety guardrail you thought you had.

Traditional monitoring would have shown green across the board.

This is the problem that observability solves. Observability is not just logging everything. It’s logging the right things, in the right way, so that when something goes wrong you can answer questions you didn’t think to ask in advance.

A good observability stack for LLMs answers questions like:

Did the quality of responses degrade this week? If yes, when did it start and what changed?

Which prompts consistently produce lower quality outputs?

Are certain user segments experiencing worse answers than others?

Is a specific retrieval source producing bad context that downstream generation can’t recover from?

Did this model version update cause a regression?

Is my cost per request drifting upward and if so, why?

Traditional logging doesn’t help with any of these. It just records that a request happened. Observability helps you reason about whether the request produced good outcomes.

The Three Layers of LLM Observability

LLM observability sits on three layers. Most teams obsess over the first and ignore the other two.

Layer 1: Operational Metrics (The Easy Part)

This is what most teams do first because it’s straightforward to instrument.

Latency: How long does an LLM call take from request to response? Track both the time to first token and time to last token. They’re different and both matter. First token latency affects perceived responsiveness. Total latency affects throughput.

Token usage: How many tokens does a request consume? Separate prompt tokens from completion tokens. Track input length and output length separately. This is essential for cost projection.

Cost: What is the total spend per request? Aggregate to per user, per feature, per day. Track cost per token, cost per unique user, cost as percentage of revenue.

Error rates: How often do API calls fail? Distinguish between network errors (retryable), rate limit errors (need backoff), and actual failures (bad credentials).

Model and parameter tracking: Which model version is deployed? What’s the temperature, top_p, max_tokens for this request?

This layer is operational because it answers “is my infrastructure working?” These are the alerts that wake you up at 2 AM.

Layer 2: Quality Metrics (The Overlooked Part)

This is what separates teams that know when things are broken from teams that find out from customer complaints.

Hallucination detection: Is the model inventing facts, or is it grounded in the provided context? For RAG applications, this means checking whether generated claims are actually present in the retrieved documents.

Relevance to user intent: Did the response actually answer the user’s question, or did it answer something adjacent?

Safety and guardrail violations: Did the response violate your content policies? Did it refuse appropriately when it should have?

Format compliance: If you asked for JSON, did you get valid JSON? If you asked for a list of three items, did you get exactly three?

Factual accuracy: For domains where ground truth exists (product data, pricing, policy documents), is the response factually correct?

Here’s the problem with quality metrics: they’re expensive to compute at scale. You can’t afford to send every response to a human evaluator. You need automation, which means you need either LLM-as-a-judge scoring, retrieval grounding checks, or human-annotated golden datasets you compare against.

<cite index=”1–1">LLM observability tools track these quality metrics: hallucination rate, toxicity/bias indicators, user satisfaction scores, groundedness, relevance to the prompt, and latency throughput.</cite>

Layer 3: Root Cause Metrics (The Strategic Part)

This layer tells you why something went wrong so you can fix it, not just detect it.

Retrieval quality: In RAG systems, did the retrieval step find relevant documents? You need to track recall (did you find the right docs?) and precision (were the docs actually useful?).

Prompt effectiveness: Which prompts consistently produce higher quality outputs? Is it a few good examples vs many? Is it explicit reasoning steps vs none?

Context window utilization: Are you sending unnecessary tokens to the model? Are certain requests wasting your budget on padded context?

Tool call accuracy: If your agent has access to tools, is it calling the right tool for the task? Is it passing the right arguments?

Intermediate step quality: For multi-step agents, where in the pipeline does quality degrade? Is it the planning step, the tool selection, or the synthesis step?

This layer requires tracing. Not just logging that something happened, but capturing the intermediate steps so you can see where the chain broke.

What to Actually Log: A Complete Implementation

Here’s a production-grade logging implementation for an LLM application. This is built for traceability, not just event recording.

import json
import uuid
from datetime import datetime
from typing import Any, Dict, List, Optional
import logging
from dataclasses import dataclass, asdict
import hashlib

# Set up structured logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
@dataclass
class LLMInteraction:
    """Complete record of an LLM call and its context"""

    # Identity
    trace_id: str  # Unique ID for entire user request
    span_id: str   # Unique ID for this specific LLM call
    parent_span_id: Optional[str] = None  # For chained calls

    # Request context
    user_id: str = None
    session_id: str = None
    feature_name: str = None  # Which feature is using this LLM call

    # LLM call details
    model: str = None
    temperature: float = None
    top_p: float = None
    max_tokens: int = None

    # Input
    system_prompt_hash: str = None  # Hash, not full prompt for privacy
    user_message_length: int = None
    retrieved_docs_count: int = None
    total_input_tokens: int = None

    # Output
    response_text_length: int = None
    completion_tokens: int = None
    total_tokens: int = None
    time_to_first_token_ms: int = None
    total_latency_ms: int = None

    # Cost
    cost_usd: float = None

    # Quality signals
    response_format_valid: bool = None  # Did it return valid JSON if requested?
    guardrail_violations: List[str] = None  # Which guardrails triggered?
    hallucination_detected: bool = None
    relevance_score: Optional[float] = None  # 0 to 1, from eval model

    # Error handling
    error_occurred: bool = False
    error_type: Optional[str] = None
    error_message: Optional[str] = None
    retry_count: int = 0

    # Metadata
    timestamp: str = None
    environment: str = None  # prod, staging, dev
    model_version: str = None

    def log_to_structured_logging(self):
        """Send to structured logging backend"""
        record = asdict(self)

        # Remove None values for cleaner logs
        record = {k: v for k, v in record.items() if v is not None}

        logger.info(json.dumps(record))
class LLMObserver:
    """Wraps LLM calls to automatically capture observability data"""

    def __init__(self, api_client, environment="prod"):
        self.api_client = api_client
        self.environment = environment

    def call_llm(
        self,
        user_id: str,
        feature_name: str,
        system_prompt: str,
        user_message: str,
        model: str = "gpt-4o",
        temperature: float = 0.7,
        top_p: float = 1.0,
        max_tokens: int = 1024,
        retrieved_docs: List[str] = None,
        trace_id: str = None,
        parent_span_id: str = None,
    ):
        """
        Call LLM with automatic observability
        """

        # Generate IDs for tracing
        trace_id = trace_id or str(uuid.uuid4())
        span_id = str(uuid.uuid4())

        # Create interaction record
        interaction = LLMInteraction(
            trace_id=trace_id,
            span_id=span_id,
            parent_span_id=parent_span_id,
            user_id=user_id,
            feature_name=feature_name,
            model=model,
            temperature=temperature,
            top_p=top_p,
            max_tokens=max_tokens,
            system_prompt_hash=hashlib.sha256(system_prompt.encode()).hexdigest(),
            user_message_length=len(user_message),
            retrieved_docs_count=len(retrieved_docs) if retrieved_docs else 0,
            timestamp=datetime.utcnow().isoformat(),
            environment=self.environment,
        )

        # Make the LLM call with timing
        start_time = datetime.utcnow()
        first_token_time = None

        try:
            response = self.api_client.chat.completions.create(
                model=model,
                messages=[
                    {"role": "system", "content": system_prompt},
                    {"role": "user", "content": user_message}
                ],
                temperature=temperature,
                top_p=top_p,
                max_tokens=max_tokens,
                stream=False,
            )

            # Extract results
            generated_text = response.choices[0].message.content
            completion_tokens = response.usage.completion_tokens
            prompt_tokens = response.usage.prompt_tokens
            total_tokens = response.usage.total_tokens

            end_time = datetime.utcnow()
            total_latency_ms = int((end_time - start_time).total_seconds() * 1000)

            # Update interaction record with results
            interaction.response_text_length = len(generated_text)
            interaction.completion_tokens = completion_tokens
            interaction.total_input_tokens = prompt_tokens
            interaction.total_tokens = total_tokens
            interaction.total_latency_ms = total_latency_ms
            interaction.error_occurred = False

            # Calculate cost (pricing varies by model)
            interaction.cost_usd = self._calculate_cost(
                model,
                prompt_tokens,
                completion_tokens
            )

            # Quality checks
            interaction.response_format_valid = self._check_format_valid(
                generated_text
            )
            interaction.hallucination_detected = self._check_hallucination(
                generated_text,
                retrieved_docs
            )
            interaction.relevance_score = self._score_relevance(
                generated_text,
                user_message
            )

            # Log the interaction
            interaction.log_to_structured_logging()

            return generated_text, span_id

        except Exception as e:
            interaction.error_occurred = True
            interaction.error_type = type(e).__name__
            interaction.error_message = str(e)
            interaction.log_to_structured_logging()
            raise

    def _calculate_cost(self, model: str, prompt_tokens: int, completion_tokens: int) -> float:
        """Calculate cost based on model pricing"""
        pricing = {
            "gpt-4o": {"prompt": 0.000005, "completion": 0.000015},
            "gpt-4-turbo": {"prompt": 0.00001, "completion": 0.00003},
            "gpt-3.5-turbo": {"prompt": 0.0000005, "completion": 0.0000015},
        }

        rates = pricing.get(model, pricing["gpt-3.5-turbo"])
        return (prompt_tokens * rates["prompt"]) + (completion_tokens * rates["completion"])

    def _check_format_valid(self, response: str) -> bool:
        """Check if response matches expected format"""
        if response.startswith("{") and response.endswith("}"):
            try:
                json.loads(response)
                return True
            except:
                return False
        return True  # If not expecting JSON, consider it valid

    def _check_hallucination(self, response: str, context_docs: List[str]) -> bool:
        """Simple hallucination check: are claims grounded in context?"""
        if not context_docs:
            return False  # Can't detect without context

        # This is a simplified check. In production, use LLM-as-judge
        combined_context = " ".join(context_docs)

        # Very basic: check if key phrases from response appear in context
        response_words = set(response.lower().split())
        context_words = set(combined_context.lower().split())

        overlap = len(response_words & context_words) / len(response_words)

        # If less than 40% of words are in context, likely hallucinating
        return overlap < 0.4

    def _score_relevance(self, response: str, user_message: str) -> float:
        """Score how relevant the response is to the user message"""
        # In production, this would be an LLM-as-judge call
        # For now, just return a placeholder
        # Values: 0 = not relevant, 1 = highly relevant
        return 0.8  # Placeholder

Now let’s see how to use this in a real application:

from openai import OpenAI

client = OpenAI()
observer = LLMObserver(client, environment="prod")
# In your API endpoint handler:
def process_support_ticket(user_id: str, ticket_text: str):

    # Retrieve relevant docs (RAG)
    relevant_docs = retrieve_context(ticket_text)

    # Call LLM with automatic observability
    response, span_id = observer.call_llm(
        user_id=user_id,
        feature_name="support_ticket_auto_response",
        system_prompt="""You are a helpful support agent. 
                         Answer based only on the provided documentation.
                         If you can't answer from the docs, say so.""",
        user_message=ticket_text,
        model="gpt-4o",
        temperature=0.7,
        retrieved_docs=relevant_docs,
        trace_id=request.trace_id,  # Correlate with incoming request
    )

    # Response is logged automatically
    return response

Every LLM call now produces a structured log record with operational metrics, quality signals, and traceability.

What to Monitor: The Dashboards That Actually Matter

Logging is half the battle. The other half is monitoring the logs and alerting when something’s wrong.

Here are the dashboards your team actually needs:

Dashboard 1: Cost per User per Day

Track total API spend aggregated by user. Alert if any single user spends more than 10x their daily average. This catches runaway loops (a user triggering the same LLM call thousands of times) or suddenly expensive behavior.

Dashboard 2: Response Quality Over Time

Plot hallucination rate, relevance score, and format validity over time. Look for regressions. If your hallucination rate goes from 2% to 8%, something changed. Was it a prompt update? A model version upgrade? A change in retrieval quality?

Dashboard 3: Error Rate by Error Type

Distinguish rate limit errors from auth errors from LLM provider issues. Each requires a different response.

Dashboard 4: Latency Percentiles by Feature

Track p50, p95, p99 latency separately for each feature. If your chatbot’s p99 latency suddenly jumped from 2 seconds to 8 seconds, something’s wrong. Is it the model being slower? Network issues? Prompt size increasing?

Dashboard 5: Quality vs Cost Scatter Plot

Plot cost per request (X axis) against relevance score (Y axis). You should see a roughly positive correlation. If you suddenly have expensive requests with low relevance, something’s off with your retrieval or prompt.

Dashboard 6: Prompt Effectiveness A/B Test Board

If you’re testing multiple prompts, track quality metrics for each. Which prompt versions produce higher relevance? Lower hallucination? This is how you optimize over time.

The Tools: What’s Actually Used in Production

<cite index=”1–1">LLM observability platforms track latency, throughput, error rates, token usage, groundedness, relevance to the prompt, hallucination rate, and toxicity/bias indicators. Leading platforms include Langfuse, Braintrust, LangSmith, and Datadog.</cite>

For teams just starting:

Langfuse (open source + cloud): Self-hosted or managed. Captures LLM traces, supports cost attribution, has built-in eval framework.

Helicone: Lightweight gateway for multi-provider LLM logging. Fast setup, minimal code changes. Good for “log the requests” but not for deep quality evaluation.

Datadog LLM Monitoring: If you’re already on Datadog for infrastructure, extends your existing APM to cover LLM calls. Correlates AI issues with application performance.

Custom logging to structured backend: If you prefer full control, send structured logs to your existing logging infrastructure (DataDog, Splunk, OpenObserve) and build dashboards on top.

Most production teams use a hybrid: Langfuse for detailed tracing and evaluation during development, plus structured logging to their existing backend for production alerting.

The Honest Reality

Building good observability for LLM applications is not optional if you care about production reliability. It’s the difference between being surprised by customer complaints and being alerted before your users notice something’s wrong.

Start with Layer 1 (operational metrics). It’s easy and immediately valuable. Then add Layer 2 (quality metrics). This is where you catch regressions. Layer 3 (root cause) comes later once you have patterns in your data.

The cost of observability is real. It’s not free to store all these traces, to run evaluations on every response. But the cost of not having it is higher: silent failures, user distrust, and 3 AM debugging sessions trying to figure out why everything looked fine but your users are complaining.

Invest in observability early. The earlier you do it, the faster you’ll be able to understand what’s actually happening in your production system.

This connects to my broader LLM engineering series covering token caching, cost optimization, context engineering, and guardrails. Together they form the operational foundation of production LLM systems. Follow for more.