DSPy 3— Evaluating DSPy Programs: Moving Beyond Prompt Guesswork

A program that runs is not yet a program we understand

Infographic by NotebookLM

In the previous article (Article 2), we built our first runnable DSPy program: an IT operations incident classifier. The classifier received an alert message, log summary, or service desk note and returned exactly one supported incident category, such as Database Issue, Scheduler Issue, or Application Server Issue. We used a class-based DSPy signature with a typed Literal output field so that the supported categories were part of the task contract rather than a loose instruction hidden in a prompt.

That first version was useful because it gave us a working language-model component. We could call it from Python, inspect the returned prediction, and use dspy.inspect_history() to see the prompt-like message DSPy generated under the hood. Yet the classifier was still only a baseline. It could produce plausible answers on a few examples, but we had not measured whether those answers were correct across a wider set of cases.

This distinction is central to DSPy. A language-model program that runs successfully is not necessarily a reliable program. It may work on the examples we happened to try, but fail on ambiguous alerts, edge cases, or slightly different phrasing. To move beyond prompt guesswork, we need labelled examples, a metric, and an evaluation loop.

Version and setup

The examples in this article use DSPy 3.x and gpt-5.4-mini. DSPy has changed across versions, so older tutorials may show different imports, configuration patterns, or return values. The package is now installed as dspy; the older dspy-ai package name is legacy and may appear in older material.

For this article, install DSPy together with python-dotenv:

pip install dspy python-dotenv

We use python-dotenv to load local environment variables from a local .env file.

Create a file named .env in the same project directory as your Python script. At minimum, it should contain your OpenAI API key:

OPENAI_API_KEY=your_api_key_here

You can also set the model name in the same file:

OPENAI_MODEL=gpt-5.4-mini

The OPENAI_MODEL value is optional. If it is not set, the script below defaults to gpt-5.4-mini. The script also adds the openai/ provider prefix when needed, because DSPy uses LiteLLM underneath, and LiteLLM commonly expects model names in a provider/model format such as openai/gpt-5.4-mini. If gpt-5.4-mini is not available in your account, or if you want to compare cost and performance, you can set OPENAI_MODEL to another OpenAI model supported in your environment.

Make sure .env is included in your .gitignore file so that the key is not committed to Git. This is a small habit, but an important one. API keys should not be hard-coded into source files or pushed to a public repository.

When you run the script, you may see LiteLLM warnings about optional AWS-related integrations such as Bedrock or SageMaker, especially if botocore is not installed. If you are using an OpenAI model, these warnings do not usually stop the example from running. They are warnings about optional provider support rather than errors in the DSPy classifier itself.

The problem with judging by inspection

Suppose our classifier sees this alert message:

The overnight batch did not start because the job stream remained 
in dependency wait.

If the model returns Scheduler Issue, the answer seems reasonable. The message refers to a batch process that did not start because a scheduler dependency was not released. On one example, a human reader can make that judgement fairly quickly.

Now consider a more ambiguous alert:

The batch job failed after the database connection timed out.

If the model returns Scheduler Issue, it has focused on the failed batch job. If it returns Database Issue, it has focused on the database connection timeout. If it returns Network Issue, that may also be defensible if the timeout was caused by connectivity. Across many examples, this becomes harder to judge informally.

Manual inspection is useful during development, but it does not scale into a reliable evaluation method. We may notice the examples where the model behaves well and miss the examples where it fails. We may also change the signature, the model, or the task description, see one improved output, and assume the whole system has improved, even if other cases have quietly become worse.

Evaluation forces us to be more explicit. We need to decide what the expected answer is for each example, even when that decision is somewhat artificial. For a classifier, this means assigning one gold category to each alert message. That gold category is not a perfect description of reality; it is an operational decision about how this system should behave.

Reusing the typed classifier from Article 2

We will keep the typed classifier from Article 2. This is important because this article is not about changing the model interface again. It is about measuring the behaviour of the classifier we already built.

The classifier signature looks like this:

from typing import Literal
import dspy

class IncidentClassifier(dspy.Signature):
    """Classify the IT operations alert into exactly one supported incident category."""
    alert_message: str = dspy.InputField(
        desc="IT operations alert message, log summary, or service desk note"
    )
    incident_category: Literal[
        "Database Issue",
        "Scheduler Issue",
        "Application Server Issue",
        "Network Issue",
        "Access Permission Issue",
        "Data Quality Issue",
        "Storage Capacity Issue",
        "Unknown",
    ] = dspy.OutputField(desc="Exactly one supported incident category")

We create the baseline classifier with dspy.Predict:

baseline_classifier = dspy.Predict(IncidentClassifier)

The call shape is simple because the taxonomy already lives in the signature:

prediction = baseline_classifier(
    alert_message=(
        "The overnight batch did not start because the job stream "
        "remained in dependency wait."
    )
)

There is no separate labels argument. The allowed categories are already declared in the typed output field. This gives us a clearer interface, but it still does not tell us whether the classifier chooses the correct category. That is the role of evaluation.

Constructing DSPy examples

A DSPy example contains the fields needed to run the program and the fields needed to judge the answer. For this classifier, each example contains an alert_message and the expected incident_category.

Here is one example:

dspy.Example(
    alert_message=(
        "The overnight batch did not start because the job stream "
        "remained in dependency wait."
    ),
    incident_category="Scheduler Issue",
).with_inputs("alert_message")

The .with_inputs("alert_message") call is important. It tells DSPy that alert_message is the input field that should be passed into the classifier. The remaining field, incident_category, is treated as reference information. In this article, that reference field is the gold category used by the metric.

Without this input marking, the evaluation setup would be ambiguous. DSPy needs to know which fields are inputs to the program and which fields are expected outputs used for scoring. For this typed classifier, the input is just the alert message.

Here are a few examples from the dataset:

examples = [
    dspy.Example(
        alert_message=(
            "The overnight batch did not start because the job stream "
            "remained in dependency wait."
        ),
        incident_category="Scheduler Issue",
    ).with_inputs("alert_message"),
    dspy.Example(
        alert_message="The application server disk reached 99 percent usage during log generation.",
        incident_category="Storage Capacity Issue",
    ).with_inputs("alert_message"),
    dspy.Example(
        alert_message=(
            "Several records were rejected because the product code "
            "was missing from the input file."
        ),
        incident_category="Data Quality Issue",
    ).with_inputs("alert_message"),
    dspy.Example(
        alert_message="A user cannot access the dashboard after their account was re-enabled.",
        incident_category="Access Permission Issue",
    ).with_inputs("alert_message"),
]

The category strings in the examples must match the supported taxonomy. The metric we will write shortly normalises casing and whitespace, which protects against small formatting mistakes, but it does not turn one category into another. Database Issue and Scheduler Issue are still different categories.

Held-out examples for evaluation

In a full DSPy optimisation workflow, we usually distinguish between training examples and held-out examples. Training examples are examples that an optimiser may later be allowed to use. Held-out examples are reserved for measuring performance.

In this article, however, we are not optimising yet. We therefore focus on the held-out set only. The aim is to evaluate the baseline classifier on examples that are treated as measurement cases, rather than examples used to tune or compile the program.

This distinction matters because measuring a system on the same examples used to design or optimise it can produce a misleading score. We want to know how the classifier behaves on examples that have been set aside for evaluation.

A small tutorial dataset will not produce statistically strong claims. In the example code below, the held-out set has 20 examples. That means one changed prediction moves the score by five percentage points. With 20 held-out examples, each example is worth five percentage points. A score of 95.0 would mean 19 out of 20 correct; a score of 100.0 means all 20 held-out examples matched their gold categories.

The training set will become important in the next article, where we introduce DSPy optimisation. For now, the important point is simpler: define labelled examples, reserve them for evaluation, run the classifier, and apply a metric.

Defining a metric

A metric is a function that compares the program’s prediction with the expected answer. For this classifier, the simplest metric is exact category match: did the predicted incident category equal the gold incident category?

A very brittle version would look like this:

def incident_metric(example, pred, trace=None):
    return pred.incident_category == example.incident_category

This may be enough when the output and gold categories are always perfectly formatted. In practice, it is safer to normalise casing and whitespace before comparing labels. For example, Database Issue, database issue, and Database Issue should not be treated as three different answers in this simple tutorial.

def normalise_label(label: str) -> str:
    return str(label).strip().lower()

def incident_metric(example, pred, trace=None):
    return normalise_label(pred.incident_category) == normalise_label(
        example.incident_category
    )

The metric returns a plain boolean. This is appropriate for an exact-match classifier: True means the prediction matched the expected category, and False means it did not. DSPy’s evaluation utility can average these boolean values to produce an overall score.

This metric is intentionally simple. It does not handle cases where two categories might both be defensible, and it does not give partial credit. That limitation is acceptable for this stage because the aim is to introduce the mechanics of evaluation. Later, for more complex tasks, the metric itself may need to become more sophisticated.

Running the evaluation

DSPy provides an evaluation utility that runs a program over a set of examples and applies a metric. For our held-out set, the evaluation object looks like this:

evaluator = dspy.Evaluate(
    devset=heldout_set,
    metric=incident_metric,
    display_progress=True,
    display_table=False,
)

We can then evaluate the baseline classifier:

baseline_result = evaluator(baseline_classifier)

In current DSPy 3.x, this call returns an evaluation result object, not a bare float. The overall score is available as:

baseline_result.score

The per-example results are available as:

baseline_result.results

This detail matters because older examples may show evaluation as if it returns only a number. In the current interface, the result object is more useful. It gives us the overall score, but also lets us inspect individual examples, predictions, and scores.

We can print both the score and the raw count:

correct_count = sum(bool(score) for _, _, score in baseline_result.results)
total_count = len(baseline_result.results)

print(f"Score: {baseline_result.score:.1f}")
print(f"Correct: {correct_count}/{total_count}")

In my run with openai/gpt-5.4-mini, the baseline classifier produced the following result:

Score: 100.0
Correct: 20/20

This means that the classifier matched the selected gold category on all 20 held-out examples. That is encouraging, but it should still be interpreted carefully. The held-out set contains only 20 examples, so it is a tutorial evaluation rather than a production benchmark. A score of 100.0 here does not prove that the classifier will handle all future operational alerts correctly. It tells us that, for this small evaluation set and this particular model, the baseline followed the intended category policy very well.

Inspecting the evaluation results

The headline score is useful, but the individual results are often more informative. We can print the examples where the classifier did not match the gold category:

print("\nMisclassified examples:")

found_error = False

for example, prediction, score in baseline_result.results:
    if not score:
        found_error = True
        print("Alert message:", example.alert_message)
        print("Gold:", example.incident_category)
        print("Predicted:", prediction.incident_category)
        print()

if not found_error:
    print("No misclassified examples.")

In this run, the classifier did not produce any misclassified examples. The output after Misclassified examples: was No misclassified examples.because every held-out example matched its gold category.

This is still useful information. Evaluation is not valuable only when it finds failures. It is also valuable when it confirms that a baseline behaves correctly on a defined set of cases. The important point is that we now know the result because we measured it, rather than because we inspected a few outputs manually.

The boundary cases are still worth examining conceptually. For example:

The dashboard is returning HTTP 500 because the application cannot write audit logs; the log directory is full.

This alert contains a strong application-server symptom, but the stated cause is lack of storage. In the held-out set, the gold category is Storage Capacity Issue.

Another example is:

The payment job failed with a database login error because the batch service account is locked.

This alert contains database-login wording, but the underlying cause is access control. In the held-out set, the gold category is Access Permission Issue.

The classifier handled these examples correctly in this run. That does not make evaluation unnecessary. It shows the opposite: by using labelled examples and a metric, we can state exactly what behaviour was tested and what result was observed.

What evaluation can and cannot tell us

Evaluation gives us a disciplined way to compare versions of the classifier. If we later change the model, adjust the signature, or optimise the program, we can run the same held-out evaluation and compare the results. This is already a major improvement over judging by a few manually inspected outputs.

However, evaluation is only as good as the examples and the metric. If the held-out examples are too easy, the score may look impressive without telling us much. If the examples are not representative, the score may not reflect the cases users actually send. If the metric is too crude, it may hide important differences between outputs.

For this reason, a small tutorial evaluation should be treated as a teaching tool, not as a production benchmark. Its value is that it shows the shape of the workflow: define a task, construct examples, mark the inputs, write a metric, evaluate the program, and inspect the results, including any failures if they appear.

Complete working code

The full example from this article is shown below. It assumes that DSPy and python-dotenv are installed, and that your OpenAI API key is stored in a local .env file or is otherwise available as an environment variable.

Your .env file should contain:

OPENAI_API_KEY=your_api_key_here
OPENAI_MODEL=gpt-5.4-mini

The OPENAI_MODEL line is optional. If it is omitted, the script defaults to gpt-5.4-mini.

The Python script is:

import os
from typing import Literal

import dspy
from dotenv import load_dotenv

# Load environment variables from the local .env file.
load_dotenv()

# Read the OpenAI API key from the environment.
# Avoid hard-coding API keys directly into source files.
openai_api_key = os.getenv("OPENAI_API_KEY")
if not openai_api_key:
    raise RuntimeError(
        "OPENAI_API_KEY was not found. "
        "Create a .env file with OPENAI_API_KEY=your_api_key_here."
    )

# Read the model name from the environment.
# Example .env entry:
# OPENAI_MODEL=gpt-5.4-mini
openai_model = os.getenv("OPENAI_MODEL", "gpt-5.4-mini")

# DSPy/LiteLLM commonly uses provider/model format.
if not openai_model.startswith("openai/"):
    openai_model = f"openai/{openai_model}"

# Configure the language model.
lm = dspy.LM(
    openai_model,
    api_key=openai_api_key,
    temperature=0,
)

dspy.configure(lm=lm)

print(f"Using model: {openai_model}")
print()

class IncidentClassifier(dspy.Signature):
    """Classify the IT operations alert into exactly one supported incident category."""

    alert_message: str = dspy.InputField(
        desc="IT operations alert message, log summary, or service desk note"
    )

    incident_category: Literal[
        "Database Issue",
        "Scheduler Issue",
        "Application Server Issue",
        "Network Issue",
        "Access Permission Issue",
        "Data Quality Issue",
        "Storage Capacity Issue",
        "Unknown",
    ] = dspy.OutputField(desc="Exactly one supported incident category")

# Create the baseline classifier.
baseline_classifier = dspy.Predict(IncidentClassifier)

# Held-out examples used for evaluation.
heldout_set = [
    dspy.Example(
        alert_message="The Oracle instance crashed overnight and every active session was disconnected.",
        incident_category="Database Issue",
    ).with_inputs("alert_message"),
    dspy.Example(
        alert_message="The application logs show database connection refused from Oracle.",
        incident_category="Database Issue",
    ).with_inputs("alert_message"),
    dspy.Example(
        alert_message="The reporting query failed after the database session was killed.",
        incident_category="Database Issue",
    ).with_inputs("alert_message"),

    dspy.Example(
        alert_message="The 02:00 job did not trigger because the scheduler calendar failed to load.",
        incident_category="Scheduler Issue",
    ).with_inputs("alert_message"),
    dspy.Example(
        alert_message="The payment batch remained waiting for a predecessor job and missed its SLA.",
        incident_category="Scheduler Issue",
    ).with_inputs("alert_message"),
    dspy.Example(
        alert_message="The release dependency was not cleared, so the job stream stayed pending.",
        incident_category="Scheduler Issue",
    ).with_inputs("alert_message"),

    dspy.Example(
        alert_message="After the latest deployment, the web portal returns 502 Bad Gateway for every request.",
        incident_category="Application Server Issue",
    ).with_inputs("alert_message"),
    dspy.Example(
        alert_message="The API container restarted repeatedly after deployment.",
        incident_category="Application Server Issue",
    ).with_inputs("alert_message"),
    dspy.Example(
        alert_message="Users receive a blank page when opening the web portal.",
        incident_category="Application Server Issue",
    ).with_inputs("alert_message"),

    dspy.Example(
        alert_message="Two office sites cannot reach any internal system because the core switch is down.",
        incident_category="Network Issue",
    ).with_inputs("alert_message"),
    dspy.Example(
        alert_message="The batch server cannot reach the SFTP host due to network timeout.",
        incident_category="Network Issue",
    ).with_inputs("alert_message"),
    dspy.Example(
        alert_message="DNS lookup fails for the internal reporting endpoint.",
        incident_category="Network Issue",
    ).with_inputs("alert_message"),

    dspy.Example(
        alert_message="A contractor whose account was disabled can no longer open the reporting dashboard.",
        incident_category="Access Permission Issue",
    ).with_inputs("alert_message"),
    dspy.Example(
        alert_message="The service account cannot write to the output directory because permission is denied.",
        incident_category="Access Permission Issue",
    ).with_inputs("alert_message"),

    dspy.Example(
        alert_message="The import rejected about 2,000 rows because the date column was in the wrong format.",
        incident_category="Data Quality Issue",
    ).with_inputs("alert_message"),
    dspy.Example(
        alert_message="The input file was rejected because product codes were blank on several rows.",
        incident_category="Data Quality Issue",
    ).with_inputs("alert_message"),

    dspy.Example(
        alert_message="The backup volume is at 100 percent and no new snapshots can be written.",
        incident_category="Storage Capacity Issue",
    ).with_inputs("alert_message"),
    dspy.Example(
        alert_message="The log directory filled up and the application stopped writing audit files.",
        incident_category="Storage Capacity Issue",
    ).with_inputs("alert_message"),

    dspy.Example(
        alert_message="What is the on-call rota for the bank holiday weekend?",
        incident_category="Unknown",
    ).with_inputs("alert_message"),
    dspy.Example(
        alert_message="Can we book a post-incident review meeting for Thursday afternoon?",
        incident_category="Unknown",
    ).with_inputs("alert_message"),
]

def normalise_label(label: str) -> str:
    return str(label).strip().lower()

def incident_metric(example, pred, trace=None):
    return normalise_label(pred.incident_category) == normalise_label(
        example.incident_category
    )

# Evaluate the baseline classifier on held-out examples.
evaluator = dspy.Evaluate(
    devset=heldout_set,
    metric=incident_metric,
    display_progress=True,
    display_table=False,
)

baseline_result = evaluator(baseline_classifier)

correct_count = sum(bool(score) for _, _, score in baseline_result.results)
total_count = len(baseline_result.results)

print(f"Score: {baseline_result.score:.1f}")
print(f"Correct: {correct_count}/{total_count}")

print("\nMisclassified examples:")

found_error = False

for example, prediction, score in baseline_result.results:
    if not score:
        found_error = True
        print("Alert message:", example.alert_message)
        print("Gold:", example.incident_category)
        print("Predicted:", prediction.incident_category)
        print()

if not found_error:
    print("No misclassified examples.")

This code evaluates the typed baseline classifier on a held-out set. It does not yet optimise the program, and it does not yet define a training set. In the next article, we will introduce training examples, use them with a DSPy optimiser, and compare the compiled classifier against the uncompiled baseline on a harder held-out set.

What comes next

In this article, we moved from informal inspection to evaluation. We kept the typed classifier from Article 2, constructed DSPy examples, marked the input field with .with_inputs("alert_message"), defined a simple boolean metric, and evaluated the classifier on held-out examples.

In this run, the baseline classifier achieved 20 out of 20 on a small held-out set. That is a strong result for the tutorial example, but it should not be read as the end of the problem. Most examples in this article were relatively clean alerts where the intended category was either explicit or strongly implied. They were suitable for introducing the mechanics of evaluation, but they do not fully test harder cases where the visible symptom and the underlying cause point in different directions.

The next article therefore makes the workflow more demanding. We will keep the same typed classifier, the same metric, and the same evaluation discipline, but we will introduce training examples and use them with a DSPy optimiser. We will also add more boundary-focused examples. Some examples deliberately test whether the classifier follows the surface symptom or the likely root cause. For instance, a scheduler-looking alert may actually be caused by a firewall change, and a database-looking error may actually be caused by invalid source data or a locked service account.

This sets up the role of optimisation. DSPy will use training examples and the metric to compile a revised version of the classifier. We will then compare the baseline and compiled programs on a held-out set. The point is not to claim that optimisation always improves a model. The point is to show how optimisation can be tested: define the behaviour, evaluate the baseline, compile a revised program, and check what changed.

Further reading

Stanford DSPy project and documentation. https://dspy.ai/

Omar Khattab et al. (2023), DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines, arXiv:2310.03714. https://arxiv.org/abs/2310.03714

Serj Smorodinsky and William Brett Kennedy, Building LLM Applications with DSPy, Manning Early Access Program. https://www.manning.com/books/building-llm-applications-with-dspy