Promptfoo in Practice: Deterministic Checks and Model-Graded Evaluations — Part II

This tutorial focuses on two important testing styles in Promptfoo: deterministic checks and model-graded checks. These two categories cover many practical evaluation needs in real LLM applications.

The process of running this tutorial is straightforward. The complete source code is available in the GitHub repository. The project mainly consists of three core files, as shown below:

How the Input Is Created: Prompt + Variables

The prompt template is stored in prompts/assistant.txt.

You are a concise, accurate assistant.
Answer the user's question clearly and factually. Do not add extra commentary.

Question: {{question}}

The important part is {{question}}. This is a variable placeholder. The value comes from promptfooconfig.yaml.

tests:
  - description: "equals — exact output match"
    vars:
      question: "Reply with exactly the word: APPROVED"
    assert:
      - type: equals
        value: "APPROVED"

Promptfoo replaces {{question}} with the value from YAML before sending the prompt to the model.

Deterministic Checks vs Model-Graded Checks

Part 1: Deterministic Evaluation

Deterministic checks are the easiest place to start because they behave like traditional unit tests. You define the input question, the expected rule, and Promptfoo checks whether the LLM output follows that rule.

Why Deterministic Checks Are Helpful

Part 2: Model-Graded Evaluation

Some outputs cannot be judged by exact matching. A customer apology can be written in many good ways. A simple explanation can use different words and still be correct. This is where model-graded checks are useful.

Why Model-Graded Checks Are Helpful

Combining Both Evaluation Types

In production LLM systems, we usually need both. Deterministic checks verify strict behavior. Model-graded checks evaluate quality. Together, they provide a stronger safety net.

Running the Evaluation

npx promptfoo@latest eval

Promptfoo runs the test cases, sends each rendered prompt to the selected provider, collects the model outputs, and applies the assertions.

The output shows that all 10 tests passed. It also separates normal model generation tokens from grading tokens. This matters because model-graded checks may use an additional model call to judge the response.

Opening the Evaluation Viewer

npx promptfoo@latest view

This starts a local server, for example:

Server running at http://localhost:15500

The viewer becomes more useful as your evaluation set grows from a few tests to dozens or hundreds of tests.

Final Takeaway

Use deterministic checks when the output must follow strict rules.

Use model-graded checks when the output quality requires judgment.

Use both together when building real LLM applications.

Promptfoo turns manual prompt testing into a repeatable evaluation workflow. You can change a prompt, switch a model, or update your instructions, then run the same tests again. If the results get worse, you know before your users do.