Loop Engineering with DSPy

Prompt engineering used to mean a human in the middle of the loop: write a prompt, run it, read the failures, tweak the wording, run it again.

The more interesting version is to engineer the loop itself. Give the system a clear goal, labelled examples, feedback on what failed, and a budget.

Let it run the prompt, score the outputs, reflect on the misses, and rewrite the instruction.

This post is a concrete example with DSPy and GEPA.

The key dependency is the open source DSPy framework

Case Study: Is this contract clause unfair?

The task involves reading one clause from a Terms-of-Service contract and deciding whether it’s unfair to the consumer (LexGLUE unfairToS).

The dataset contains real clauses labelled by legal experts.

“Unfair” follows specific legal criteria a general model doesn’t reliably know:

• unilateral termination, price-change-at-will, forced arbitration, choice of foreign law, broad content licences, etc

I balanced the data 50/50 so accuracy is honest.

The raw unfair tos data is heavily skewed (~89% fair / 11% unfair), so a lazy model that always says “fair” would score 89% and look great while catching zero violations.

Balancing 50/50 makes accuracy honest.

Experiment Setup

The starting point is a deliberately bare prompt: “Decide whether this Terms-of-Service clause is unfair to the consumer.” No criteria. Let it discover them.

The clauses are split three ways:

• 200 train, what GEPA optimises on
• 120 validation, what it scores candidate prompts against, mid-run and
• 300 test locked away for the final number.

The test set isn’t touched until the run is over, so every headline figure is measured on clauses GEPA never saw.

The split is fixed for every run, so the four runs are directly comparable.

Starting Point

• 77.7% accuracy.
• 65% recall on the unfair class.

It was missing a third of the violations.

Looking under the stat into the data, the model didn’t know that “governed by the laws of the Netherlands”, “we may discontinue our services”, or “may update pricing at any time” are unfair.

Reflective Evolution enters the Chat

With GEPA, misses are fed back as targeted feedback.

Baseline vs average & best runs

From a one-line prompt, GEPA added clauses to the rubric.

It wrote them as general rules, not memorised specifics, which is what makes them transferable.

For example, “we may discontinue our services” and “may update pricing at any time” gets translated into a generic unilateral-change rule: “the provider can unilaterally change/modify/cancel terms, services, pricing … unfair even if the clause promises notice.”

The rubric only ever encodes principles, so the model is not memorising test answers, but how to think about clauses.

The optimiser created the final rubric with all criteria encoded. The violation-catch went from 65% to 86.5% on average across 4 runs (91% on the best run)

That’s a 33% relative improvement without a human having to add criteria to a prompt manually, we get AI to do it.

How Does this Work under the hood with DSPy?

DSPy is a framework in which you can declare your task as a typed program instead of writing prompt strings.

GEPA is one of DSPy’s optimisers. It takes that declared program and rewrites its prompt for you.

Gepa is also an open source project of its own (gepa-ai/gepa)

A [dspy.Signature](https://dspy.ai/getting-started/expanding-signatures/) declares the typed inputs and outputs, and its docstring is the instruction the model runs on. A [dspy.Module](https://dspy.ai/getting-started/first-program/) ([ChainOfThought](https://dspy.ai/tutorials/math/) here) makes it runnable:

Verdict = Literal["fair", "unfair"]

class ClauseFairness(dspy.Signature):
    """Decide whether this Terms-of-Service clause is unfair 
    to the consumer."""

    text: str = dspy.InputField(
        desc="A single clause from an online Terms-of-Service contract.")
    label: Verdict = dspy.OutputField(
        desc="'unfair' if the clause is unfair to the consumer, else 'fair'.")

The docstring is the prompt. The label field is can only have two values, so the output comes back as a checked label, not text we scrape.

GEPA then optimises that program.

You hand it the program, a metric, and a strong reflection model. .compile() returns a new program with a rewritten instruction:

optimised = dspy.GEPA(
    metric=fairness_metric,   # code check + written feedback
    reflection_lm=opus,       # the prompt-writer
    # ... plus budget and minibatch knobs
).compile(classify, trainset=train, valset=val)

Because the prompt is a declared field, not a string hard-coded into a model call, GEPA has something well-defined to rewrite: the instruction.

It rewrites that and nothing else. The field names, the types and your code stay exactly as written.

Under the hood. How does this work?

Three moving parts:

1. A metric that gives feedback
2. A reflection prompt that turns failures into a better instruction
3. A Pareto search that keeps what works.

1. The Metric

Correctness is a one-line code check. On a miss, it returns why. Here are the exact strings we wrote:

Missed unfair clause:

“Wrong: you said ‘fair’ but this clause is UNFAIR to the consumer. Pinpoint the specific mechanism - e.g. the provider can change terms or terminate unilaterally, liability is limited/excluded, disputes are forced to arbitration or a specific court, the consumer grants a broad content/data licence, or contract terms bind the consumer to things outside the document — and name the principle it offends.”

Over-flagged fair clause:

“Wrong: you said ‘unfair’ but this clause is FAIR. A clause being legalistic, dry, or merely one-sided is NOT itself unfair, it is unfair only if it imposes an unreasonable disadvantage on the consumer. Do not over-flag routine or informational terms.”

2. The reflection prompt

Under the hood, each round GEPA feeds the current instruction plus a minibatch of failures into a fixed template it controls:

“Here is the current instruction. Here are examples of inputs, the assistant’s responses, and feedback on how it could be better. Your task is to write a new instruction. Identify all niche and domain-specific factual information and include it. If the assistant used a generalisable strategy, include that too.”

3. The search: Genetic-Pareto

Each round GEPA:

• runs the current candidate over a minibatch of 5 training clauses, collects scores + feedback;
• hands that to Opus, gets a mutated instruction;
• if the candidate beats the parent on the minibatch, scores it on the full 120-clause validation set and adds it to the pool;
• picks the next parent from a Pareto fronties. Candidates best on some validation clauses, not just best on average, so diverse strengths survive instead of collapsing to one local optimum. It also occasionally merges two strong candidates;
• repeats until 6,000 metric calls are exhausted.

This is what the improvement against the validation accuracy and test set look like over the runs

Validation accuracy across 4 runs (band = best–worst, line = average); validation sits a few points above the held-out test we headline.

The same climb measured as unfair-recall. Violations caught, the metric that matters. The dip near 2,500 calls is real: when GEPA promotes a new accuracy-best prompt, recall can step back before climbing.

Where else this works

Anywhere experts apply non-obvious criteria to labelled decisions. GEPA mines those labels for the rubric your experts were applying implicitly.

Close to home: Additional compliance related artifacts and judgements, Compliance e.g. to advertising standards, suspicious-activity flagging, complaints triage, contract-clause review. The key is to have a robust dataset that you can consider the ground truth.

Further out: code review against your team’s house rules (rubric = past PR comments), clinical-guideline adherence, rubric-based essay grading, incident severity from past triage, systematic-review screening against inclusion criteria.

What GEPA actually produces is a readable, auditable policy document distilled from your own labelled decisions, the kind of thing that takes a compliance team months to write. It breaks the moment your experts disagree. If GEPA can’t converge, that’s also a clue: your labels may be inconsistent.

Try it

Give it a go, Clone the repo, all items are reproducible

git clone https://github.com/anastasiosyal/dspy-gepa-optimizer && cd dspy-gepa-optimizer
pip install -r requirements.txt
cp .env.example .env     # task model = cheap; reflection model = strong
python baseline.py --split test
python optimize.py