Eval-guided iterative prompt optimization for AI-powered workout parsing

I’ve been building Featherweight, a strength training tracking app, for the past year. One of its main selling points is AI-powered programme parsing: you paste a training programme as text, and the AI converts it into structured data that the app can use, including things like weeks, workouts, exercises, sets, reps, weight.

TL;DR

I improved AI strength training programme parsing accuracy from 76% to 89% using deterministic eval graders, an LLM-as-judge for ambiguous cases, and Promptimizer — an OPRO-style optimization loop where Claude iteratively rewrites a GPT prompt guided by eval scores.

The Problem

The AI parsing works… for the most part. For the same input, I get different outputs. Exercises get collapsed when they shouldn’t be (paused bench press is NOT the same as bench press). Sets go missing. The model decides “3x10” means 1 set of 10 with a note saying “do 3 sets” instead of creating 3 actual set objects. And every time I tweak the prompt to fix one issue, something else breaks that was working fine before.

I have no way to know if a prompt change is a net improvement or a net regression. I’m playing whack-a-mole with a system prompt, testing against whatever programme I happen to have on my clipboard, and hoping for the best.

AI programme parsing is the feature that differentiates Featherweight from Strong and Hevy and every other tracking app out there. If it doesn’t work reliably, I have no product.

So I decided to build a proper eval harness for the AI programme parsing functionality.

A Quick Glossary (for the Non-Lifters)

For non-gymgoers, some of the terminology in this post will be opaque. Here’s a cheatsheet:

• Programme (or program): A structured training plan, usually spanning multiple weeks.
• Workout: A single training session. “Leg day” is a workout. A programme contains multiple workouts.
• Exercise: A specific movement. Bench press, squat, deadlift, bicep curl — each is an exercise.
• Set: A group of consecutive repetitions without rest. When someone says “do 3 sets”, they mean perform the exercise 3 separate times with rest in between.
• Rep (repetition): One complete movement of an exercise. Picking the barbell up and putting it down once = 1 rep.
• 3x10: Shorthand for “3 sets of 10 reps”. The first number is sets, the second is reps.
• Weight: The load used. Measured in kg or lbs.
• RPE (Rate of Perceived Exertion): A 1–10 scale of how hard a set felt. RPE 10 = maximum effort, couldn’t do another rep. RPE 8 = could have done 2 more reps.
• 1RM (One-Rep Max): The heaviest weight you can lift for a single repetition. Everything else is often expressed as a percentage of this. “80% 1RM” = 80% of your max.
• AMRAP: As Many Reps As Possible. Do the exercise until you can’t anymore (with good form).
• Superset (SS): Two exercises performed back-to-back with no rest between them.

What Even Is AI Programme Parsing?

For context: users copy training programmes from Reddit, coaching apps, spreadsheets, PDFs, or wherever, and paste them into Featherweight. The text goes to a Cloud Function that calls GPT-4.1-mini (or gpt-4.1-nano for simpler inputs), which parses it into structured JSON:

{
  "name": "Wendler 5/3/1 BBB",
  "durationWeeks": 4,
  "weeks": [
    {
      "weekNumber": 1,
      "workouts": [
        {
          "name": "OHP Day",
          "exercises": [
            {
              "exerciseName": "Overhead Press",
              "sets": [
                {
                  "reps": 5,
                  "weight": null,
                  "rpe": null,
                  "percentage": 65
                },
                {
                  "reps": 5,
                  "weight": null,
                  "rpe": null,
                  "percentage": 75
                },
                {
                  "reps": 5,
                  "weight": null,
                  "rpe": null,
                  "percentage": 85
                }
              ]
            }
          ]
        }
      ]
    }
  ]
}

The parsed JSON then goes through an exercise matching service that fuzzy-matches the AI’s exercise names against our database of 500+ canonical exercises (each with 5–10 aliases). “OHP” becomes “Overhead Press” on the BARBELL equipment. "RDL" becomes "Romanian Deadlift".

The Research

Before building anything, I wanted to understand two things:

1. How diverse is the input space? (How many different ways do people write training programmes?)
2. What’s the state of the art for evaluating LLM structured output?

The Input Space Is Enormous

Here’s a taste of what people actually paste into the app:

Set notation alone has at least 7 major variants:

• 3x10 (standard)
• 3×10 (Unicode multiplication sign)
• 3 sets of 10 (verbose)
• 12/10/8 (pyramid)
• 1x5+ (AMRAP suffix)
• 75% x 2 x 5 (weight-first, Olympic lifting style — this means 5 sets of 2 at 75%)
• bench 225x5 (forum shorthand)

And that’s before we get into weight notation (80kg, 175lbs, 135#, 70% 1RM, 70% TM, BW+20kg), RPE notation (@RPE 8, @8, RPE 7-8, 2 RIR), tempo notation (3010, 3-0-1-0), and special set types (AMRAP, EMOM, supersets with A1/A2 notation, drop sets, rest-pause, myo-reps, cluster sets...).

Then there’s the structural diversity. People paste:

• Minimal text: "bench 3x5\nsquat 5x5\ndeadlift 1x5"
• Reddit markdown tables with pipe separators
• Tab-separated spreadsheet data
• Full paragraphs from coaching apps
• OCR’d text from screenshots (where 10 becomes 1O and words get split randomly)

And real programmes from popular sources like Starting Strength, Wendler 5/3/1, GZCL, Reddit PPL, Renaissance Periodization — each with their own conventions and notation quirks.

I catalogued all of this into a 1000+ line research document. The diversity of input that a real user base produces is orders of magnitude beyond what I was testing against, or even expecting.

Eval Frameworks

I surveyed Promptfoo, Braintrust, LangSmith, Arize Phoenix, and OpenAI Evals. Promptfoo won convincingly: open source, YAML config, 40+ assertion types, native CI/CD integration (exits with a failure code when tests fail), built-in response caching, and it supports both OpenAI and Claude as target models and judge models.

The Evaluation Strategy

My output is structured JSON, not free-form text. So I can do most of the grading deterministically without ever calling an LLM to judge the output.

Tiered Grading

I decided on five tiers, from cheapest to most expensive:

Tier 1 — Schema Validation (free, instant): Does the output parse as valid JSON? Are the required fields present? Are the types correct?

Tier 2 — Structural Match (free, instant): Does the output have the right number of weeks? Workouts per week? Exercises per workout? Sets per exercise? If the input says “4 weeks, 4 days per week”, and the output has 3 weeks with 2 days each, that’s a structural failure.

Tier 3 — Numeric Accuracy (free, instant): Are the reps correct? The weight? The RPE? These are exact integer/float comparisons. If the input says 3x10 @80kg and the output says 8 reps at 80kg, that's wrong. No ambiguity.

Tier 4 — Exercise Name Matching (cheap, instant): The AI outputs exercise names as strings. I score them against our canonical exercises:

• Exact canonical name match: 1.0
• Match to a known alias (I have 5–10 per exercise): 0.9
• Equipment + core name match (e.g., “Barbell Bench Press” matches “Bench Press”): 0.85
• Levenshtein similarity > 0.8 against any canonical name or alias: 0.7
• No match at all: 0.0

A “perfect” parse produces canonical names. An “acceptable” parse produces names that the matching service can resolve. An “unacceptable” parse produces names that don’t match anything.

Tier 5 — LLM-as-Judge (expensive, slow): For genuinely ambiguous cases — “does this parsed programme faithfully represent the original text?” — use Claude as a judge with a rubric. This is the expensive check, and I use it sparingly.

Weighted Scoring

Not all fields matter equally. I assign weights:

| Field         | Weight | Why                                                  |
|---------------|--------|------------------------------------------------------|
| Reps          | 3      | Wrong reps = wrong programme. Potentially dangerous. |
| Weight        | 3      | Wrong weight = potentially dangerous                 |
| Set count     | 2      | Missing sets = wrong training volume                 |
| Exercise name | 2      | Wrong exercise = wrong programme entirely            |
| RPE           | 1      | Important when present, often absent in input        |
| Metadata      | 0.5    | Programme type, difficulty, day names — nice to have |

The Test Corpus

I built ~45 test cases across four tiers of complexity, based on online research of how users write their programmes:

Simple (~10 cases): StrongLifts 5x5, basic push pull legs, single workout days, minimal notation.

Medium (~15 cases): Wendler 5/3/1 with percentages, Reddit PPL with rep ranges and supersets, GZCLP with tiered structure, spreadsheet pastes, markdown tables.

Complex (~10 cases): 12-week periodized programmes, Olympic lifting with weight-first notation, CrossFit WODs, coaching-platform-style autoregulation.

Edge Cases (~10 cases): OCR artifacts, abbreviation-heavy programmes, exercise variants that should NOT be merged, very long inputs.

Each test case is a pair: the raw text (exactly as a user would paste it) and the expected JSON output (verified against the original programme).

Implementation and Results

The eval harness lives in functions/evals/, built on Promptfoo (open source, MIT). I extracted all prompt logic into a standalone promptBuilder.ts with zero Firebase dependencies — the Cloud Function and the eval harness import the same module. Same prompt, no drift.

The stack: Promptfoo for orchestration, 4 custom JavaScript graders (schema, structural, numeric, exercise names), YAML test cases with verified ground truth, and a canonical exercise database (500+ exercises with aliases) for name scoring. One prerequisite fix: setting temperature: 0 in production — the OpenAI default of 1.0 was adding noise to every parse.

45 test cases across four tiers, run against gpt-4.1-mini and gpt-4.1-nano:

| Tier            | gpt-4.1-mini    | gpt-4.1-nano    |
|-----------------|-----------------|-----------------|
| Simple (10)     | 10/10 (100%)    | 9/10 (90%)      |
| Medium (15)     | 9/15 (60%)      | 5/15 (33%)      |
| Complex (10)    | 7/10 (70%)      | 3/10 (30%)      |
| Edge cases (10) | 8/10 (80%)      | 6/10 (60%)      |
| **Total**       | **34/45 (76%)** | **23/45 (51%)** |

Mini wins by 25 percentage points. Nano fails more than it succeeds on anything beyond 3x10 notation — it drops sets, collapses exercises, and chokes on percentages. The cost difference? $0.004 per parse. Two cents per user per month. I dropped nano and switched to mini for everything.

I also tested gpt-5-mini (67%, 3x cost, 5x slower due to reasoning tokens) and gpt-5.4-mini (no improvement, 2x cost, API breaking changes). Neither justified switching. gpt-4.1-mini remains the best option: cheapest, fastest, highest quality.

| Model          | Score  | Cost/parse | Notes                   |
|----------------|--------|------------|-------------------------|
| gpt-4.1-mini   | 76%    | $0.008     | Best overall            |
| gpt-4.1-nano   | 51%    | $0.004     | Dropped                 |
| gpt-5-mini     | 67%    | $0.024     | 3x cost, worse quality  |
| gpt-5.4-mini   | ~69%   | $0.016     | 2x cost, API issues     |

Promptimizer: Teaching the Prompt to Improve Itself

Now the fun part.

Once you have an eval harness with deterministic scoring, you have an objective function. And once you have an objective function, you can optimize against it.

What if an AI agent could run the eval suite, analyze which test cases failed and why, modify the prompt to fix those failures, and re-run the eval? Then repeat. Each iteration tracked with full provenance — the prompt diff, the score change, and the agent’s reasoning.

The concept has academic foundations, most notably:

• OPRO (Google DeepMind, 2023) feeds previous prompts and their scores into an LLM and asks it to propose a better one. It outperformed human-written prompts by up to 8%.

I colloquially called my implementation Promptimizer.

How It Works

The loop is simple:

1. Run the eval suite → get scores per test case
2. Identify the 5 worst-performing cases
3. Feed the optimizer LLM (Claude Sonnet — deliberately a different model than the target GPT) the current prompt, the failures, and the score history
4. The optimizer proposes a modified prompt with reasoning
5. Re-run the eval with the new prompt
6. Repeat for N iterations (configurable)

The optimizer sees the score trajectory. It knows what previous changes helped and what hurt. It’s not guessing — it’s hill-climbing with memory.

Why a Different Model?

I use Claude as the optimizer even though the target model is GPT-4.1-mini. A different model family brings different biases and catches blind spots the target model doesn’t even know it has.

Overfitting Prevention

I split the test corpus 70/30: the optimizer only sees failures from the training set, but scores are computed on everything. If the training score goes up while the test score goes down, I’ve overfit to the training cases and revert.

What Gets Tracked

Every iteration produces:

• A unified diff of the prompt changes
• The optimizer’s chain-of-thought reasoning
• Per-tier score breakdown (schema, structural, numeric, exercise names)
• The overall score trajectory

This gives full auditability. I can trace exactly why the prompt evolved the way it did, which changes helped, and which were dead ends.

The Bigger Picture

Promptfoo doesn’t support this natively (there’s an open feature request at the time of writing). Neither does any other eval framework I found. The existing tools — DSPy, TextGrad, DeepEval — are Python-only and want to own the entire pipeline.

So I built a thin orchestration layer on top of promptfoo. The eval framework handles measurement. The optimizer handles improvement.

Cost per Promptimizer run: ~$2 for 5 iterations.

First Run Results

I ran the Promptimizer against the simple + medium tiers (25 cases) for 3 iterations:

| Iteration    | Score           | Change                      |
|--------------|-----------------|-----------------------------|
| 1 (baseline) | 76% (19/25)     | —                           |
| 2            | **80% (20/25)** | +4pp                        |
| 3            | 76% (19/25)     | -4pp (regression, reverted) |

The optimizer (Claude Sonnet 4.6) analysed the 5 worst-performing cases and proposed targeted fixes: adding explicit rep range handling (3x8-12 → use lower bound as integer), clarifying superset notation (SS means separate exercises), fixing chin-up vs pull-up distinction, and adding CSV format parsing rules. One additional medium case started passing.

Iteration 3 regressed — the optimizer over-corrected. The system detected the -4pp drop and flagged it. The winning prompt from iteration 2 is saved with full provenance: the diff, the reasoning, and the per-case score breakdown.

That was the canary run. Then I ran it on the full 45-case suite, starting from the canary’s winning prompt:

| Iteration | Score     | Simple   | Medium  | Complex | Edge    |
|-----------|-----------|----------|---------|---------|---------|
| 1 (start) | 77.8%     | 100%     | 60%     | 70%     | 80%     |
| 2         | 82.2%     | 100%     | 73%     | 70%     | 90%     |
| 3         | **86.7%** | **100%** | **93%** | **70%** | **80%** |

77.8% → 86.7% in three iterations. The medium tier — where all the interesting notation lives — went from 60% to 93%. The optimizer figured out superset notation, pipe-separated formats, CSV parsing, and AMRAP set expansion. Each fix was targeted and explained: “the model was merging A1/A2 superset pairs instead of keeping them as separate exercises” or “the alternating exercise rule was causing set counts to be halved.”

Complex stayed at 70%. Olympic lifting weight-first notation and CrossFit WODs are genuinely hard problems that need deeper prompt work or possibly a different parsing strategy.

The winning prompt is saved with full provenance. To ship it, I review the diff and merge the changes into promptBuilder.ts. The prompt is now a living thing with a measurable quality score and a documented improvement history.

Variance Testing: How Deterministic Is Temperature 0?

Temperature 0 doesn’t guarantee determinism. I ran every test case 3 times to find out.

| Tier            | Consistency (3/3 same result) | Inconsistent Cases                 |
|-----------------|-------------------------------|------------------------------------|
| Simple (10)     | 10/10 (100%)                  | —                                  |
| Medium (15)     | 13/15 (87%)                   | wendler-531-week1, reddit-ppl-push |
| Complex (10)    | 10/10 (100%)                  | —                                  |
| Edge Cases (10) | 9/10 (90%)                    | amrap-mixed                        |
| **Total**       | **42/45 (93%)**               | **3 flaky cases**                  |

93% of cases are fully deterministic at temperature 0. The 3 inconsistent cases are all notation-heavy — AMRAP mixed with straight sets, superset notation with alternating exercises, percentage-based with AMRAP. These are the cases where minor floating-point differences in the attention mechanism can tip the model’s interpretation one way or the other. This is consistent with Anthropic’s infrastructure noise findings: “Leaderboard differences below 3 percentage points deserve skepticism.”

Therefore, when I report a score difference between two prompt versions, I need at least a 3–4pp gap before I can be confident it’s real and not just run-to-run variance.

LLM-as-Judge: When Deterministic Grading Isn’t Enough

Four deterministic graders (schema, structural, numeric, exercise names) handle most of the evaluation. But some cases land in a grey zone: the structure is close but not exact, the exercise names are reasonable but non-canonical, and the deterministic graders can’t say whether the parse is “good enough” or “fundamentally wrong.”

For these cases, I added an LLM judge — Claude Sonnet 4.6 evaluating GPT-4.1-mini’s output. Using a different model family as judge avoids self-evaluation bias.

I implemented the “Trust or Escalate” pattern:

1. Run quick structural + exercise name checks (~1ms)
2. If both scores are above 0.95 → skip the judge (confident pass)
3. If either score is below 0.3 → skip the judge (confident fail, already caught)
4. Grey zone (0.3–0.95) → call Sonnet with a faithfulness rubric

In practice: 27 out of 35 non-simple cases were gated out. Only 8 actually called the LLM. Cost per run: ~$0.20. The judge catches things deterministic graders can’t — like the olympic lifting case where sets and reps were systematically transposed (judge score: 0.30), or the reddit PPL case where two session variants were incorrectly collapsed into one (judge score: 0.40).

Final Results

With the LLM judge as a fifth grader, complex tier improved from 70% to 80% — the judge correctly identified that the CrossFit WOD parse was faithful despite the deterministic graders flagging minor structural deviations.

| Tier            | Baseline | After Promptimizer | + LLM Judge |
|-----------------|----------|--------------------|-------------|
| Simple (10)     | 100%     | 100%               | 100%        |
| Medium (15)     | 60%      | 93%                | 93%         |
| Complex (10)    | 70%      | 70%                | 80%         |
| Edge Cases (10) | 80%      | 80%                | 80%         |
| **Total**       | **76%**  | **87%**            | **89%**     |

76% → 89%. By measuring what was wrong, automating the fix cycle, and adding a judge for the cases where rules aren’t enough.

Keeping Costs Under Control

Running AI evals against paid APIs could get expensive fast. 45 test cases times 2 models (when comparing nano with mini) times whatever token count the prompt generates adds up.

Here’s what a full run actually costs:

• 5-case canary (simple tier): ~$0.01, 1 second (cached) to 30 seconds (fresh). Use this for quick sanity checks after grader changes.
• Full 45-case suite: ~$0.35, under 5 minutes with concurrency 10. This is the number that matters.
• Full + variance (3 repeats): ~$1.00. For measuring consistency before publishing results.

A few things keep costs manageable:

Promptfoo’s response cache. Once a prompt+input pair has been evaluated, the response is cached for 14 days. Re-running the suite after a grader fix costs $0 and takes 1 second. You only pay when the prompt or test cases change.

Canary runs. Before running the full suite, run just the simple tier (10 cases, 22% of the corpus). If the canary looks good, expand to 50%, then 100%. This saves money during iterative grader development.

Cheap models. gpt-4.1-mini costs $0.40/1M input tokens and $1.60/1M output. gpt-4.1-nano is even cheaper at $0.10/$0.40. The full suite uses ~300K tokens total.

On throttling: I haven’t hit any rate limits. 90 concurrent API calls (45 cases x 2 models) with concurrency 10 completed in 5 minutes with zero errors. OpenAI’s rate limits for mini/nano are generous. For the eval harness, throttling is a non-issue.

What’s Next

• Expanding the test corpus with anonymized real user submissions
• Tracking quality over time as OpenAI and Anthropic release new models
• Running Promptimizer whenever I expand the test corpus or change the output schema
• Adding latency and cost benchmarking alongside quality metrics
• Multi-language support evaluation (the notation catalogue already covers Spanish, French, German, Portuguese)