Why an LLM Classifier Passes Eval and Fails in Production

A balanced test set hides the imbalance production has. Score per-class on the real distribution, report macro and weighted F1 together, calibrate per class, and debug from the confusion matrix.

An LLM classifier can pass every eval a team runs and still fail in production, and the reason is almost always the same: the eval set was balanced and production traffic is not.

A typical result makes it concrete. A classifier scores 0.94 accuracy and 0.91 F1 on a balanced test set with a hundred examples per class, and it ships. In production, refund tickets route to billing, escalations close as resolved, and the minority class the project existed to catch drops to 0.31 recall, while that same class still reads 0.88 on the eval set. The model did not change. The eval was scoring a distribution that does not exist in production.

So evaluating an LLM classifier is two problems that look like one. The metric that matters reports per-class precision and recall on the production distribution, macro F1 next to weighted F1, a calibration curve per class, and a confusion matrix you read before touching the prompt. Here is what each metric answers and where it lies:

Why balanced-set accuracy misleads

A hundred examples per class, stratified and hand-labeled, accuracy as one number. It looks fair, and production traffic does not look like that. Real traffic is heavy-tailed: a few classes carry most of the volume, and the long tail is where escalations and revenue events live.

A balanced set teaches the model to handle every class equally. Production drops most of its weight on the easy classes, overall accuracy stays high because those are easy, and the minority classes collapse unnoticed until a downstream team sees tickets in the wrong queue.

The fix is two eval sets, scored separately.

• A production-distribution set that mirrors live traffic frequency. This is what you report, because it is what users experience.
• A per-class oversampled set with 100 to 200 examples per class regardless of frequency. This is what you debug against, because it shows whether the model can do a class at all.

Macro F1 on the oversampled set says whether the model can handle every class. Weighted F1 on the production set says what users get. The gap between them is the drift waiting to happen.

Read precision and recall per class

Aggregate numbers hide the failure. Per-class numbers expose it. Compute precision, recall, support, and F1 for each class on the production set, and three rules turn that table into fixes.

• Low precision means the class is over-predicted. Usually the prompt description is too broad, or the model uses the class as a fallback when unsure.
• Low recall means the class is under-predicted. Usually the prompt does not surface the signal users actually send, or the model confuses it with a familiar neighbor.
• Low precision and low recall together is the danger zone: an ambiguous prompt, a fuzzy label, or a class the model cannot separate from its neighbors. The confusion matrix tells you which.

Report macro and weighted F1 together

F1 is per-class, but dashboards need one number. There are two ways to summarize, and both mislead alone.

• Macro F1 weights every class equally, so minority-class failure drags it down regardless of volume. Use it when every class matters: escalation routing, safety labels, anything where a rare class drives a tail outcome.
• Weighted F1 weights by support, so common classes dominate. Use it when cost is proportional to volume, like generic tagging where common classes are most of the work.

Macro punishes a model for being honest about a genuinely rare class if you score it on a balanced set. Weighted hides a disaster: 0.97 on the common classes and 0.21 on three minority classes still posts 0.92 weighted and looks healthy. Report both and read the pair. Macro near weighted means consistent behavior. Weighted far above macro means the common classes are propping up a model that is broken on the tail, and a gap past about ten points is a regression flag, not a ship signal.

Calibrate per class before setting a threshold

A confidence score is only meaningful when calibrated: 0.8 should mean 80 percent of predictions in that band are right. LLM classifiers miss this in two ways.

• Bimodal extremes. The model returns 0.95-plus or 0.05-minus on almost everything and skips the middle, so every route-or-abstain decision fires at the same effective cutoff.
• Class-dependent skew. It is over-confident on common classes and under-confident on rare ones, so one threshold ships a different precision profile per class.

This matters the moment anything is threshold-driven: routing to a fallback, abstaining for human review, gating a downstream action. Measure it with a reliability diagram on the production set, binning by confidence and checking empirical accuracy per band. If the 0.7-to-0.8 band is only 0.55 accurate, the high-confidence routing is firing on cases the model gets wrong half the time. The fix is per-class threshold tuning against a precision target, or a calibration transform like Platt scaling or isotonic regression on the production set. Do not trust the raw score until you have seen the curve.

Debug from the confusion matrix

The confusion matrix is the debugger. Rows are the actual class, columns are the predicted class, the diagonal is correct, and every off-diagonal cell maps to a prompt or rubric fix.

Read it as a checklist.

• Biggest off-diagonal cells first. Refund and billing swap in both directions here, so that is the dominant confusion. The fix is a sharper refund-versus-billing definition plus two contrastive few-shot examples.
• Asymmetric confusions weighted by support. Fraud looks like only a few errors until you notice its support is 26, and 15 misses out of 26 is a recall disaster. The prompt does not surface the real fraud signal, so the fix is example coverage, not a bigger model.
• Silent classes. If a column sums to zero, the model decided the class does not exist, which is a rubric problem: the description does not match the inputs, or there is no positive example in the few-shot set.

Three or four prompt edits aimed at the top cells usually lift macro F1 more than a frontier-model swap. Read the matrix first, and swap models only after the prompt is doing what it can.

Run a cascade, not a judge on every call

Most classifier workloads do not need a frontier model on every call. The pattern that holds up is a cheap deterministic floor plus a calibrated judge for the residual.

• The deterministic floor is regex, contains-checks, exact match on a keyword set, and schema validation. For intent and sentiment, a regex on high-signal keywords routes 30 to 60 percent of traffic without calling a model.
• The judge handles what the floor cannot decide, configured with the class taxonomy in plain language and a few contrastive examples, calibrated against the labeled set.

Three rules keep the judge honest: calibrate before shipping and set per-class thresholds against the cost profile (high precision for fraud, high recall for refunds); pin the judge model and rubric text together in version control, since a rubric change is a model change; and audit the off-diagonal cells monthly, because a new feature shifts the distribution and last month’s clean classes become this month’s confusion pair.

When an LLM is the wrong classifier

When you have plenty of labeled data and a fixed label space, a fine-tuned encoder trains in an afternoon, runs in single-digit milliseconds, and usually beats an LLM judge on both accuracy and cost. LLMs win on cold start, open-ended label spaces, and when the rationale has to be human-readable. They lose on high-throughput fixed-class work. Use the cheaper layer when it can do the job.

Three takeaways

• F1 on a balanced set is theater. Score on the production distribution, debug on the oversampled per-class set, and report macro and weighted F1 together. The gap between them is the regression flag.
• Calibrate per class, not globally. LLM confidence scores are miscalibrated by default and skewed by class, so reliability diagrams on production data are the only honest input to a threshold.
• Read the confusion matrix before changing the model. The off-diagonal cells point at prompt fixes that lift macro F1 more than any model swap.