Head-to-Head Testing for AI Translation Quality: How We Validate Model Changes Before They Hit Production

A Head-to-Head testing system does one thing: takes a hypothesis — “configuration B produces higher-quality translations than configuration A” — and returns a statistically-verified answer before any change reaches production.

The attribution analysis system generates those hypotheses. It surfaces candidates ranked by causal confidence. But a high-confidence hypothesis is still a hypothesis. The Head-to-Head test is the gate that turns it into a decision.

This is step 3 of a three-step quality improvement loop:

1. Observe — production data: which configurations ran, what quality scores came back
2. Attribute — causal analysis: which settings drive quality differences (Article 1)
3. Test — controlled experiments on the highest-confidence candidates ← this article

Why Attribution Alone Isn’t Enough

The attribution analysis system uses causal methods to separate correlation from effect. But even after causal adjustment, the output is a probability estimate. A hypothesis ranked at 87% confidence still has a 13% chance of being noise.

More importantly, attribution operates on observational data. The configurations were never randomly assigned. The causal adjustment methods all make assumptions — and those assumptions can be violated.

What we need is a controlled experiment: same source content, both configurations run on the same segments. That eliminates confounding entirely. The problem is that we can’t randomize live production jobs. So we run the controlled experiment offline, against a defined set of source segments, with both configurations executing in parallel.

What Can Be Tested

To understand what can be tested, start with the pipeline itself — every factor type in the table below corresponds to a specific node in this diagram:

Step 1 (blue) uses a fine-tuned NMT model — purpose-built for linguistic accuracy, no context injection. Step 2 (optional) is LLM post-editing: a context prompt is assembled from the context version template, retrieved Translation Memory (TM) segments, and glossary terms, then passed with the Step 1 output to a configurable LLM. Iterative refinement can loop Step 2 multiple times. Quality Evaluation (purple) scores the final output with COMET and AIQE — these scores are what the H2H analysis runs on.

Note the TM bypass: segments with a TM match at ≥0.99 similarity skip the LLM entirely. Both arms get the same cached result, making them uninformative for the comparison — which is why they are excluded from all metrics (more on this in Stratified Analysis and Honest Limitations).

The factor type is declared at test creation and determines which single configurable node above differs between control and treatment:

One factor type per test. Testing multiple dimensions simultaneously re-introduces confounding — exactly what we built the attribution system to avoid.

In every test, the control arm is the configuration currently in production for the target locale and content type. The treatment arm is the candidate change. All other pipeline parameters are held constant between arms.

For CONTEXT_VERSION tests, the system captures a snapshot of each selected context version’s content at test creation time. What gets tested is the exact prompt template and style rules as they existed when you clicked “Run” — not whatever those versions contain when results come back an hour later. This is what makes context version results reproducible and auditable.

Data Sources

Every test requires a set of source segments to translate. Two sources are available.

BENCHMARK — a curated, stable set of segments maintained specifically for cross-test comparability. Coverage is balanced across content types and source lengths. Because the benchmark set is fixed and maintained independently of production traffic, results from different tests are directly comparable: a COMET improvement of +0.02 on the benchmark in March means the same thing as a +0.02 improvement in August. The benchmark is the default for most tests.

PRODUCTION_SAMPLE — segments sampled from recent production jobs, filtered to match the test’s target locale and content type. Closer to the actual distribution the configuration will encounter in production. The tradeoff: production samples are drawn from jobs that completed successfully through the current configuration. Jobs that failed, timed out, or were routed differently are not represented — a selection bias that matters when the hypothesis involves a configuration change that affects failure rates.

The data source is set at test creation and is fixed for that test. A Pilot and its promoted Full test use the same source type.

Two Stages: Pilot Then Full

Every hypothesis goes through two test stages, in sequence.

Pilot test (n=100): a fast screening gate designed to rule out hypotheses that show no signal even at moderate effect sizes. At n=100 per arm (α=0.05), the Pilot has ~80% power to detect Cohen’s d ≈ 0.28 but is underpowered for effects below that threshold. This is by design — the Pilot is not trying to detect small effects; it is trying to quickly eliminate hypotheses that fail a moderate-effect screen before committing to Full-scale execution.

The promotion decision uses a deliberately relaxed rule, not the standard ±0.10 composite threshold. At n=100, the composite threshold often lacks the statistical power to be meaningful.

Directional consistency reinforces the case: COMET and AIQE both trending positive — even without individual significance — is a meaningful Pilot signal.

Full test (n=1000): the definitive test. At n=1000, the test has adequate power to detect effects as small as Cohen’s d ≈ 0.09. The standard ±0.10 composite threshold applies, and winner determination is authoritative.

The human reviewer makes the final ship/no-ship call. A “treatment wins” composite score is the default recommendation — not an automatic approval. Reviewers override it when: a high-priority stratum shows statistically significant degradation; human grading contradicts the automated metrics; or the effect size is real but too small to justify the operational cost of the configuration change. A “no clear winner” at Full with directionally consistent signals near the effect size cutoffs is a judgment call, not an automatic discard.

Pilot → Full effect size comparison Effect sizes typically shrink from Pilot to Full as noise averages out — a Cohen’s d of 0.4 at n=100 converging to 0.25 at n=1000 is expected, not suspicious. A large Pilot effect collapsing to near zero at Full is a red flag: the Pilot result was likely noise from small-n variance. Conversely, a weak Pilot signal (d ≈ 0.15) that clarifies at Full is exactly the case the two-stage design exists to catch — effects too small for Pilot detection but real enough to act on.

The UI makes this a two-click workflow. From the Pilot results page, a single “Promote to Full Test” button creates the Full test — copying all configuration from the Pilot, linking the two via a shared testGroupId. The Full test results page shows the Pilot results alongside for direct comparison.

Before committing to either stage, the create-test UI runs a live cost estimate: sample size × 2 arms × token estimates × current LLM pricing, with the formula breakdown shown. For a Pilot at n=100 using gpt-4o-mini for AIQE scoring, this is typically $2–5 and 3–8 minutes. For a Full test at n=1000, $20–50 and 30–90 minutes.

The Three Metrics

Every translation in the test gets scored on three metrics, each capturing a different dimension of translation quality.

**BLEU** — n-gram precision between the translation and a reference string. Reference strings come from previously approved translations for those same source segments in the TM; segments with no prior approved translation are skipped for BLEU. Fast, deterministic, no API call. Its weakness: it measures surface form, not meaning. A translation can have low BLEU and be semantically correct; it can have high BLEU and be wrong. We include it as a baseline cross-check, not a primary signal.

**COMET** — a neural MT evaluation model. It takes source text + hypothesis translation and returns a quality score without needing a human reference translation. (This is what “reference-free” means: no approved reference string is required — not that COMET runs locally. It calls an external scoring API, which is why API failures are a documented failure mode.) Trained on human preference judgments. Score range 0–1. Captures fluency and adequacy jointly. More sensitive than BLEU to meaning-level quality differences.

AIQE (AI Quality Evaluator) — an LLM-as-judge scorer used here as a metric within each H2H test. Takes source + translation + a scoring rubric, returns 0–100. Interpretable: it can explain its rating. More expensive per segment (one LLM call per translation per arm), and its biases track the judge model’s biases. The judge model is configurable per test and defaults to a fast, cost-effective model (e.g., gpt-4o-mini). (AIQE also exists as a standalone system that learns from H2H test results over time — that’s covered in Article 3. Here it refers specifically to the per-segment LLM scoring call.)

All three metrics produce the same output structure per comparison: control mean, treatment mean, mean difference (treatment − control), p-value, Cohen’s d effect size, significance flag, and 95% confidence interval on the mean difference.

The system sets a dataQuality flag on every completed test — COMPLETE when all COMET scoring succeeded, PARTIAL when some segments failed (rate limits, transient errors), and INCOMPLETE when COMET scoring failed entirely. Human-readable warnings surface on the results page. A test with INCOMPLETE data quality does not produce a winner recommendation; conclusions would be misleading. PARTIAL produces a recommendation with a prominent warning — treated as provisional, since the unscored segments could shift the result if their quality distribution differs from the scored ones.

How We Determine a Winner

Winner determination runs on the automated metrics. The experimental design is paired: each of the n segments runs through both arms, and scores are compared at the segment level. Pairing eliminates segment-level quality variance as a confound — inherently difficult segments contribute equal difficulty to both arms, so only the configuration effect remains. This is what makes the paired t-test valid and gives it substantially more power than a two-sample test at the same n.

For each of the three metrics, the backend computes a paired t-test p-value, Cohen’s d effect size, and 95% confidence interval on the mean difference.

Step 1 — directional signal per metric. The frontend converts each metric’s stats into a single signed value:

signal = sign(Δ) × min(|d|, 1)    when p < 0.05
signal = 0                          when p ≥ 0.05

where Δ = treatmentMean − controlMean and d = Cohen’s d. A non-significant metric contributes nothing. A significant metric scales continuously with effect size — d = 0.05 contributes far less than d = 0.50. Effect sizes above 1 are capped to prevent outlier results from dominating.

Step 2 — weighted composite score. Signals are combined with fixed weights:

normalizedScore = 0.5 × signal_COMET + 0.4 × signal_AIQE + 0.1 × signal_BLEU

Step 3 — winner + confidence label. The normalizedScore maps to a verdict and a confidence tier:

BLEU can never win alone. Its maximum contribution is 0.1 × min(d, 1) — capped at 0.10, exactly at the decision threshold and never above it. COMET alone clears the threshold at d ≥ 0.21; AIQE alone at d ≥ 0.26.

Conflict flag. When significant metrics disagree in direction — COMET pointing to TREATMENT while AIQE points to CONTROL — the results page raises a conflict warning. The composite score still produces a verdict, but conflicting signals reduce confidence in the result.

“No winner” is a valid outcome. It means the evidence isn’t strong enough to justify a production change in either direction. At Pilot scale (n=100), “no winner” is the correct result for small effects — which is why we don’t ship from Pilot results alone.

The Human Grading Layer

Automated metrics are fast and cheap. They are not ground truth. For high-stakes decisions — a major model upgrade, a new context version that will affect millions of translations — we add a human grading step after the automated test completes.

From the results page, an admin sends a sample of test segments to an internal grading queue. Graders evaluate each translation (control and treatment separately, blinded to which is which) using a standard LQA error taxonomy:

From the graded pairs, two additional signals are computed:

LM-LQA score (per arm) converts raw LQA error counts into a 0–100 quality score:

AdjPenalty = (critical×9 + major×5 + minor×1) × 20
LM-LQA     = (−100 × AdjPenalty / SourceWordCount) + 100

The ×20 scaling normalizes raw error density into the same 0–100 range as AIQE, making the two scores directly comparable on the results page. A perfect translation scores 100; each error reduces it proportionally to its severity and the source word count. Higher is better.

Sign test: for each graded segment pair where both arms have been graded, compare the penalty weights — lower penalty wins. The sign test reports what percentage of segment pairs each arm wins outright, and the net win margin — (treatment pair wins − control pair wins) / total graded pairs. No distributional assumption; a clean non-parametric cross-check on the automated metrics.

The human grading layer is not required for every test. It’s the escalation path when automated metrics show a close result or when the decision has high production risk.

Stratified Analysis

Aggregate numbers hide interactions. A treatment that improves average COMET by 0.015 across all segments can simultaneously degrade COMET by 0.08 on a specific stratum that represents a high-priority content type.

Every test produces stratified results across five dimensions:

Within each stratum, the same significance and effect size thresholds apply as in the overall analysis — a stratum is flagged as degraded only when the difference clears both p < 0.05 and a minimum effect size, not on p-value alone. The results page shows a collapsible stratified breakdown: per-stratum winner, per-stratum metrics, and showsDegradation flags.

The design rule: a production change can be blocked by a statistically significant degradation in any stratum, even if the overall result is a treatment win. The aggregate does not override the stratum. This is where we catch cases where a model upgrade is broadly better but regresses on, say, short strings in a specific namespace — a failure mode that aggregate metrics would mask.

A final consideration cuts across both the overall and stratified results: TM hit rate. Segments that hit the Translation Memory at ≥0.99 similarity bypass LLM translation entirely — the same TM result is returned for both arms, making those segments uninformative. They are excluded from BLEU, COMET, and AIQE metrics, and the effective sample size (total segments minus TM hits) is shown on the results page. A test with a high TM hit rate may have a nominal n=1000 but an effective n of 600 — a fact that matters for interpreting confidence intervals in both the aggregate and every stratum. For the Full test power implications of high TM hit rates, see Honest Limitations.

The Execution Architecture

Scoring segments through two configurations in parallel — each requiring an LLM call for AIQE — cannot run synchronously. A Full test at n=1000 takes 30–90 minutes; a Pilot at n=100 takes 5–8.

Execution phases are visible in the UI as the test runs: each phase has a label, a short description of what it does, and an estimated time remaining. A progress percentage (0–100) updates in real time. The list page auto-polls running tests.

The recovery scheduler handles two failure modes automatically: tests stuck in PENDING for 15 minutes (dispatcher never picked up) get re-queued; tests stuck in RUNNING for 60 minutes (workers died mid-execution) get marked FAILED with an error message. A “Retry” button on the results page re-queues a FAILED test without re-entering configuration.

The UI Flow

The execution architecture described above maps directly to four UI states. This section walks through each one.

Create page — the form opens with test identity fields: test name, target language, and an optional description.

Below that, the “What Are You Testing?” section presents factor type as a set of selectable cards. The configuration fields adapt to the chosen factor — for a CONTEXT_VERSION test, two context version dropdowns appear; for a MODEL test, two model selectors. Shared pipeline flags (TM lookup, iterative refinement) sit below the factor-specific fields:

At the bottom of the form, Pilot and Full test options are presented side by side with live cost and duration estimates. The estimate updates as configuration changes — this is a commitment check before an API spend, not decoration:

List page — status board. Filter by stage (Pilot / Full), status (Pending / Running / Completed / Failed / Cancelled), creator, factor type, locale, date range. Running tests show a live progress indicator. Pilot and Full tests that share a test group are visually linked in the same row via an expand toggle — clicking it reveals the related test inline without leaving the list.

Running test — while a test executes, the results page shows a progress bar, current phase label, and estimated time remaining. The page auto-refreshes. A Cancel button is available at any point during execution; cancellation stops the scoring workers and marks the test CANCELLED, preserving whatever partial results exist in the database. When execution completes, the page automatically transitions to displaying full results — no manual navigation needed.

Results page — two main areas sit below the page header.

Test Configuration panel — renders the pipeline as an interactive DAG (TM Lookup → Step 1 → Step 2 → Quality Eval → Iterative), with each node annotated to show which arm (A·Control, B·Treatment) uses which model or context version. Clicking a node highlights its configuration row in the table below. A “Differences only” toggle collapses the table to show just the rows where control and treatment diverge — useful when comparing configurations that share most settings. While a test is running, the active phase is highlighted live. For CONTEXT_VERSION tests, a collapsible “Context content diff” section shows a side-by-side diff of the actual prompt template and style guide content between the two versions.

Results Analysis — three layers of increasing detail:

Overall results card: the single-screen answer. Winner badge, composite evidence score, per-metric breakdown (p-value, Cohen’s d, mean difference with 95% CI). Data quality warning if metrics are incomplete. TM hit rate and effective sample size. Cost and duration of the run.

Stratified results: collapsible section. One row per dimension × value stratum. Per-stratum winner, per-stratum metrics, degradation alert flag. This is where you look if the overall result is unexpectedly close or if you’re making a high-risk production decision.

Sample-level data: for Full tests, the complete 1,000-segment dataset is available for download as CSV — source text, control translation, treatment translation, and per-segment COMET, BLEU, and AIQE scores. This is the audit layer: if the overall result looks suspicious (a COMET win without a corresponding AIQE gain, for instance), the CSV lets you inspect individual segments to understand why.

From the Pilot results page, “Promote to Full Test” copies all configuration into a new Full test — only the name is customizable. The Full test results page shows the Pilot results alongside for direct comparison of effect sizes.

Honest Limitations

COMET has its own error mode. Trained on general MT preference data. For domain-specific content (job descriptions, legal notices), its calibration degrades. It can favor fluent-but-inaccurate translations. A COMET win that doesn’t hold up in human grading is a known failure mode, not a surprise.

AIQE biases track the judge model. If you’re evaluating a proposed GPT-4o upgrade using GPT-4o as the AIQE judge, the judge has a systematic preference for its own generation style. We partially address this by defaulting to a separate scoring model (e.g., gpt-4o-mini) for evaluation, but the bias is not fully eliminated.

**PRODUCTION_SAMPLE has selection bias.** Sampled segments came from jobs that ran successfully through the existing configuration — see the Data Sources section. If the hypothesis involves a configuration change that affects which jobs succeed, the sample is not representative of the full distribution.

TM hits inflate nominal sample size — and can undermine Full test power. A test with n=1000 but 40% TM hit rate has an effective n=600. The confidence intervals on the results page are computed against the effective n — but someone reading “1000 samples” in the test setup will expect tighter intervals than they actually have. More critically: a 60% TM hit rate reduces effective n to 400, at which point the Full test operates closer to Pilot-level power. The statistical guarantee that Full tests detect Cohen’s d ≈ 0.09 depends on ~1000 effective segments, not 1000 nominal ones. High TM hit rates should be investigated before treating a Full test “no clear winner” as definitive.

Stratified significance tests have uncorrected multiple comparisons. With 5 stratification dimensions and potentially dozens of stratum values, running per-stratum significance tests at α=0.05 without correction inflates the false positive rate — a degradation flag appearing by chance becomes increasingly likely as the number of tested strata grows. The current implementation does not apply Bonferroni or FDR correction. Stratified degradation flags are best treated as signals warranting further investigation, not confirmed statistical findings on their own.

“Better on the test set” ≠ “better on the full distribution.” The BENCHMARK set is curated for coverage across known content types. If it underrepresents a production edge case — say, a specific namespace or plural form that behaves differently — the test won’t catch regressions there. A configuration change can pass with a clean “treatment wins” result and still degrade quality in a long-tail segment class.

What This Feeds Into

Confirmed wins feed into production configuration changes and — via the new production data they generate — back into the attribution analysis system as additional causal evidence.

Rejected hypotheses are equally valuable. A clean negative at adequate power tells the attribution system that the observed correlation in historical data was not causal. That tightens the prior for future runs.

The third component in this loop is the AIQE system — a standalone quality evaluator that is distinct from, but related to, the AIQE metric used inside each H2H test. The per-segment LLM scoring calls in each test produce labeled (source, translation, score) pairs; the standalone AIQE system learns from this accumulating data to improve its quality models over time. That’s the next article.

Part of a three-article series on AI-powered translation quality. Article 1: Attribution Analysis | Article 3: AIQE (coming soon)