Fine-Tuning mBART-50 with LoRA on SageMaker: How We Replaced GPT-4.1 in a Production Translation System

Every translation in this system passes through two steps: a neural model generates the raw linguistic output, then a context layer injects domain terminology, translation memory, and project-specific instructions. Step 1 sets the ceiling — a weak baseline means Step 2 has more errors to correct, and no amount of context engineering fixes a fundamentally poor translation.

This article is about replacing Step 1. We fine-tuned a 610-million-parameter open-source model on 48,000 production translations using LoRA, achieving BLEU 41.8 against GPT-4.1’s BLEU 15.58 — a +168% improvement — at $0.001 per request instead of $0.05, with latency under 500ms P95.

The full picture is more nuanced: GPT-4.1 fine-tuned leads on COMET (0.8504 vs 0.8126), the neural metric measuring semantic adequacy. That gap drives the production routing strategy: 80% of traffic goes to the self-hosted model, 20% stays on GPT-4.1 for the complex segments where semantic fidelity matters most. Blended cost: ~$0.01/request — an 80% reduction.

This article covers the fine-tuning pipeline for the translation model itself — not the quality evaluator (covered in Article 3: AIQE), but the model whose output AIQE measures.

Where This Fits

The translation system operates across three connected components: AIQE measures quality (Article 3), Attribution Analysis ranks hypotheses about what drives quality changes (Article 1), and H2H Testing validates those hypotheses in controlled experiments (Article 2). Articles 1–3 covered measurement and experimentation. This article covers the model being measured.

That model sits in a two-step pipeline:

• Step 1 — A fine-tuned neural machine translation model generates the raw translation. This is what this article covers.
• Step 2 — A context engineering layer injects domain terminology, translation memory matches, and project-specific instructions. Future article.

The key property of this design: Step 1 is the linguistic baseline. A stronger baseline means Step 2 has fewer errors to correct — improvements in Step 1 propagate through the entire output.

Why Self-Host

The cost case is straightforward — $0.001 vs $0.05 per request speaks for itself. Three less-obvious reasons matter more for production quality.

Terminology consistency. GPT-4.1 translates correctly but freely — it may render a product feature name differently across segments unless explicitly constrained each request. A model trained on five years of human translations from the same product corpus has internalized the terminology distribution. This shows directly in BLEU: the metric rewards n-gram overlap against reference translations, so a model that learned the exact phrasing translators used outperforms one generating semantically equivalent but lexically different output.

Controlled vocabulary. Human translators made deliberate choices about how to render UI verbs, formality levels within product areas, and romanization of proper nouns. Those choices are implicit in the training corpus — the model absorbs them through training, not through prompts. You cannot reliably inject that knowledge into an API call at inference time.

The retraining loop. When every post-edited translation flows back into the next training cycle, the model improves continuously from production corrections. An API model you don’t control cannot do this.

The tradeoff: self-hosting requires infrastructure, experiment tooling, and a validation discipline to ensure new versions don’t regress. The rest of this article covers how each of those was addressed.

Model Selection: mBART-50

Two open-source models were evaluated:

The license difference was decisive: NLLB-200’s CC BY-NC 4.0 restricts commercial use. mBART-50 under Apache 2.0 has no such constraint.

mBART-50 is a sequence-to-sequence transformer trained on 50 languages. The encoder-decoder design is native to translation: the encoder builds a representation of the source, the decoder generates target tokens autoregressively. This matters for fine-tuning — the model already understands translation structure; fine-tuning teaches it how this product translates, not what translation is.

Language code format. mBART-50 uses specific language tokens: source en_XX, target ja_XX. The decoder requires a forced beginning-of-sequence token for the target language — for Japanese, forced_bos_token_id: 250004. Getting this wrong produces silent failures: the model generates text in the wrong language or code-switches mid-segment.

Beam search at inference. Generation uses 5-beam search rather than greedy decoding. For short UI strings, the quality difference is measurable but small. For longer legal or marketing segments where early token choices constrain later ones, beam search consistently produces better outputs. The latency cost is acceptable at under 500ms P95.

The Data Foundation: 48,000 Production Translations

Model quality is bounded by data quality. The training corpus is 48,052 human translations from our translation management system (TMS) — professional translator output across UI strings, job descriptions, legal disclosures, and marketing copy, produced between 2020 and 2025.

This is not scraped web data or automatically aligned bilingual text. These are the exact translations serving production, for the exact domains the model will encounter. That provenance matters for two reasons:

Domain specificity. General-purpose translation models trained on WMT data see balanced text-type distributions. Our training set is weighted toward the actual production distribution — short, placeholder-heavy UI strings dominate, with some longer legal and marketing segments. Fine-tuning on this corpus teaches the model those specific patterns rather than averaging over all genres.

Implicit terminology encoding. Five years of human translations encode five years of translator decisions. The model learns that certain product names get romanized a specific way, that certain UI verbs map to specific Japanese equivalents, and that formality is consistent within a product area.

Data split. The split is 65% training (31,242 samples), 17.5% validation (8,409 samples), 17.5% test (8,401 samples). The test set is intentionally large — the reasoning is covered in the Evaluation Methodology section below.

Data extraction. The pipeline queries the production database for source-target pairs, filters empty strings and exact duplicates, runs language detection to remove misclassified rows, and uploads split CSVs to S3. This step runs on CPU and takes approximately 5 minutes.

Each training example is a (source, target) pair. The corpus spans four content types with very different characteristics:

# UI strings — short, high placeholder density
source: "Your application has been viewed by {count} employers"
target: "あなたの応募書類は{count}社の雇用主に閲覧されました"

# Job descriptions — medium length, formal register
source: "We are looking for a motivated software engineer to join our team."
target: "やる気のあるソフトウェアエンジニアをチームに迎えたいと思っています。"

# Legal / compliance — long sentences, precise terminology
source: "By submitting this form you agree to our Terms of Service and Privacy Policy."
target: "このフォームを送信することにより、利用規約とプライバシーポリシーに同意したものとみなされます。"

# Marketing copy — idiomatic, tone-sensitive
source: "Find your next great opportunity."
target: "次の大きなチャンスを見つけよう。"

UI strings dominate the distribution (~70% of the corpus), which is why BLEU improvements on this dataset translate directly to visible consistency gains in the product.

LoRA: Training 1.2% of Parameters

Full fine-tuning of mBART-50 updates all 610 million parameters. LoRA (Low-Rank Adaptation) injects trainable low-rank matrices into the model’s linear layers and updates only those — approximately 7.3 million parameters, or 1.2% of the total.

The mechanism. For a weight matrix W of shape d×k, LoRA adds two small matrices: A of shape d×r, B of shape r×k, where r ≪ min(d, k). During forward pass, the effective weight is W + AB. Only A and B are trained; W stays frozen. The rank r is the key hyperparameter: lower r means fewer trainable parameters and less adaptation capacity, higher r risks overfitting to training idioms.

Our configuration:

strategy: lora
peft_config:
  r: 8                      # Rank → 7.3M trainable params (1.2% of 610M)
  lora_alpha: 16            # Scaling factor = alpha / r = 2.0
  lora_dropout: 0.1         # Dropout on LoRA matrices
  target_modules: all-linear   # Applied to every linear layer, not just attention

Translated into code using HuggingFace PEFT:

from peft import LoraConfig, get_peft_model, TaskType

lora_config = LoraConfig(
    task_type=TaskType.SEQ_2_SEQ_LM,
    inference_mode=False,
    r=8,
    lora_alpha=16,
    lora_dropout=0.1,
    target_modules="all-linear",
)
model = get_peft_model(model, lora_config)

Why all-linear instead of q_proj, v_proj? Standard LoRA targets only attention projection matrices. For translation, the decoder’s cross-attention to encoder representations is critical — but so are the feed-forward layers that store the vocabulary mapping from source to target concepts. Restricting LoRA to attention leaves FFN layers frozen, limiting domain adaptation to the attention mechanism. Applying to all linear layers increases trainable parameters from ~0.3% to ~1.2%, while remaining parameter-efficient.

Why r=8? With 48K training samples, higher rank (r=16 or r=32) risks memorizing training set phrasing rather than generalizing. r=8 is a conservative choice that keeps trainable parameters at 1.2% while giving the model enough capacity to adapt domain vocabulary.

Training hyperparameters:

learning_rate: 5e-5          # Higher than full fine-tuning; LoRA is stable at higher LR
num_train_epochs: 3
per_device_train_batch_size: 4
gradient_accumulation_steps: 2  # Effective batch: 8 samples
warmup_steps: 50
weight_decay: 0.01
max_grad_norm: 1.0
fp16: true                   # Mixed precision; ~50% GPU memory reduction
generation_num_beams: 5      # Beam search during eval generation
eval_strategy: epoch         # Validate once per epoch
load_best_model_at_end: true # Restore lowest val-loss checkpoint after training
metric_for_best_model: eval_loss

The validation set is used exactly once per epoch to compute held-out loss. After all three epochs, the trainer automatically restores the checkpoint with the lowest validation loss — not necessarily the final epoch. This guards against the model slightly overfitting in the last epoch on a 48K corpus.

Post-training weight merge. After training, model.merge_and_unload() folds the LoRA adapter back into the base weights. The merged model is a standard mBART-50 checkpoint — no PEFT dependency at inference time, loads identically to any other HuggingFace seq2seq model.

Config-Driven Experiments: From 2 Days to 3 Hours

Before the current framework, running a new experiment — adjusting rank, changing data split, trying different learning rates — required manual coordination: provision a GPU instance, run the training script, manually copy metrics to a spreadsheet, compare against previous notes. Two to three days per experiment, mostly waiting.

The framework reduces that to 3–4 hours end-to-end through three design choices.

Single-Job Sequential Pipeline

Four steps execute inside one SageMaker Training Job container:

No manual handoffs exist between steps. Submit at 9am, read results at noon. Total cost per experiment: $1.34 (60 minutes on ml.g4dn.xlarge).

Pydantic + Hydra Configuration

Every experiment parameter lives in a YAML file. Launching a new experiment with different LoRA rank requires one file, no code changes:

# configs/experiment/mbart_lora_ja_r16.yaml
defaults:
  - base_experiment
  - model: mbart
  - data: ja

strategy: lora
peft_config:
  r: 16          # Changed from 8
  lora_alpha: 32 # Scaling = 2x rank
  lora_dropout: 0.1
  target_modules: all-linear
learning_rate: 5e-5
num_train_epochs: 3

Configuration classes are defined with Pydantic, which validates types and ranges at startup — before the GPU instance spins up:

class LoRAConfig(BaseModel):
    r: int = Field(default=8, ge=1, le=64)
    alpha: int = Field(default=16, ge=1)
    dropout: float = Field(default=0.05, ge=0.0, le=0.5)
    target_modules: List[str] = Field(default_factory=lambda: ["q_proj", "v_proj"])

A missing required field or an out-of-range value fails immediately at config parse time, not 55 minutes into a training run.

Checkpoint Resume for Evaluation

The evaluation step — computing BLEU and COMET over 8,400 samples — takes about 15 minutes. GPU preemption or network timeout would previously mean restarting from zero. The checkpoint system saves translation outputs every N batches to S3; evaluation resumes from the last saved state on restart:

if checkpoint_manager and checkpoint_manager.has_checkpoint():
    saved_state = checkpoint_manager.load()
    predictions = saved_state["predictions"]
    start_batch = saved_state["batch_index"]
else:
    predictions = []
    start_batch = 0

for batch_idx in range(start_batch, len(batches)):
    ...
    if checkpoint_manager and batch_idx % checkpoint_frequency == 0:
        checkpoint_manager.save({"predictions": predictions, "batch_index": batch_idx})

Results log automatically to MLflow with all hyperparameters, metrics, and generated translations as queryable artifacts. Each run is reproducible from config alone.

That covers the what and how of running experiments. The next section covers where — the AWS infrastructure that makes a one-command submission actually work.

AWS Infrastructure

The training pipeline runs on AWS SageMaker as a managed Training Job. The choice of managed infrastructure over a self-managed GPU server solves three problems at once: job isolation (no state leaks between experiments), automatic artifact persistence to S3, and spot instance support for cost reduction.

Here’s the full architecture — the sections below explain each component:

The Training Job

Each experiment runs on a single ml.g4dn.xlarge instance — an NVIDIA T4 GPU with 16GB VRAM. Spot instances are enabled by default, giving roughly 70% cost reduction over on-demand. The job has a 4-hour maximum runtime; in practice it finishes in ~80 minutes.

VPC placement is automatic: the submission script discovers the correct SageMaker security group from AWS EC2 by looking for the most recent sagemaker-security-group-* resource provisioned by our Terraform configuration. That security group routes traffic to the internal services the job needs to reach — GitLab (for code), the MLflow tracking server, and the PostgreSQL database that holds training data.

# Auto-discover VPC config from Terraform-managed security group
def get_vpc_config(region="us-east-2"):
    ec2 = boto3.resource("ec2", region_name=region)
    sgs = [sg for sg in ec2.security_groups.all()
           if sg.group_name.startswith("sagemaker-security-group-")]
    # Pick the most recent by name timestamp
    sg = sorted(sgs, key=lambda x: x.group_name.split("-")[-1])[-1]
    subnets = [s.id for s in ec2.subnets.filter(
        Filters=[{"Name": "vpc-id", "Values": [sg.vpc_id]}])]
    return {"subnets": subnets, "security_group_ids": [sg.id]}

Container and Code Deployment

The training container is based on AWS Deep Learning Containers (DLC), which ship PyTorch 2.2.0 with CUDA 12.1 pre-installed. Additional dependencies — HuggingFace Transformers, PEFT, sacreBLEU, COMET, Hydra, MLflow — are installed on top:

FROM 763104351884.dkr.ecr.us-east-2.amazonaws.com/pytorch-training:2.2.0-gpu-py310-cu121-ubuntu20.04-sagemaker

RUN pip install --no-cache-dir \
    transformers==4.44.0  datasets==2.19.0  peft==0.11.0 \
    accelerate==0.30.0    sacrebleu==2.4.0  unbabel-comet==2.2.2 \
    hydra-core==1.3.2     mlflow==2.14.0

COPY sagemaker_entrypoint.sh /opt/ml/code/
ENV SAGEMAKER_PROGRAM=sagemaker_entrypoint.sh

The container contains only the entry point script — not the training code. Rather than baking code into the image, the entry point clones the repository from GitLab at job start using a token stored in AWS Secrets Manager:

# sagemaker_entrypoint.sh (simplified)
git clone "https://oauth2:${GITLAB_TOKEN}@${REPO_HOST}/boxxo-loom.git" /opt/ml/code/repo
cd /opt/ml/code/repo/finetuning

# Checkout the exact commit that submitted the job
git checkout "$GIT_COMMIT"

# Build the Hydra command from environment variables
CMD="python pipeline.py experiment=$EXPERIMENT_NAME"
[ -n "$CHECKPOINT_URI" ] && CMD="$CMD pipeline.checkpoint_uri=$CHECKPOINT_URI"
[ -n "$HYDRA_OVERRIDES" ] && CMD="$CMD $HYDRA_OVERRIDES"
exec $CMD

The submission script auto-detects the current git branch and commit hash before submitting, and passes them as environment variables to the container. This means every training run in MLflow is linked to an exact commit — the experiment is fully reproducible by anyone with repo access.

For local development with uncommitted changes, a second mode exists: the submission script creates a clean tarball of the repository (excluding .git, venv, __pycache__, outputs), uploads it to S3, and passes the S3 URI as the source_dir for the estimator. This way you can iterate on code without pushing to the remote.

Data Flow and Job Submission

Training data and model artifacts flow through S3 along a fixed path structure:

s3://bucket/translation-finetuning/
  data/ja/
    train.csv          (31K samples, 3.6 MB)
    val.csv            (8.4K samples, 979 KB)
    test.csv           (8.4K samples, 982 KB)
  models/{experiment}/
    model.tar.gz       (merged mBART-50 checkpoint, ~2.3 GB)
  checkpoints/{experiment}/
    checkpoint-*.json  (evaluation resume state)

SageMaker mounts training data at /opt/ml/input/data/training/ and expects the final model at /opt/ml/model/. The pipeline config detects SageMaker vs local mode from standard environment variables:

def load_from_env(self):
    if sm_model_dir := os.getenv("SM_MODEL_DIR"):
        self.output_base_dir = sm_model_dir    # /opt/ml/model/
    if sm_channel := os.getenv("SM_CHANNEL_TRAINING"):
        self.training.data.path = sm_channel   # /opt/ml/input/data/training/
    if checkpoint := os.getenv("CHECKPOINT_URI"):
        self.pipeline.checkpoint_uri = checkpoint

Launching a new training run from the command line:

python scripts/submit_training.py \
  --language ja \
  --config mbart_lora_ja \
  --instance-type ml.g4dn.xlarge \
  --use-spot

The script validates preflight conditions (git clean check, S3 bucket access, ECR image exists), constructs the sagemaker.estimator.Estimator, and submits. On success it prints the AWS console URL for the job:

✅ Training Job Submitted: translation-finetuning-mbart-lora-ja-20260123-151200
   AWS Console: https://us-east-2.console.aws.amazon.com/sagemaker/home#/jobs/...
   Estimated time: ~80 min (train 60 + eval 15 + register 2)

MLflow Model Registry

After evaluation, the pipeline registers the model to MLflow — the internal model registry. Registration is conditional: if only_if_improved=true, the model is only registered when BLEU improves over the current production baseline. Registration logs all hyperparameters, metrics, and evaluation artifacts (per-sentence COMET scores, generated translations, statistical test results) as a queryable run:

with mlflow.start_run(run_name=f"{experiment_name}-{timestamp}"):
    mlflow.log_metrics({"bleu": 41.8, "comet": 0.8126, "bleu_improvement": 6.6})
    mlflow.log_params({"r": 8, "lora_alpha": 16, "target_modules": "all-linear", ...})
    mlflow.log_artifacts(str(model_path), artifact_path="model")
    mlflow.log_artifacts(str(eval_output_dir), artifact_path="evaluation")

    if cfg.mlflow.register_model:
        mv = mlflow.register_model(model_uri, model_name)
        print(f"✅ Registered: {model_name} v{mv.version}")

The git-clone-at-runtime pattern is the key to reproducibility. Every MLflow run records the exact commit hash. Any run can be reproduced by checking out that commit and resubmitting with the same YAML config.

Cost & Performance

Monitoring Stack

Evaluation Methodology

A single metric is insufficient for production deployment decisions. The validation framework runs three tiers in sequence, each gating the next:

Each tier catches a different failure mode. Tier 1 detects metric regression on the full test distribution. Tier 2 catches nuanced quality issues — the nine AIQE dimensions (accuracy, terminology, style, etc.) surface problems that aggregate BLEU/COMET miss. Tier 3 catches the residual: correct-by-metrics but wrong-by-native-speaker failures that both automatic metrics and AI evaluation miss.

Tier 1: BLEU and COMET

Both metrics run automatically at the end of every training job.

**BLEU** (Bilingual Evaluation Understudy) measures n-gram precision against reference translations. For Japanese, word-level tokenization is inappropriate — there are no word boundaries in running text. The implementation uses character-level tokenization via sacreBLEU’s tokenize='char' setting, selected automatically when Japanese characters are detected:

def compute_bleu(predictions, references):
    # Detect CJK content → switch to char-level tokenization
    sample = " ".join(predictions[:20])
    has_cjk = bool(re.search(r'[぀-鿿]', sample))
    tokenizer = 'char' if has_cjk else '13a'

    return corpus_bleu(
        predictions,
        [references],
        tokenize=tokenizer,
        use_effective_order=True,  # Handles short strings correctly
    )

**COMET** (Crosslingual Optimized Metric for Evaluation of Translation) uses multilingual encoders trained on human direct assessment scores from WMT. Unlike BLEU, it takes the source string as input alongside the hypothesis and reference, making it sensitive to meaning preservation in ways n-gram overlap cannot detect. The specific model is Unbabel/wmt22-comet-da, the top-ranked reference-based metric from WMT22.

Per-sentence COMET scores are retained for each evaluation. A paired t-test on those scores — fine-tuned model vs production baseline, matched by segment — determines whether any observed improvement is statistically significant:

from scipy.stats import ttest_rel

t_stat, p_value = ttest_rel(finetuned_scores, baseline_scores)
improvement = (mean(finetuned_scores) - mean(baseline_scores))
is_significant = p_value < 0.05

The 8,400-sample test set was sized to give statistical power > 0.99 for detecting Δ = 0.01 COMET at p < 0.05. This threshold was chosen because production A/B tests (Article 2) consistently show that COMET improvements below 0.01 don’t translate to detectable human quality differences.

Beyond BLEU and COMET, the evaluation step automatically flags degenerate model behavior: empty translations, repeated outputs (same generated text for multiple different inputs — a sign of model collapse), and source/target length ratio outliers. These signals are logged to MLflow as artifacts and reviewed before the model is considered for Tier 2.

Tier 2: AIQE

AIQE (Article 3) scores translations on nine dimensions: accuracy, terminology, style, readability, consistency, localization, placeholders, reference material, and language quality. An aggregate score below 80 blocks the model from Tier 3 review.

For a new model version, AIQE runs on 10–20% of production traffic in canary deployment, or on a fixed development sample of ~1,000 segments. The canary approach has the advantage of testing on the actual production distribution, including edge cases not represented in the test set.

Tier 3: Human QA

Expert reviewers evaluate 100–200 segments on seven dimensions including terminology accuracy, post-edit effort, fluency, and consistency. This tier primarily catches failure modes that both metrics and AIQE miss: idiomatic failures specific to the product domain, or segments that score well numerically but read awkwardly to a native speaker.

Deployment gate. A model passes Tier 3 if it shows no critical errors and reduces post-edit effort by at least 30% compared to the outgoing production version. That 30% threshold is not arbitrary — it’s the point at which translators report that working with the model output is faster than translating from scratch.

Results

The initial Japanese experiment compared four conditions on the full 8,410-sample test set:

Four observations worth unpacking.

The BLEU gap reflects terminology learning, not generic quality. GPT-4.1 at BLEU 15.58 vs mBART fine-tuned at 41.8 is a +168% improvement. BLEU rewards n-gram overlap against reference translations — and our references are the production human translations that mBART was trained on. GPT-4.1 translates correctly but with different vocabulary choices; mBART fine-tuned learned to match the translators’ exact phrasing. That’s not a trivial distinction for a product where UI consistency across thousands of strings matters.

mBART fine-tuned leads on BLEU; GPT-4.1 fine-tuned leads on COMET. COMET measures semantic adequacy using multilingual encoders trained on human judgments. GPT-4.1 fine-tuned reaches 0.8504 — higher than mBART fine-tuned’s 0.8126. The gap is real: GPT-4.1’s general language model capacity gives it an edge on meaning preservation, especially for longer, more complex segments. But mBART’s COMET of 0.8126 is still above GPT-4.1 raw (0.8084), confirming that domain fine-tuning improves semantic fidelity as well as lexical precision. The tradeoff is clear: better meaning on rare complex segments (GPT-4.1 FT) versus superior phrasing consistency across the full production distribution (mBART FT), at 50× lower cost and 4–10× lower latency.

mBART baseline (no fine-tuning) at BLEU 35.20 already beats GPT-4.1 Fine-tuned (25.31). Even without any domain training, the seq2seq architecture designed for translation outperforms a fine-tuned autoregressive LLM on lexical precision. The architecture prior matters — and fine-tuning mBART on 48K production examples pushes that further to 41.8.

The cost and latency gap dwarfs the COMET gap. mBART fine-tuned at $0.001/request vs GPT-4.1 at $0.050/request is a 98% cost reduction. At 100K requests/month, that’s $4,900 in monthly inference savings. The COMET difference of 0.0378 points (0.8504 vs 0.8126) is meaningful for complex segments but not uniformly impactful across all content types.

Production Routing: Hybrid 80/20

The results directly informed the deployment strategy. Rather than a full swap to mBART, the production system uses hybrid routing:

• 80% → mBART fine-tuned — standard product strings, UI copy, job descriptions, short legal segments. High BLEU consistency is the critical metric here.
• 20% → GPT-4.1 — complex, context-heavy segments where the 0.0378 COMET advantage matters: long legal disclosures, idiomatic marketing copy, segments with no translation memory coverage.

Blended cost: ~$0.01/request (80% × $0.001 + 20% × $0.050) vs the current $0.05/request baseline — an 80% cost reduction while preserving GPT-4.1 quality on the segments that most benefit from it.

Production Deployment Path

Once a model passes all three validation tiers, the deployment sequence is:

1. MLflow run is tagged for deployment; model artifacts stored in S3.
2. INFSP (the internal ML serving platform) picks up the tag and deploys the checkpoint to Kubernetes GPU pods — typically within 15 minutes, with zero manual steps.
3. The Boxxo translation API updates its routing weights to begin serving traffic: starting at 5% canary, expanding to 50%, then 100% following the rollout schedule in the AWS Infrastructure section above.
4. SageMaker Console, MLflow, and Datadog continue monitoring: GPU utilization, per-request latency, and COMET score distribution on live traffic.

An auto-rollback fires if daily COMET on live traffic drops more than 5% relative to the previous version.

On the three-tier framework: The results above cover Tier 1 (automatic metrics on the full test set). Tier 2 (AIQE scoring on canary traffic) and Tier 3 (Human QA on 100–200 segments) are running as part of the current production rollout and are not yet complete. The deployment described here is the 5% canary phase; full 100% deployment is gated on Tier 2 and Tier 3 passing.

The Feedback Loop

Step 1 alone doesn’t close the loop. What makes the system self-improving is the connection between production output and training data.

When translators post-edit Step 1 output, they generate labeled examples — the original machine output paired with the corrected human version. Each correction is a signal about what the model gets wrong. Those post-edit pairs flow back into the training corpus for the next fine-tuning cycle. Retraining on the expanded dataset runs through the full three-tier validation before any new version is promoted.

The 48K base grows with each cycle. AIQE detects whether a new version is better or worse. H2H testing confirms that the signal is real. The fine-tuning pipeline acts on the signal H2H confirms.

The measurement, hypothesis, experiment, and improvement layers are only useful in combination — and this is where they connect.

Next: Step 2 — how context engineering layers terminology constraints, translation memory, and project instructions on top of the Step 1 baseline.

References

• mBART-50: Multilingual Translation with Extensible Multilingual Pretraining and Finetuning
• LoRA: Low-Rank Adaptation of Large Language Models
• sacreBLEU: A Call for Clarity in Reporting BLEU Scores
• COMET: A Neural Framework for MT Evaluation
• HuggingFace PEFT Library