What 4 Years of Fine-Tuning LLMs in Production Actually Taught Me

Not a tutorial. A reckoning.

I didn’t plan to become the person teams call when their fine-tuning jobs blow up.

It happened gradually. A recommendation system at scale. Then an NLP pipeline for a fintech. Then someone asked me to “just fine-tune this LLaMA model real quick.” That was four years ago. Since then I’ve shipped fine-tuned models into production across AWS SageMaker, GCP Vertex AI, and bare-metal GPU clusters. I’ve debugged NCCL hangs at 2am, watched Spot instances evaporate six hours into a run, and once spent three days hunting a data pipeline bug that turned out to be a silent UTF-8 encoding issue corrupting 12% of my training examples.

I’ve touched models from 1B to 70B parameters. I’ve done full fine-tuning, LoRA, QLoRA, RLHF pipelines, DPO, and combinations of all of the above. I’ve worked in healthcare AI, legal tech, fintech, and developer tooling — domains where a confident wrong answer isn’t just annoying, it’s a liability.

This post is what I wish existed when I started. Not the “here’s how to call the Trainer API” post. The other one.

First: The Mental Model That Changes Everything

Most engineers approach fine-tuning as a modeling problem. Pick the right base model, dial in the hyperparameters, get good loss curves, ship.

That framing is why most fine-tuning projects underdeliver.

Fine-tuning is a systems problem. The model is the output of a pipeline — a data pipeline, a training infrastructure pipeline, an evaluation pipeline, a deployment pipeline. Every one of those pipelines can fail quietly and make your model worse in ways that aggregate loss curves will never surface.

The engineers I’ve seen consistently ship good fine-tuned models are not necessarily the best ML researchers in the room. They’re the ones who treat every component of the system with production engineering discipline. The ones who version their data, harden their checkpointing, and write eval suites before they write training code.

Keep that frame in mind for everything that follows.

1. Your Training Infrastructure Will Fail. Engineer for It.

The single most expensive lesson I’ve learned — measured in actual GPU-hours lost — is that long-running distributed training jobs are not reliable by default. You have to make them reliable.

Here’s an incomplete list of ways I’ve lost training runs:

• AWS Spot instance reclaimed mid-run. Checkpoint interval was per epoch. One epoch scheduled. Zero checkpoints saved.
• GCS sync job silently failed due to an expired service account token. Discovered post-mortem.
• NCCL deadlock on a 16xA100 run. One node fell behind on a data loading bottleneck. The entire job hung. No timeout. No alert. Just silence until I checked the dashboard eight hours later.
• PyTorch OOM during a gradient accumulation step at hour 11 of a 13-hour job. No checkpoint in the last 4 hours because I’d set save steps too conservatively to “save storage costs.”
• Partial checkpoint write caused by a node failure mid-save. The checkpoint loaded without error but produced garbage outputs because the weights were half-updated.

Every single one of these was preventable. Here’s what I now treat as non-negotiable:

Checkpoint every 200–500 steps. Verify every checkpoint. After every save, read the file back, check byte size against a baseline, do a quick load_state_dict sanity check. An unverified checkpoint is not a checkpoint. Use atomic writes — write to a .tmp path, then os.rename(). A partial write caused by a crash mid-save will corrupt your checkpoint silently.

Run a background sync thread, completely decoupled from your training loop. Every N minutes, sync your checkpoint directory to S3 or GCS. Don’t rely on an end-of-job hook — your job may not end gracefully. I’ve recovered runs because the background thread finished its last sync 40 seconds before the instance was reclaimed.

Handle Spot preemption explicitly. On AWS, I use a lifecycle hook that fires on instance interruption and triggers a checkpoint + sync before the 2-minute termination window closes. On GCP, Spot VMs give you a 30-second preemption notice — enough time to save a lightweight checkpoint if your handler is already registered. If you’re running Spot without interruption handling, you’re gambling.

Monitor GPU utilization across all ranks in real time. One underperforming node tanks a distributed job — NCCL barriers make every rank wait for the slowest one. I use dcgm-exporter with Prometheus and Grafana. If any rank drops below 75% sustained utilization, I get an alert. Nine times out of ten, it's a data loading bottleneck or a storage I/O issue that I can fix mid-run before it compounds.

Set hard timeouts on NCCL operations. NCCL_TIMEOUT defaults to 30 minutes. On most well-functioning clusters that's fine. In mixed-instance environments or jobs with heterogeneous data loading, I've had legitimate all-reduce operations take longer. Know your expected operation times and set timeouts that are strict enough to catch hangs but loose enough not to fire on slow-but-valid operations.

Infrastructure work isn’t glamorous. Nobody writes it up in a paper. But it is the difference between a team that runs 20 experiments a week and a team that runs 4.

2. Your Dataset Is a Production System. Version It, Validate It, Audit It.

The phrase “garbage in, garbage out” is repeated so often it’s lost all meaning. Let me make it concrete.

I spent three weeks trying to coax a model into consistent structured JSON output. I swept learning rates, adjusted LoRA rank, tried different base models, read every ablation I could find. Nothing worked reliably — output consistency plateaued at around 62%.

I finally ran a full audit of my training data. What I found: 23% of my “ideal output” examples had subtly malformed JSON — trailing commas, inconsistent key casing, occasional markdown code fences that had survived preprocessing. The model was learning from inconsistent signal. It was doing exactly what the data taught it.

Two days of data cleaning. Consistency went to 91%. No architecture changes. No hyperparameter tuning.

This is the pattern. The model learns precisely what your data teaches — including the noise, the inconsistencies, and the edge cases where a labeler was tired and shipped something wrong.

Here’s the data discipline I now enforce without exception:

Version your datasets with DVC. Every dataset artifact has a content hash. Every training run records the exact dataset hash it consumed. Every experiment is reproducible from first principles. When a model regresses unexpectedly — and it will — you can bisect data state the same way you bisect code. Without this, debugging data-related regressions is guesswork.

Write your validation suite before you build your dataset. Before any example touches a tokenizer, it goes through a validation layer: schema enforcement, length distribution checks, label balance assertions, deduplication, and encoding validation. I use Great Expectations for the automated layer. But automated checks will not catch everything — I also manually review 200 random examples from every dataset before training. Every time. It takes 45 minutes. It has caught critical issues on four separate projects.

Profile your token length distribution before training. Run your entire dataset through the tokenizer. Plot the distribution. Find out what percentage of examples exceed your max_seq_len. If that number is above 5%, you have a truncation problem. More importantly, understand which content gets truncated. On one project, the "ideal answer" was always at the end of training examples — exactly where truncation hits first. The model never saw the correct output. It learned from the wrong signal for three full training runs before we caught it.

Handle synthetic data with appropriate skepticism. I use synthetic data — at scale, because you often have no other option for rare but important cases. But LLM-generated synthetic data carries the hallucinations, biases, and stylistic patterns of the generator model in concentrated form. I run all synthetic examples through a quality filter (a separate judge model or a trained binary classifier). I keep synthetic data under 35% of any training mix unless I have strong quality evidence. And I never — ever — generate synthetic eval data using the same model family I’m fine-tuning. The benchmark contamination you create will make your entire eval infrastructure worthless.

3. Distributed Training Complexity Is Nonlinear

Going from single-GPU to multi-GPU feels like a scaling step. It is not. It’s a qualitative change in operational complexity.

The things that don’t matter at all on a single GPU become your primary failure modes at 8 or 16 GPUs:

NCCL is not self-configuring. NCCL_SOCKET_IFNAME, NCCL_IB_DISABLE, NCCL_P2P_DISABLE, NCCL_SHM_DISABLE — these environment variables control how NCCL discovers and uses network interfaces. Get them wrong and you'll get either silent performance degradation (NCCL falls back to a slower transport) or deadlocks. In environments without InfiniBand, always set NCCL_IB_DISABLE=1. When debugging NCCL issues, start with NCCL_DEBUG=INFO and read every line of the output. There are no shortcuts.

Know the real tradeoffs between DeepSpeed ZeRO and FSDP. I’ve used both extensively in production. ZeRO-3 is more aggressive — it shards optimizer states, gradients, and parameters across all ranks, giving you the lowest peak memory per GPU. FSDP has tighter PyTorch integration and more predictable behavior, but doesn’t push as far on memory efficiency. My rule of thumb: FSDP for jobs under 32 GPUs on homogeneous hardware. ZeRO-3 for larger jobs or memory-constrained configurations. The ZeRO-Infinity extension (NVMe offload) is genuinely useful for 70B+ jobs where you’re hitting memory ceilings — I’ve used it to run fine-tuning that would otherwise be impossible on available hardware.

Gradient checkpointing is always worth profiling. Recomputing activations during backward adds ~30% to compute time but reduces peak activation memory by 60–70%. On a 70B model with a 4096-token sequence, this isn’t optional — it’s the difference between fitting on 8 A100s and needing 16. Profile the memory/compute tradeoff for your specific setup rather than applying it as a default.

Efficient data loading is not optional at scale. At 8+ GPUs, a slow data loader becomes the training bottleneck. Your GPUs sit idle waiting for the next batch. I use multiple DataLoader workers (typically num_workers = 4 * num_gpus), pin memory for GPU transfer, and pre-tokenize and cache datasets to avoid tokenizing on the fly during training. On high-throughput runs, I use WebDataset format with sharded .tar files on GCS — sequential reads on object storage are dramatically faster than random access.

4. Hyperparameter Defaults Are Someone Else’s Answers to Someone Else’s Problem

The defaults in any training script were tuned for the specific model, dataset, and hardware configuration the author was using. Change any of those variables — and you will — and the defaults may be actively wrong for your situation.

The parameters I see misused most often:

Learning rate. Full fine-tuning of a 7B+ model on a domain-specific dataset: I rarely go above 2e-5, and more often land at 5e-6 to 1e-5. If you've come from a general deep learning background where 1e-3 is a reasonable starting point, that number will catastrophically destroy the weights you're trying to preserve. For LoRA adapters you have more room — 1e-4 to 3e-4 is typical — but this varies with model, dataset size, and rank. Run a brief LR sweep on a 5% data sample before committing a full run to a learning rate you haven't validated.

LoRA rank and alpha together. These are almost always discussed separately and almost always need to be considered together. The effective scaling of LoRA updates is proportional to alpha / rank. If you double the rank without adjusting alpha, you've halved the effective learning rate of your adapter. My default starting point: alpha = 2 * rank. From there I tune based on validation loss behavior. Rank 16–64 covers the majority of use cases; going higher adds parameters with diminishing returns unless you're working on a data-rich task with high output diversity.

Warmup. Always use it. Always. A linear warmup over 3–5% of total training steps prevents the loss spikes and weight instability you see when the optimizer takes large gradient steps before it has a reliable second-moment estimate. On runs shorter than 1000 steps, I use a flat 50–100 step warmup regardless of the percentage. On longer runs, cosine schedule with warmup is my default — it consistently produces better final checkpoints than linear decay for fine-tuning.

Sequence length and its interaction with batch size. Short sequences with large batch sizes give stable gradient estimates but may miss long-range dependency patterns in your data. Longer sequences require gradient accumulation to maintain effective batch size — which works, but accumulated gradients over long sequences are noisier. Know your actual data distribution before making this tradeoff. I’ve seen projects default to 2048-token sequences on a dataset where 85% of examples were under 512 tokens — they were padding to 2048 and wasting 75% of their compute on padding tokens.

5. Evaluation Is an Engineering Problem, Not a Formality

Here’s a failure pattern that kills well-built models in the last mile: rigorous training, sloppy evaluation.

I’ve been on teams that ran dozens of training experiments with careful hyperparameter sweeps — and then evaluated the final model with a 200-example held-out set sampled from the same distribution as the training data, using an LLM judge that had the same biases as the model they were evaluating.

Those teams shipped models they thought were good. Some of them were. Many weren’t.

Build your evaluation suite before you build your dataset. The eval defines what “good” means. Your dataset should produce models that succeed on the eval. If you build the dataset first, you’ll fit it and then construct evals that confirm your existing model works — that’s rationalization, not evaluation.

Keep your held-out set sealed. It lives on a separate S3 path with restricted IAM access. Nobody runs experiments against it during development. It gets opened once — when making a ship/no-ship decision. All iterative development happens against a development eval that has never touched the held-out set. Treat it like a final exam. You get one shot.

LLM-as-judge is a tool, not a ground truth. It’s useful for dimensions that are hard to score programmatically. But LLM judges have known biases: preference for verbosity, self-similarity bias when evaluating outputs from the same model family, and susceptibility to confident-sounding wrong answers. I always pair LLM judge scores with human evaluation on a random sample and track their correlation. When they diverge significantly, I trust the humans and investigate why the judge is wrong.

Track calibration explicitly. Expected Calibration Error (ECE) should be in your standard eval suite for any model that expresses confidence or makes factual claims. A model that is accurate 80% of the time but expresses high confidence when it’s wrong is more dangerous than a model that is accurate 75% of the time but reliably uncertain when it might be wrong. Calibration is a direct user trust metric and it’s almost universally ignored in fine-tuning evaluations.

Production monitoring is part of your eval infrastructure. Pre-launch evals tell you how the model behaves on the distribution you anticipated. Production monitoring tells you how it behaves on what users actually send. These are different, and the gap between them is where your next iteration comes from. I sample 1–2% of production outputs for weekly review, run semantic clustering to find failure mode patterns, and maintain a structured feedback loop from domain experts where applicable.

6. Cloud Cost Discipline Is Not Optional

I’ve watched ML teams burn $40,000 in a weekend. Not on fraud. On legitimate training jobs with no budget guardrails, no tagging, and nobody watching the billing dashboard.

GPU time is expensive. Misconfigured GPU time is very expensive. Here’s how I prevent cost spirals:

Tag everything at the resource level. Every training job, every endpoint, every storage bucket gets cost-allocation tags at minimum: project, team, experiment ID, environment. Without tagging, your cloud bill is an undecomposable lump sum. With it, you can trace cost to individual experiments, identify which approaches are burning budget fastest, and make data-driven prioritization decisions.

Profile before you scale. Before any full training run, I run a 100-step profile pass. I measure GPU utilization, memory bandwidth utilization, MFU (Model FLOP Utilization), and step time. If GPU utilization is below 80%, I have a bottleneck — data loading, tokenization, suboptimal batch packing, or I/O contention. I fix the bottleneck before scaling. An inefficient job scaled to 8 GPUs costs 8x as much and takes roughly the same time.

Use Spot for training with hardened checkpointing, On-Demand for inference. Spot/preemptible instances offer 60–90% cost reduction on training workloads if you’ve done the work in Lesson 1. Never use Spot for inference endpoints — the interruption SLA is incompatible with serving latency requirements. For inference, right-size your instances against actual load profiles and use reserved instances for predictable baseline traffic.

Set budget alerts and actually act on them. AWS and GCP both support budget alerts at configurable thresholds. I set alerts at 50%, 80%, and 100% of each experiment’s allocated budget. At 50%: sanity check against expected progress. At 80%: decision point — extend budget or terminate. At 100%: the job stops. This is enforced by policy, not by goodwill. I’ve seen teams with budget alerts that everyone ignored. The alert is useless if nobody owns the response.

7. Alignment Failures Are Engineering Failures

I put this last because it’s the most serious and the most often treated as someone else’s problem.

Fine-tuning does not just change what a model knows. It changes how it behaves, what it prioritizes, what it suppresses, and how it relates to uncertainty. These behavioral changes are not reliably visible in aggregate benchmark scores. They appear in specific failure cases — often high-stakes ones — that your eval suite didn’t anticipate.

Two examples from my own projects:

A healthcare information model I shipped was subtly overconfident. The benchmarks looked excellent. In practice, it gave definitive-sounding answers on questions where clinical uncertainty should have been explicitly expressed. The base model was appropriately hedged; fine-tuning on expert-authored clinical content had trained the hedging out. A clinician caught it in user testing. We caught it before harm because we had domain experts in our testing process. Many teams don’t.

A customer service model I shipped had learned sycophancy. It had been fine-tuned on data where positive user sentiment correlated with helpfulness labels. It learned to optimize for sounding helpful rather than being helpful — telling users what they wanted to hear rather than accurate information. Aggregate satisfaction metrics were fine. Resolution rate was mediocre. Took three weeks to trace the root cause to the labeling methodology.

What I now do without exception:

Red-team every model before shipping. I use external red teamers who had no involvement in building the model — they see blind spots the team has normalized. I also run structured adversarial evaluations: jailbreak probing, edge case distribution testing, adversarial inputs designed to elicit failure modes specific to the deployment domain.

Train explicit uncertainty. I include training examples where the correct response is “I don’t have sufficient information to answer this reliably” or “this is outside my area of expertise.” This is not a nice-to-have. A model that can express appropriate uncertainty is dramatically safer and more trustworthy in production than one that always sounds confident.

Define rollback criteria before launch. Before any model goes to production, I define the metrics and thresholds that trigger an automatic rollback. If user-reported error rate exceeds X% in the first 48 hours, the model reverts. If calibration score drops below Y, we investigate. This is standard practice in software deployments. It is not standard practice in model deployments. It should be.

What I Actually Believe After All of This

Fine-tuning is not machine learning with some infrastructure bolted on. It’s systems engineering that happens to involve gradient descent.

The math matters. Understanding why LoRA works, what catastrophic forgetting is doing mechanistically, how cosine annealing interacts with weight update dynamics — this knowledge is real and useful. But it’s not what separates teams that consistently ship reliable models from teams that don’t.

What separates them is: hardened infrastructure. Disciplined data pipelines. Honest evaluation methodology. Production monitoring. Cost governance. And a genuine engineering culture around the alignment and safety properties of the systems they’re shipping.

The model is the output of the system. Build the system well.

Building fine-tuning infrastructure at scale, or debugging a training run that’s behaving unexpectedly? I’m in the comments — this is the kind of problem I find genuinely interesting to think through.