My Fine Tune Got Worse and I Had No Way to Ask It Why

I fine tuned a small Llama model on a support dataset. It was a normal LoRA run, nothing exotic, and the eval numbers looked fine at first glance. Then I checked the one capability I actually cared about, formatting answers as clean JSON for a downstream parser, and it had fallen off a cliff. It went from getting the format right nine times out of ten to barely six.

Nothing in the training run explained this. No errors, no warnings, no obvious spike in the loss curve. The model just quietly got worse at one specific thing while everything else looked normal.

The debugging routine that does not scale

My first instinct was the one everyone has. Open the dataset, scroll through examples, look for anything weird. Squint at a few hundred rows of JSON completions and try to spot the poison by eye. When that turned up nothing convincing, I moved to the usual knobs. Lower the learning rate. Fewer epochs. Different batch size. Each retrain took an hour, and each one either fixed nothing or fixed it for reasons I could not explain, which is almost worse, because then I do not know what to keep doing next time.

The actual question I needed answered was never “what looks suspicious.” It was: which training examples, specifically, caused this regression, and is it even the data at fault, or something in how I set up the run. Eyeballing a dataset cannot answer either half of that question. It can only produce guesses.

Finding the tool that answers the actual question

That is the gap Gradian is built for. It is a training data attribution engine for LoRA fine tunes, and its entire premise is that you already have a regression in hand, a fine tune that got worse at something, and you want to know why, not another framework for training in the first place.

It splits the problem the way I eventually realized I should have from the start, into two separate questions:

Is it a specific cluster of training data that taught the model something that fights the capability I care about?

Or is it something structural in the setup, like a loss mask that covers the wrong tokens, or completions that get silently truncated, that no amount of staring at individual rows would ever catch?

The first question needs real math, per example gradients traced back through the adapter. The second one does not need math at all, it needs a checklist run against your dataset, your config, and your trainer logs. Gradian ships both, and the report it produces joins them, because which one is true changes what you actually do next.

Running it against my own regression

The audit is the fast path, no GPU involved, so I ran that first.

gradian diagnose \
  --dataset train.jsonl \
  --run runs/my-finetune \
  --base-model meta-llama/Llama-3.2-1B

That alone is worth doing before anything else, because a huge share of fine tune failures are mechanical rather than about content: a missing EOS token so the model never learns to stop, a loss mask that covers the prompt instead of the completion, train and eval sets that quietly overlap, twelve epochs over four hundred examples. None of these crash. None show up as a training error. They just make the model worse in a way that is easy to blame on the data.

With nothing conclusive from the audit alone, I ran the actual attribution, pointing it at the base model, the adapter I had already trained, my dataset, and an eval set built from the cases that had regressed:

gradian attribute \
  --base-model meta-llama/Llama-3.2-1B-Instruct \
  --adapter runs/my-finetune/adapter \
  --dataset train.jsonl \
  --eval-dataset capability_eval.jsonl \
  --capability json_formatting \
  --out reports/

The report was the first time in this whole process I got an actual answer instead of a hunch. It named the regression precisely, the confidence interval around it, and then it named the cause: one cluster of training examples, grouped around invoice and currency fields, was responsible for the large majority of the negative influence. And then it went one step further, and told me most of those examples had truncated completions. The model was not learning bad JSON habits from that cluster. It was learning to stop mid object, because that is literally what the training data showed it.

That second fact is the one I would never have found by reading the dataset. Truncation is invisible unless you know exactly which examples to look at and what to look for, and by the time you know that, you have already done the hard part.

The part that made me trust the answer

What convinced me this was not just a plausible sounding number was one design choice buried in how the attribution actually works. When you build the signal that says “explain this failure,” the obvious approach is to push the model toward the correct answer and see which training examples get credit. Gradian does the opposite by default, it takes the gradient of the correct answer and subtracts the gradient of what the model actually said. The reasoning, once I understood it, was obvious in hindsight. A lot of what fine tuning teaches is answer format, and format genuinely does lower the loss on the right answer too, so the naive signal ranks the poisoned examples as the most helpful data in the whole set. Subtracting out what the model actually produced cancels that shared direction and leaves the real culprit standing alone.

It also refused to answer when it should not. On a different capability I checked afterward, the regression I thought I saw turned out to sit entirely inside the noise band, and the tool said so plainly instead of forcing a diagnosis onto a number that was not really there. That refusal mattered more to me than any of the clusters it did find, because it meant the clusters it did name, I could actually act on.

What changed for me

I fixed the actual problem in one line, raising max_seq_length so those completions stop getting cut off, and retrained. The capability came back.

What stuck with me was not that specific fix. It was that I stopped treating a regressed fine tune as a mystery to be poked at with hyperparameter guesses. There is a real, answerable question underneath every one of these failures: which examples, or which setting, caused this. I just did not have a way to ask it directly until I had a tool built to answer exactly that question, over gradients I could not have inspected by hand, joined with a set of mechanical checks I would never have thought to run in the right order.

That is the whole shift. Not a smarter guess. An actual answer.