One Night with Karpathy’s autoresearch

What happens when you let a coding agent loose on a machine learning training loop — with one hand tied behind its back?

TL;DR

Autoresearch is really fun to use . the true metric for the program is val_bpb, which means validation bits per byte or how many bits the model needs to predict each byte of text it’s never seen before. Lower number is better

autoresearch. https://github.com/karpathy/autoresearch

Andrej Karpathy recommends use of a single Nvidia H100 GPU. I could not get that processor from Google Cloud , so I went with the Nvidia L4 GPU. Overnight the AI agent essentially discovered: on the Nvidia L4, the optimal model is half the size, trains 28x faster, uses 73% less VRAM, and is 18% better. That’s a genuinely useful finding — it tailored the config specifically to the hardware.

The genius of the simple set up is that an AI agent “lab” with a small budget and less sophisticated equipment is forced to come up with smarter ways to use the equipment , when they do not have the luxury of using the state of the art GPU nvidia H100 to “force” a better word prediction by using more attention heads, batches, etc. There is an easy setup and run script in the github repository. I ran the program for 10 hours on Google Cloud virtual server using a single nvidia L4 GPU costing me about 70 cents per hour and I used my claude code max plan to launch in the server using sonnet as the base model. Our single Autoresearch overnight run had the val_bpb decrease from 1.395 → 1.138.

The program contains about 800 lines of python code. The code extensively uses nanochat architecture to run the training with pytorch tensors. The data is prepared for training by the prepare.py section. The agent experiments and after each 5 minute run edits the train.py parameters based on the results. If Andrej had made prepare.py editable along with train.py ; then the agent probably would have done what he suggested in his repo readme — used simpler data, smaller vocab, or shorter sequences — and gotten a fine result through the expected path. But with the less powerful GPU the read-only restriction forced autoresearch to learn to solve the same problem in a different domain using the equipment as the constraint, not the data prep.

Karpathy baked the read-only constraint in deliberately. I think he knew what would happen.

I’ve been running Andrej Karpathy’s [autoresearch](https://github.com/karpathy/autoresearch) project on a GCP VM, and last night something happened that I keep turning over in my head. Not because an AI hit a good benchmark. Because of how it hit the benchmark — and what it tells us about machine learning intuition emerging from constraint.

The Setup

autoresearch is Karpathy's experiment in agentic ML research. A coding agent is handed a training script (train.py), a fixed dataset pipeline (prepare.py), and a simple mandate: improve the validation score — measured as val_bpb, bits per byte on held-out text — within a 5-minute wall clock budget. Every run is timed. Every score is honest. The agent can run the script, read the logs, and edit train.py. That's it.

The catch: prepare.py is read-only. The agent cannot touch the data pipeline. At all.

This matters enormously, because Karpathy’s own tuning guide says the primary levers for getting good results on small models are all in prepare.py:

— Use a dataset with a lot less entropy, e.g. TinyStories.

— Decrease vocab_size from 8192 down to 1024 or even byte-level.

— Lower MAX_SEQ_LEN down to 256.

— Decrease EVAL_TOKENS.

Every suggestion Karpathy considered most important was locked behind a door the agent wasn’t allowed to open. My initial run on the Nvidia L4 single GPU gave an initial “no mods” val_bpb

The baseline val_bpb: 1.395. Higher is worse — the model needed nearly 1.4 bits of information per byte to predict unseen text. After 87 overnight “experiments” by the autonomous agent ( claude code launched in the google cloud VPS running sonnet — on my Max plan )

By morning: 1.138.

What Karpathy Suggested, and What the Agent Actually Did

Here’s the table that made me stop scrolling through the logs:

The agent followed the spirit of Karpathy’s direction — smaller model, smaller batch — but it was working in a fundamentally different problem space. It couldn’t reduce noise at the input level. It had to find noise reduction somewhere else.

It found it in seven lines of changed code and one entirely new idea.

The Seven Changes

The actual diff was clean. No sprawling refactor. Seven targeted modifications:

1. DEPTH: 8 → 5 Fewer transformer layers. With only 5 minutes of compute, a depth-8 model barely completes 87 training steps. A depth-5 model gets 2,506. It’s better to be a smaller person who’s had time to actually think than a bigger one still clearing their throat.

2. TOTAL_BATCH_SIZE: 2¹⁹ → 2¹⁵ A 16× reduction. More optimizer steps per clock. This is the single largest source of improvement — but it introduced a problem the agent had to solve.

3. DEVICE_BATCH_SIZE: 32 → 16 Matched to the smaller total batch to keep the gradient accumulation math consistent.

4. WINDOW_PATTERN: SSSL → SSLL Karpathy suggested going full “L” (all full attention, no banded sliding window). The agent tried it. It was worse. The hybrid — two sliding-window layers, two full-attention layers — turned out to be the winner. The agent found something Karpathy’s guide didn’t specifically recommend.

5. MATRIX_LR: 0.04 → 0.025 Softer Muon learning rate. Smaller batches mean noisier gradient estimates; you need gentler steps to keep training stable.

6. Optimizer tuning bundle:

• WEIGHT_DECAY: 0.2 → 0.0 (removed)
• ADAM_BETAS: (0.8, 0.95) → (0.85, 0.95)
• FINAL_LR_FRAC: 0.0 → 0.055 (don’t decay all the way to zero)
• Muon beta2: 0.95 → 0.90 (faster variance adaptation)

7. One new line of code:

torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=0.35)

The Gradient Clipping Insight

This is the one that keeps me thinking.

When the agent first tried TOTAL_BATCH_SIZE = 2^15 — no other changes — the score was 1.153. The logs noted: "too noisy, discarded."

The second time it tried 2^15, now with gradient clipping: 1.143. Kept. Same batch size. One line of difference. Entirely different outcome.

Here’s what makes this elegant: Karpathy’s suggested fixes for noise were all upstream. Simpler vocabulary, shorter sequences, easier dataset — reduce the entropy of what the model sees before training even begins. The agent, locked out of that approach, reduced noise downstream, at the optimization step itself.

Gradient clipping says: regardless of what the data is telling you on any given mini-batch, cap the size of the update you’ll make. With 32,768 tokens per step instead of 524,288, every gradient estimate is inherently sketchier — you’re drawing conclusions from a smaller sample. Without clipping, those noisy spikes destabilize learning. With clipping, you still move forward, just… carefully.

Same goal. Different mechanism. The agent invented an alternative path to the same destination.

A Word on val_bpb and Why It Matters

One question came up naturally in reviewing the session: what exactly does val_bpb measure, and why is it the right metric here?

It’s bits per byte on held-out text — text the model never saw during training. This is the honest test of whether a model learned patterns or merely memorized content.

The three regimes:

• Underfitting: both training loss and val_bpb are high. The model hasn’t learned enough of anything yet. This was our start: 1.395.
• Generalizing: training loss is low, val_bpb is also low. The model internalized real structure — grammar, narrative flow, how stories resolve — not specific sentences.
• Overfitting/memorization: training loss keeps dropping but val_bpb starts rising. The model aces its homework but fails the pop quiz.

With only 2,500 training steps in 5 minutes, overfitting wasn’t really a risk here. The model barely sees the data once. The real tension was different: model capacity vs. training time. A deep model that’s barely trained loses to a shallow model that’s well-trained. val_bpb captures this cleanly because it measures generalization, not performance on what the model already memorized.

Going from 1.395 to 1.138 isn’t just a number. It means the model needs 0.257 fewer bits of “hint” per byte to predict language it’s never seen. It got genuinely smarter at English — not just better at recalling its training set.

The Constraint Was the Lesson

What strikes me most isn’t the final score. It’s that the constraint produced the insight.

If prepare.py had been editable, the agent probably would have done what Karpathy suggested — simpler data, smaller vocab, shorter sequences — and gotten a fine result through the expected path. The read-only restriction forced it to solve the same problem in a different domain, and in doing so it demonstrated something about the relationship between data-level noise and optimization-level noise that a more straightforward experiment might have glossed over.

Karpathy baked the read-only constraint in deliberately. I think he knew what would happen.

What’s Next

The model is still running on the VM. The next round of experiments will focus on whether there are gains to be found in the validation evaluation itself — and whether the SSLL attention hybrid was a lucky find or something reproducible.

But honestly, I keep coming back to that one line of code. Gradient clipping as a proxy for dataset simplification. It’s the kind of equivalence that only becomes visible when you’re locked out of the obvious solution.

Sometimes the best way to understand a system is to be prevented from touching half of it.

Running Karpathy’s autoresearch on GCP with Claude Code as the agent. All experiments logged. val_bpb is bits per byte on held-out validation data — lower is better. Baseline: 1.395. Best overnight run: 1.138.

Interestingly Andrej is also using the autoresearch agent as a type of benchmark to illustrate the significant decrease in gpt-2 model pretraining costs from 2019 to 2026 ( $43,000 to $48 ). https://github.com/karpathy/nanochat

all credit to Andrej Karpathy and nanochat

@misc{nanochat,
  author = {Andrej Karpathy},
  title = {nanochat: The best ChatGPT that \$100 can buy},
  year = {2025},
  publisher = {GitHub},
  url = {https://github.com/karpathy/nanochat}
}