I Run an AI Loop for a Living. Here Are the 5 Things That Keep It From Rotting.

The frontier moved from writing prompts to running loops — and every failure I have had running one was the harness, never the model.

Photo by Jahanzeb Ahsan on Unsplash.

In March 2026, Andrej Karpathy released a small repository called autoresearch and let an agent run on it for two days. Unattended, it ran roughly 700 experiments on a training script, kept the changes that improved the metric, threw away the ones that did not, and surfaced about 20 optimizations that — applied to a larger model — cut training time by around 11%.

Shopify’s Tobi Lütke pointed it at his own problem overnight and reported a 19% gain by morning. Fortune called it “the Karpathy Loop.”

The impressive part is not that a model found the optimizations. The impressive part is that nobody was in the room while it did.

That is the shift almost no one names out loud. A prompt is something you write once and forget. A loop is something that runs while you sleep. The industry spent two years perfecting the first sentence; the people shipping agents that actually do work have quietly moved on to the second thing — the harness around the model, the scaffolding of triggers, checks, memory, and restarts that decides whether an autonomous run produces something useful or just burns tokens all night.

After a year of running one, I read the shift the same blunt way: the raw capability of the model was already high enough. The bottleneck had moved outside the model — into how you combine agents, verify them, make them remember, and, above all, keep yourself out of the way.

I know this one from the inside, because I run a loop like this.

The Thing I Learned Running One

My loop is called writeflow. Every day, unattended, it pulls what is trending, drafts article candidates, scores each one against how well it fits a target and how likely it is to earn, keeps the good ones, and folds what actually performed back into its own examples so the next run is a little sharper. Generate, evaluate, learn, repeat. It works while I sleep, and most mornings there is something on disk that was not there the night before.

Here is the one sentence I would tape to the monitor of anyone building these: in a year of running it, not a single failure was the model’s fault.

Every outage was the harness. A fetch that hit an error, swallowed it, and moved on as if nothing happened. A parser that rejected a whole document it should have tolerated. A step that quietly produced nothing for two days before I noticed, because nothing in the loop was watching the one number that would have told me.

The model never even got the chance to be wrong. The system around it failed first, silently, in the boring plumbing.

So when I read the pattern lists making the rounds — the taxonomies of critic loops and memory loops and planning loops — I do not read them as a menu of clever ideas. I read them as a list of the specific places my plumbing has already burst. A handful of them are the difference between a loop you can leave alone and a loop that quietly rots. These are the ones.

What Actually Keeps It Alive

State lives on disk, not in the context window. A loop’s memory cannot live in a conversation that gets truncated. writeflow’s state is a database and a folder of files — every draft, every score, every decision is written down before the next step runs. When it dies at 3 a.m., I do not reconstruct what it was thinking. I read the row.

If your agent’s only memory is the transcript it is currently holding, you do not have a loop that runs for days. You have a very long single turn that will lose its mind the moment the window fills.

Let it restart, and treat that as healthy. The instinct is to make the loop never fail. The better instinct is to make failure cheap: let it throw away a bad run and start clean from what is on disk. Karpathy’s autoresearch treats “discard and restart” as normal behavior, and that is exactly right.

A run you can safely kill and resume is a run you can walk away from. A run you have to nurse is not autonomous — it is a manual job wearing a loop’s clothing.

Never let the thing that writes grade its own work. This is the single highest-leverage split, and it is the one people skip. The generator and the evaluator must be different roles with different instructions. In writeflow, the drafter never decides whether a draft is good; a separate scorer does, and a separate check after that.

An agent that grades its own output does not converge on quality — it converges on output that flatters the grader, which is the same thing as slop that is very confident. Split the roles or the loop will lie to you politely, forever.

[embed]

Debug the transcript like a stack trace, not by poking at it. When a two-day run goes wrong, you do not re-run it and hope. You read the log from the top and find the exact turn it went off the rails.

The loop’s observability is the loop. If you cannot reconstruct why last night’s run did what it did by reading a file, you are not going to be able to fix it — you are only going to be able to restart it and pray.

And then get out of the way. The sharpest rule is the one that sounds the least technical: you must not be the bottleneck yourself. I have caught myself hovering over a run that was only ever waiting on me, not the model. If you are sitting there watching the loop tick, you do not have a loop. You have a slow, expensive version of yourself.

The Part the Pattern Lists Leave Out

None of this is magic, and the honest failure mode is worth more than the patterns.

This one is structural, not a bug you can patch. The loop only accepts a change if it immediately improves the score, so the agent gets stuck making tiny variations of whatever worked last — it can never take a step backward to set up a bigger step forward. A human researcher reasons “this will get worse before it gets better.” A naive ratchet has no room for that thought.

autoresearch’s own issue tracker flags a cousin of this: the agent can push the metric down without the model actually getting better. Every loop I have run has its own version of the same disease: it optimizes the number you gave it, not the thing you meant. The evaluator is where your actual judgment goes to live or die, and no amount of loop plumbing rescues a metric that measures the wrong thing.

There is a second uncomfortable truth. The harness is tech debt. Every check and guardrail you write to babysit a weak model becomes, as the model gets stronger, the part most likely to break for no reason. I have deleted more of writeflow’s scaffolding this year than I have added. The best loop is the least loop that still cannot hurt you.

The Job Changed

Put it all together and what you are looking at is not a new prompting trick. It is a change of job.

You will hear this is just DevOps with fashionable nouns. Half right — the loop is old operations in new clothes. But the wrong half is the one that stings: everyone who spent two years getting good at prompts was sharpening the one skill this shift just made cheap.

Prompt engineering was a writing skill — find the words that get the model to behave. Loop engineering is an operations skill — state, observability, blast radius, the discipline to let a thing fail cheaply and restart clean. The people running agents that survive for days are not better at prompts than you. They are better at running systems that run without them.

A prompt is something you write once and forget. A loop is something you are on call for. If you are going to build one that works while you sleep, build the part that pages you before the part that impresses you.

References

• “‘The Karpathy Loop’: 700 experiments, 2 days, and a glimpse of where AI is heading,” Fortune (Mar 17, 2026) — https://fortune.com/2026/03/17/andrej-karpathy-loop-autonomous-ai-agents-future/
• Andrej Karpathy, autoresearch — the minimal autonomous experiment loop (repo + issue tracker) — https://github.com/karpathy/autoresearch