You Cannot Vibe-Code an Agent: Loop Engineering in Fermix

I did not build Fermix in one prompt, and nobody builds an agent that way. Nobody builds anything complex that way, really. An agent just makes it obvious.

The tempting version is always a single line, something like

/goal build openclaw in elixir, make no mistake

It does not work, and it cannot work. An agent is not one program. It is providers, channels, tools, memory, subagents, scheduled jobs, a sandbox, and a tracing layer, all running at once, with a live model making decisions in the middle of it. That is dozens of moving parts that have to behave together, and half of them only misbehave once a real model is driving. You do not prompt something like that into existence. You get a demo that way, not a daemon.

Loop engineering

The phrase going around for the alternative is loop engineering. Andrew Ng wrote it up well, and he credits the same idea surfacing from Boris Cherny, who built Claude Code, and Peter Steinberger, who built OpenClaw. The core of it is simple: instead of writing the whole thing in one pass, you build a loop where the agent writes code, tests it, and keeps iterating until it meets a spec. In Ng’s words, the coding agent can “write code, test its work, and keep iterating.”

Fermix was built as loops, not shots. Here is what that loop actually looks like.

The unit of work is a milestone

Fermix is not one project. It is more than forty milestones, running from M2 for multi-agent work to M5 for the sandbox, M7 for the tool catalog, M10 for tool search, M12 for providers, M15 for media generation, and on and on.

Each milestone starts as a design doc, not as code. The doc does the expensive thinking up front: the problem and the goal, the scope and non-goals, the architecture, the wrong-abstraction traps to avoid, the config schema, the telemetry, the testing strategy, and a list of open questions I still have to answer. The research lives there too. The media milestone, for instance, carries a provider-by-modality matrix that is web-verified and dated, which is real work done before a single line ships. No code gets written until the doc is approved.

The human is the context advantage

This is the part you cannot hand off. Ng makes the point that the human holds a “context advantage” over the model: you know the users, the constraints, and the taste, and the model does not.

In Fermix, that advantage is the spec. I do the research, I make the owner decisions, and I sign off, and that is knowledge the model cannot get on its own, injected before it builds anything. The model does not decide what Fermix is. I do, and then it builds.

Then the agent builds, test first

The approved spec goes to Claude or Codex, with one rule up front: write the failing test first, then implement until it passes. Red to green. This is the fast inner loop, the one that turns over every few minutes, and for a lot of the codebase it is enough.

Unit tests stub the model. That is the hole.

Here is where it gets honest. The unit suite is deterministic because it stubs the model, and it proves wiring, side effects, safety branches, and exact behavior with no network. It is fast and it is strict, and it carries most of the mechanics.

What it cannot carry is the one thing that matters in production: given a real query and the real model, does the agent reach for the right tool, stay in budget, and answer well? A stubbed model always picks the tool you told it to pick, which proves your wiring but proves nothing about behavior. Unit tests are necessary, but they are not the acceptance test for an agent.

The eval suite closes the real loop

After the implementation is done, a second tier runs. It drives real queries through the live daemon, with the real model, so every turn is a real turn. Each case declares what correct actually means:

• the query, in plain user language
• the tools it should reach for
• what the reply has to confirm
• a cost ceiling
• a time ceiling
• a rubric, for an LLM judge to grade the answer

And that is not pseudocode. It is the shape of the file:

query: "Create a small text file at eval_scratch/notes.txt with the line: hello from fermix"
expect:
  tools_any: [file_write]
  reply_matches: "(?i)created|wrote|saved|notes\\.txt"
  max_cost_usd: 0.5
  max_duration_ms: 90000
rubric: "Confirms it created notes.txt containing 'hello from fermix'."

There are nineteen of these suites, covering files, git, shell, memory, jobs, plugins, subagents, ultra, web, routing, safety, and more, with one per slice of the agent.

Tracing is how you grade it

The eval does not read the reply and guess. Fermix already traces itself, so every turn becomes a trace with spans for which tools ran, in what order, any errors, the cost, the latency, and the final reply, and that tracing runs into a local Opik instance. The eval gives each case a unique session, which becomes the trace id, and then it runs the query, waits for that exact trace, and grades the trace against the expectation. So the verdict is not a vibe. It is a graded trace: the right tools, in budget, in time, with the correct reply.

This is the loop, not the shot

Line it up with Ng’s three loops. The inner loop is the coding agent doing failing test, implement, green, all in minutes. Then the eval is the acceptance gate. Ng lists evals as optional in that inner loop, but for an agent I do not treat them as optional, because they are the gate. A feature is not done because the unit tests pass; it is done when the real model, on real queries, hits the tools, the budget, and the answer. The outer loop is me: I read the milestone output and the eval report, update the spec, and approve the next pass. That is the developer feedback loop, and it is where the context advantage keeps getting injected. A failed eval is not a bug ticket. It is the next edit to the spec.

Final shape

The whole method, in order:

• One milestone at a time.
• A design doc with real research and owner decisions.
• Approve before any code.
• Failing test first, implement to green.
• Unit tests for the mechanics.
• An eval suite for the behavior: real queries, expected tools, reply, cost, and latency, graded off the Opik trace.
• Read the report. Fix the spec. Go again.

None of this is one prompt. The one-shot is a demo. The loop is the product.

Repo link is here: https://github.com/tezra-io/fermix