Designing a Local-First Multi-Agent System with Ollama and FastAPI

Designing a Local-First Multi-Agent System with Ollama and FastAPI

There is a pattern I kept running into with AI demos.

They looked impressive for about thirty seconds.

You typed a prompt, the model responded confidently, and everybody in the room nodded. Then the second prompt arrived. It asked for comparison, trade-offs, evidence, or a rollout plan. That is usually the moment the demo stopped being clever and started becoming fragile.

That was the point where I decided I did not want another chatbot demo. I wanted a workflow.

So I built a local-first multi-agent system in Python using Ollama, FastAPI and a simple no-key web search tool. The architecture is intentionally explicit:

• one agent to plan

• one agent to research

• one agent to execute

• and one agent to review.

Nothing here is magical. That is exactly why it is useful.

## The problem with one giant prompt

A single-agent chatbot is fine when the question is small. But the moment the task becomes multi-step, the weaknesses show up quickly.

The model tries to reason, gather facts, structure the answer, and self-correct all at once. Sometimes it gets away with it. Often it does not.

From an architecture point of view that creates three problems.

First, you cannot see where the failure happened. Was the plan weak? Did the model use poor evidence? Did it ignore an important constraint? In a single giant response, that is hard to answer.

Second, there is no operational boundary between reasoning and action. The model thinks and writes in the same pass. That makes debugging painful.

Third, the system is not inspectable. If you want to improve it, you are left nudging prompt wording instead of improving a pipeline.

That is why I prefer an agent system where every stage has a job.

## The architecture I wanted

I did not want to start with a giant framework or a paid API stack. I wanted something I could run locally, understand end to end, and still write about honestly.

So I kept the architecture small:

1. Planner Agent — turns the user goal into a short execution plan.

2. Researcher Agent — runs web search, fetches a few pages, and compresses the evidence.

3. Executor Agent — writes the final architect-level answer.

4. Critic Agent — reviews the draft and asks for one correction pass if needed.

5. Run Store — saves the entire workflow as JSON for inspection.

That already gives you a much better system than “send prompt, hope for the best.”

- -

Web based interface give feel of real chat-box with complete flow

## Why I chose Ollama

I wanted an open-model path that stayed practical.

Ollama solved two problems for me immediately.

The first was cost. I could run the whole system without wiring up a paid API.

The second was control. I could choose a small model for planning, reuse the same model for research summarization and later swap a stronger model into the executor role without touching the rest of the architecture.

For the first version I kept the defaults simple:

• llama3.2:3b for planning,

• llama3.2:3b for research summarization,

• llama3.2:3b for execution,

• and the same again for review.

That is not because it is the strongest model. It is because it is a sane starting point when you want something small, local, and easy to reproduce.

If I wanted tighter structured output later, I would probably switch the planner or executor to qwen2.5:3b.

## The part that changed the quality the most: explicit planning

The planner returns JSON.

That sounds like a tiny implementation detail, but it changes the feel of the system.

Instead of asking the model to immediately answer the whole problem, I force it to define the work first:

• what steps are needed,

• why each step exists,

• what search query should be executed,

• and what deliverable should come out of that step.

That structure does two things.

It improves the workflow, and it gives me something I can inspect later.

If the final answer is weak, I can open the saved run artifact and ask a much better question than “why was the model bad?” I can ask “did the planner generate a weak path?” That is a far more useful debugging conversation.

## Research is where most agent demos become hand-wavy

This is the part I wanted to keep grounded.

The researcher does not just ask the LLM to “think harder.” It uses tools.

For each plan step, it:

1. runs a web search,

2. keeps a few results,

3. fetches HTML from the top pages,

4. compresses that content into a short research note.

That gives the executor something more useful than raw search snippets, but it still keeps the system understandable. I did not want to jump straight into a full RAG stack for this version. Search plus fetch was enough to prove the architecture.

This is also where I made a practical decision: I only fetch a small number of pages per step. In theory, more evidence sounds better. In practice, too much evidence creates latency, token waste, and noisy prompts.

Architecturally, I would rather have fewer high-signal inputs than a bloated context window.

## The executor is not a chatbot. It is the assembly layer.

This was an important mental shift for me.

The executor is not responsible for doing everything. It is responsible for turning the plan and research notes into a final answer.

That means it can focus on structure, trade-offs, and practicality instead of also trying to invent a plan on the fly.

I asked it to write like a senior architect:

• start with a title,

• then an executive summary,

• then the main answer,

• then risks and trade-offs,

• then practical next steps,

• and finally a source list.

That one instruction made the output feel much less like chatbot text and much more like something I would actually share with a team.

## Why I added a critic but kept it on a short leash

It is tempting to build an agent loop that keeps reviewing itself forever.

I avoided that.

The critic gets one job: review the draft and optionally request one revision pass.

That gives a noticeable quality bump without turning the workflow into an open-ended loop. In my experience, uncontrolled loops are where agent systems become slow, expensive, and annoying to operate.

A single critique pass keeps the system honest without making it unstable.

## What I liked about the final design

The biggest advantage is not that it sounds smarter. The biggest advantage is that it is inspectable.

Every run leaves behind artifacts:

• the goal,

• the plan,

• the research notes,

• the draft,

• the final answer,

• and the timing for each stage.

That means I can look at a bad run and actually learn something from it.

That is a much better engineering experience than staring at one disappointing paragraph and trying to guess which hidden prompt went wrong.

## What broke first

A few things became obvious very quickly.

### 1. Small models are not always consistent at JSON

Even with structured prompting, small models occasionally drift. That is why I added a JSON extraction helper and schema validation around the planner and critic.

### 2. Search quality matters more than people admit

If the researcher gathers weak results, the executor can only polish weak material. A better orchestration layer cannot fully compensate for poor evidence.

### 3. Page fetching is noisy

Raw web pages are messy. Navigation text, cookie banners, boilerplate, and repeated template content all get in the way. Even a simple HTML cleaner helps a lot.

### 4. More steps does not mean a better answer

When I let the planner create too many steps, the system became slower and less focused. Four strong steps felt better than eight mediocre ones.

## Why I think this is a better learning project than another chatbot clone

A chatbot clone teaches interface wiring.

A project like this teaches systems thinking.

You have to think about:

• orchestration

• state

• evidence flow

• separation of concerns

• error handling

• and observability.

That is much closer to real architecture work.

It is also more honest. If you are learning Python and AI engineering at the same time a multi-agent project forces you to make design choices that actually matter.

## Final thought

The most useful thing I learned from this build is that agent systems become easier to trust when they become easier to inspect.

I do not think the future belongs to giant prompts pretending to be workflows.

I think the more durable pattern is this:

• explicit planning,

• explicit evidence gathering,

• explicit synthesis,

• explicit review.

That is not the fanciest demo.

It is just the first one I built that felt like a real system.

## Repo contents for reference :

This project includes: https://github.com/subhashjugran/open-llm-multi-agent-system