The Crawler the LLM Gave Me — And Why I Rewrote It

Building a web crawler for myIR, a Java-based search engine built from scratch

When I started building **myIR** — a search engine written in Java from the ground up — one of the first things I needed was a web crawler. I asked an LLM to generate a starting point. It gave me the classic answer: a recursive method that fetched a page, extracted links, and called itself for each one.

It worked. But I didn’t like it.

Not because it was wrong — recursion is the textbook answer for tree traversal. But because it felt like the wrong shape for this problem. The fetcher knew too much. The call stack was doing the work that a queue should do. And parallelism would have been a nightmare to add later.

So I rewrote it. This is the story of that rewrite, and the design decisions behind it.

What Was Wrong With Recursion

The recursive crawler looked roughly like this:

[embed]

The problems:

1. The fetcher had too much responsibility. It fetched, parsed, extracted links, and drove the traversal — all in one place. Single Responsibility Principle, violated.

2. The stack was the frontier. Depth was managed implicitly through call depth, not explicitly through data. You couldn’t inspect it, pause it, or replace it.

3. Parallelism was structurally difficult. To parallelize, you’d have to restructure the recursion into something else entirely. Better to start with that something else.

The Key Insight: Invert the Responsibility

The redesign started with one question: what does the fetcher actually need to know?

Answer: nothing about traversal. The fetcher’s job is:

1. Fetch a URI
2. Parse the HTML
3. Report the links it found

That’s it. It shouldn’t decide what to do with those links. So instead of the fetcher calling back into the crawler, I gave it a Consumer<Set<URI>>:

[embed]

The fetcher calls linkSubscriber.accept(discoveredLinks) and forgets about them. It has no idea what happens next. The traversal strategy receives those links and decides what to enqueue.

This is inversion of responsibility — not quite inversion of control, but the same spirit. The fetcher pushes data out through a callback instead of pulling control in through recursion.

The Frontier: A Blocking Queue

With the fetcher decoupled, the traversal logic lives in SiteTraversalStrategy. The frontier is explicit:

[embed]

Each node in the frontier carries three things:

[embed]

The rootUri is important — it lets domain filtering rules know which crawl session a link belongs to, even in a future multi-root scenario.

The rootUri is important — it lets domain filtering rules know which crawl session a link belongs to, even in a future multi-root scenario.

The main loop is straightforward BFS:

[embed]

No recursion. No implicit stack. The frontier is data, not call frames.

Concurrency Without Locks

The strategy uses Java Virtual Threads through a VTExecutor. Each node is processed concurrently, with two atomic counters keeping everything honest:

[embed]

inFlightTasks tracks how many tasks are running so the main loop knows when it’s truly done — not just when the queue is momentarily empty.

emittedPages uses a compareAndSet loop to prevent over-emission under concurrent writes:

[embed]

The visited registry uses ConcurrentHashMap.newKeySet() — which means markVisited is atomic by construction:

[embed]

No explicit locking anywhere in the hot path.

The Full Pipeline

The crawler fits into a larger ingestion pipeline designed around three interfaces:

DocumentSource<T> → DocumentMapper<T> → Indexer

readInto(Consumer<T>) map(T) : Document index(Document)

The entire ingestion service reduces to:

source.readInto(item -> indexer.index(mapper.map(item)));

The crawler is just one implementation of DocumentSource<WebPage>. Tomorrow it could be an S3 bucket, a sitemap, or a database — the indexer wouldn’t notice.

The WebPage → Document mapping is equally clean:

[embed]

Fields, attributes, raw content — all separated. The indexer can choose what to use.

URI Canonicalization: The Detail That Matters

One thing the LLM-generated crawler completely ignored: the same page can be reached through multiple URIs.

https://demo.dotcms.com/about

https://demo.dotcms.com/about/index

https://demo.dotcms.com/about?com.dotmarketing.persona.id=xxxx

Without canonicalization, you visit the same content three times and the visited registry doesn’t know they’re duplicates.

The solution is a composable UriCanonicalizer:

[embed]

Each canonicalizer is a function URI → URI. They compose. The default web canonicalizer handles scheme normalization, fragment stripping, and trailing slash normalization. Site-specific rules layer on top.

This is the kind of detail that separates a toy crawler from one that actually works on real sites.

What I Learned

The recursive crawler wasn’t wrong — it was the right answer for the wrong shape. Once I asked “what does each component actually need to know?”, the design fell into place naturally:

• The fetcher knows how to turn a URI into a page and a set of links. That’s all.
• The strategy knows how to traverse. It owns the frontier, the visited registry, and the concurrency model.
• The source knows how to push pages into a consumer. It knows nothing about indexing.
• The ingestion service wires them together in one line.

Each piece is independently testable, replaceable, and understandable in isolation.

The LLM gave me a working crawler in minutes. Thinking about the design gave me one I’m not embarrassed to show.

myIR is an open-source information retrieval laboratory built in Java. It covers lexical indexing, BM25 ranking, sparse vector search, and — eventually — hybrid retrieval, centroid-based summarization, and semantic graph analysis.

JSanca

Platform Architect exploring the boundaries between simplicity and scalability