Building a High-Performance Web Crawler: A Deep Dive into Go’s Concurrency Patterns

“Don’t communicate by sharing memory, share memory by communicating.”

Concurrency is often described as the “final boss” of backend engineering. In most traditional languages, handling multiple tasks at once involves a stressful dance of mutexes, semaphores, and critical sections. You spend 10% of your time writing logic and 90% of your time building walls around your data, hoping your threads don’t trip over each other and cause a deadlock.

When Rob Pike and the Go team introduced the language, they offered a different philosophy based on Communicating Sequential Processes (CSP). The idea is simple but radical: Instead of using locks to coordinate access to shared state, use channels to pass the state itself between independent processes.

Goroutines and Channels: Go’s Concurrency Toolkit

Before we dive into the crawler’s architecture, let’s recap the two fundamental building blocks of Go’s concurrency model:

• Goroutines: A goroutine is a lightweight thread of execution managed by the Go runtime. Spawning one is as simple as prefixing any function call with the go keyword (e.g., go fetchURL() ). They have a tiny initial memory footprint and grow dynamically, allowing you to run hundreds of thousands of them concurrently on a single machine (https://go.dev/tour/concurrency/1).
• Channels: Channels are the pipelines that connect concurrent goroutines. You can send values into a channel from one goroutine and receive them in another. They handle the synchronization under the hood — ensuring safe communication without having to write low-level locking code (https://go.dev/tour/concurrency/2).

The “Go Way” in Practice

In this project — a high-performance, concurrent web crawler — I decided to take this proverb literally.

A web crawler is a perfect case study for this tension. By nature, it is highly parallel (fetching hundreds of pages at once) but also strictly stateful (you must track which URLs you’ve already visited to avoid infinite loops).

In a traditional architecture, you would protect a visited URLs map with a heavy global mutex, creating a massive bottleneck as your worker count grows.

Instead, I built an orchestrator-based system where:

• Workers never touch the visited map. They just fetch and return links.
• The coordinator is the only one that knows the state. It “shares memory” by sending jobs over channels.

The result? A system that processed 42,000 URLs with 100 concurrent workers — all without a single mutex in the core logic.

In this post, I’m going to break down the architecture of this crawler, from the nil-channel trick for job orchestration to the graceful shutdown patterns that ensure every byte of data is safely persisted to disk.

1. The Architecture: Decoupled State & Worker Pools

To keep the application lock-free, we divide responsibility into separate roles:

1. The Workers (Worker Pool): A fixed number of goroutines that run in the background. They do not know about the crawled history or how URLs link together. They simply read a URL from a channel, make the HTTP GET request, parse the links, and push a result object onto another channel.
2. The Coordinator: A single goroutine that holds the system’s state: the queue of pending URLs and the map of visited URLs. Since only the coordinator reads or writes these data structures, we don’t need locks.
3. The Storage Engine: A dedicated worker that consumes crawled results from a buffered channel and writes them sequentially to a JSON Lines (.jsonl) file.

2. The Core Coordinator & the “nil-channel” trick

At the heart of the orchestrator is a loop that coordinates sending jobs to workers and receiving crawled results. However, this creates a classic concurrency problem:

• If the queue is empty, we shouldn’t attempt to send jobs to workers (doing so would send garbage or block).
• We still need to listen for incoming results from workers that are currently fetching pages.
• We must also be ready to handle system shutdowns.

To solve this cleanly, we use Go’s nil-channel behavior in a select block. In Go, sending to or receiving from a nil channel blocks indefinitely. In a select statement, a blocked channel case is simply ignored.

Here is how the Coordinator utilizes this trick:

func (f *Fetcher) runCoordinator(
    ctx context.Context, 
    startURLs []string, 
    jobsChan chan Job, 
    resultsChan chan Result, 
    outChan chan Result, 
    wg *sync.WaitGroup,
) {
    defer func() {
        close(jobsChan)
        wg.Wait()
        close(outChan)
    }()

    visited := make(map[string]bool)
    var queue []Job

    for _, url := range startURLs {
        visited[url] = true
        queue = append(queue, Job{URL: url, Depth: 1})
    }

    activeJobs := len(startURLs)

    for activeJobs > 0 {
        var sendChan chan<- Job
        var nextJob Job

        // Enable the send case ONLY if we have jobs in the queue
        if len(queue) > 0 {
            sendChan = jobsChan
            nextJob = queue[0]
        }

        select {
        case <-ctx.Done():
            return

        // If sendChan is nil, this case is completely ignored!
        case sendChan <- nextJob:
            queue = queue[1:]

        case res := <-resultsChan:
            activeJobs--
            outChan <- res
            f.handleResult(res, visited, &queue, &activeJobs)
        }
    }
}

When the queue is empty, sendChan is nil. The case case sendChan <- nextJob is disabled, meaning we don’t spin or block the loop trying to send a job. Instead, the select statement waits exclusively for either the context cancellation (ctx.Done()) or for one of the active workers to finish a request (resultsChan).

As soon as a worker returns a result, handleResult processes it, finds new links, adds them to the queue, and increments activeJobs. On the next iteration, because len(queue) > 0, sendChan is reassigned back to jobsChan, and sending jobs to workers is re-enabled.

3. Graceful Shutdown: No data left behind

When building a scraper that processes tens of thousands of pages, you will inevitably want to interrupt it using Ctrl+C. If you hard-kill the process, you risk corrupted files and lost progress.

A clean shutdown in our pipeline requires a coordinated cascade of events:

1. Signal Catching: The main routine listens for SIGINT (interrupt) signals using os/signal.
2. Context Cancellation: Upon catching the signal, we invoke the context’s cancel() function.
3. Coordinator Exit: The Coordinator intercepts <-ctx.Done(), exits its run loop, and executes its deferred cleanup:
4. It closes jobsChan.
5. It waits for all workers to finish their current requests via wg.Wait().
6. It closes the output channel outChan.

Here is the CLI entrypoint in main.go orchestrating this flow:

func main() {
    cfg := cli.ParseFlags()

    ctx, cancel := context.WithCancel(context.Background())
    defer cancel()

    // Listen for OS Interrupt signals
    sigChan := make(chan os.Signal, 1)
    signal.Notify(sigChan, os.Interrupt)

    go func() {
        <-sigChan
        fmt.Println("\n[SHUTDOWN] Signal received! Cancelling outstanding tasks gracefully...")
        cancel() // Triggers the cascade
    }()

    store := crawler.NewJSONLStorage(cfg.FileName, 100)
    defer store.Close() // Flushes remaining buffers to disk last

    results := fetcher.Start(ctx, cfg.Seeds)

    for res := range results {
        fmt.Printf("[SUCCESS] %s -> Found %d links\n", res.URL, len(res.FoundLinks))
        store.Write(res)
    }
}

And inside json.go, the storage worker runs its own simple drain loop:

func (s *JSONLStorage) Close() {
    close(s.resultsChan) // Signal that no more writes are coming
    <-s.doneChan         // Wait for background routine to finish flushing
}

func (s *JSONLStorage) startWorker() {
    file, _ := os.OpenFile(s.fileName, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0644)
    defer file.Close()

    // Processes all items remaining in the channel even after Close() is called
    for result := range s.resultsChan {
        jsonData, _ := result.MarshalJSON()
        file.Write(append(jsonData, '\n'))
    }
    s.doneChan <- true
}

4. Key Takeaways

• State Isolation: Locking a structure is sometimes necessary, but isolating the structure to a single owner goroutine is often cleaner. The coordinator pattern turns shared memory issues into a sequential processing problem.
• Channel States matter: A nil channel blocks. A closed channel returns immediately with zero values. Mastering these details turns complex orchestration state-machines into elegant, short select loops.
• Buffered Channels as Speed Buffers: By using buffered channels for results and job scheduling, we prevent workers from blocking the coordinator (and vice-versa) during temporary network spikes.

The complete code, including tests and execution instructions can be found on my Github repository.

Remember that the program allows several parameters:

• URLs: a comma-separated list of urls to crawl. At least one url is mandatory.
• Depth: maximum depth boundary for recursive link extraction. Default set to 2.
• Workers: number of concurrent routines to use. Default set to 3.
• Timeout: in seconds, maximum wait time for a link to return a response. Default is 5 seconds.
• File: the name for the output file. Default is set to “crawl_output.txt”.
• Exclude: a comma-separated list of paths to exclude from the crawl. Example: “/docs/,/help/”

Feel free to share any thoughts or comments, as well as raising any issue you notice.