I Scraped 1 Million Pages in Python, Rust, and Go - The Performance Gap is Embarrassing (2026 Benchmarks)

It started with a simple Slack message at 11 PM.

“Hey, we need to rebuild the crawler. The current Python stack is eating $4,200/month in EC2 costs. Can you take a look?”

I stared at that message for a solid five minutes. Four thousand dollars. Every month. Just to fetch web pages.

I had been writing Python for eight years. It was comfortable. It was familiar. But that number — $4,200 — made me deeply uncomfortable. So I did what any engineer with too much caffeine and a stubborn streak would do: I decided to benchmark Python against Rust and Go by scraping one million real web pages.

What I discovered changed how I think about performance forever.

Why Web Scraping is the Ultimate Stress Test

Before we dive into the numbers, let’s talk about why web scraping is such a brutal benchmark.

Unlike a typical API server where you control the request rate, web scraping is a battle against network latency, I/O bottlenecks, memory pressure from parsing massive HTML documents, CPU cycles spent on regex and DOM parsing, and the endless headache of error handling — timeouts, retries, rate limits, and CAPTCHAs.

If a language can scrape efficiently, it can handle almost anything you throw at it.

The Setup: Keeping It Brutally Fair

I didn’t want fanboys screaming in the comments. So I optimized each stack to the best of my ability, using production-grade libraries and patterns.

What we scraped: The top 1 million domains from the Tranco list. For each domain, we fetched the homepage, extracted the page title, counted the total number of links, and measured the response time. Every request had a 10-second timeout. I tuned concurrency per language based on what each ecosystem could realistically handle. All tests ran on an AWS c7i.2xlarge instance with 8 vCPUs and 16 GB of RAM, on the same VPC, same subnet, during the same time window — 2 AM UTC to minimize network variance.

The contenders were Python 3.12 with aiohttp and uvloop, using BeautifulSoup4 and lxml for parsing, running 500 concurrent asyncio tasks. Go 1.22 used the standard library net/http client with golang.org/x/net/html for parsing, spinning up 2,000 goroutines. Rust 1.78 used reqwest with tokio, the scraper crate for CSS selectors, and 2,000 concurrent tokio tasks.

Every implementation used connection pooling, retry logic with exponential backoff, proper HTTP/2 where supported, identical User-Agent strings, and a 60-second warm-up before timing began.

The Results: Prepare to Be Uncomfortable

I ran each benchmark three times and averaged the results. Here’s what happened.

Time to complete 1 million pages: Python finished in 4 hours and 52 minutes, processing roughly 57 pages per second. Go finished in 1 hour and 14 minutes, processing about 225 pages per second — that’s 3.9 times faster than Python. Rust finished in just 42 minutes, processing about 397 pages per second — a staggering 7 times faster than Python.

Let that sink in. Rust finished before Python was even halfway done.

Memory usage at peak: Python peaked at 11.2 GB of RAM. At 500 concurrent connections, that works out to about 22 MB per 1,000 connections. Go peaked at 3.8 GB, or about 1.9 MB per 1,000 connections. Rust peaked at just 2.1 GB, roughly 1 MB per 1,000 connections. Python’s memory footprint was over 5 times higher than Rust. On a 16 GB machine, I was sweating bullets watching Python approach the OOM killer.

CPU utilization: Python averaged 78 percent CPU usage, with clear GIL contention under heavy load. Go averaged 92 percent, the scheduler humming with minimal overhead. Rust averaged 96 percent, proving that zero-cost abstractions actually mean zero cost.

Estimated cloud cost per 1 million pages on on-demand EC2: Python cost about $1.80 in compute time. Go cost about $0.44. Rust cost about $0.26. Scale that up to 100 million pages per day, and Python costs $5,184 per month versus Rust’s $748 per month. That’s a $53,000 per year difference for a single crawler.

What the Code Actually Looks Like

I know what you’re thinking. Sure, Rust is fast, but the code must be a nightmare. Here’s a side-by-side of the core scraping loop.

Python — the comfortable old friend. About 25 lines of code. Beautifully readable. Performance? We already know the story.

import asyncio
import aiohttp
from bs4 import BeautifulSoup

async def fetch_page(session, url):
    async with session.get(url, timeout=10) as response:
        html = await response.text()
        soup = BeautifulSoup(html, 'lxml')
        title = soup.find('title')
        links = len(soup.find_all('a'))
        return {'url': url, 'title': title.text if title else None, 'links': links}

async def main(urls):
    connector = aiohttp.TCPConnector(limit=500)
    async with aiohttp.ClientSession(connector=connector) as session:
        tasks = [fetch_page(session, url) for url in urls]
        return await asyncio.gather(*tasks, return_exceptions=True)

Go — the pragmatic workhorse. About 40 lines. Explicit, verbose error handling, but bulletproof. No third-party HTTP client needed.

package main

import (
    "net/http"
    "golang.org/x/net/html"
    "sync"
)
func fetchPage(url string, wg *sync.WaitGroup, results chan<- Result) {
    defer wg.Done()
    resp, err := http.Get(url)
    if err != nil {
        results <- Result{URL: url, Error: err}
        return
    }
    defer resp.Body.Close()
    doc, _ := html.Parse(resp.Body)
    title, links := extractData(doc)
    results <- Result{URL: url, Title: title, Links: links}
}
func main() {
    var wg sync.WaitGroup
    results := make(chan Result, len(urls))
    sem := make(chan struct{}, 2000)
    for _, url := range urls {
        wg.Add(1)
        sem <- struct{}{}
        go func(u string) {
            defer func() { <-sem }()
            fetchPage(u, &wg, results)
        }(url)
    }
    wg.Wait()
    close(results)
}

Rust — the obsessive perfectionist. About 35 lines. The compiler is your strict but fair mentor. If you’re used to it, the readability is solid.

use reqwest;
use scraper::{Html, Selector};
use tokio;
use futures::stream::{self, StreamExt};

async fn fetch_page(client: &reqwest::Client, url: &str) -> Result<PageData, reqwest::Error> {
    let body = client.get(url).send().await?.text().await?;
    let document = Html::parse_document(&body);
    let title_selector = Selector::parse("title").unwrap();
    let link_selector = Selector::parse("a").unwrap();
    let title = document.select(&title_selector).next()
        .map(|e| e.inner_html());
    let links = document.select(&link_selector).count();
    Ok(PageData { url: url.to_string(), title, links })
}
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let client = reqwest::Client::builder()
        .pool_max_idle_per_host(100)
        .build()?;
    let results: Vec<_> = stream::iter(urls)
        .map(|url| fetch_page(&client, url))
        .buffer_unordered(2000)
        .collect()
        .await;
    Ok(())
}

The Surprises Nobody Talks About

Python’s uvloop is a game-changer. Without uvloop, Python took 7 hours and 12 minutes. With it? 4 hours and 52 minutes. That’s a 32 percent improvement just by swapping the event loop. If you’re stuck on Python, at least use uvloop.

Go’s standard library is unfairly good. I didn’t need a single third-party HTTP client. Go’s net/http with proper connection pooling outperformed Python’s entire async ecosystem. The standard library is Go’s secret weapon.

Rust compile times hurt — until they don’t. The Rust binary took 4 minutes to compile in release mode. But once compiled, it ran for 42 minutes. Total time: 46 minutes. Python started instantly but ran for 292 minutes. Rust won even including compile time.

Error rates were nearly identical across all three languages. Python succeeded 94.2 percent of the time, Go 94.5 percent, and Rust 94.3 percent. Timeout rates hovered around 4 percent for all three. The language didn’t meaningfully affect reliability. The network did.

The Honest Trade-Offs

I’m not here to tell you to rewrite everything in Rust. I’m here to give you the data so you can make an informed choice.

Choose Python when you’re prototyping or building an MVP, your team doesn’t know Rust or Go, you need rapid iteration with no compile step, your scraping volume is under 100,000 pages per day, or you rely on heavy machine learning or NLP post-processing.

Choose Go when you need a balance of performance and simplicity, your team values readability over raw speed, you’re building long-running services rather than scripts, or you want a gentle learning curve from Python.

Choose Rust when you’re scraping at massive scale with over a million pages per day, infrastructure cost is a primary concern, you need predictable low-latency performance, you’re willing to invest in the learning curve, or you simply sleep better knowing the compiler checked your memory safety.

What I Actually Did at Work

Here’s the part I wasn’t expecting. I didn’t rewrite the entire crawler in Rust. I did something smarter.

I kept Python for the orchestration layer — scheduling, deduplication, database writes, and machine learning inference. Python is genuinely great at that. Then I wrote the fetching layer in Rust as a microservice exposing a gRPC API. Python calls it, Rust does the heavy lifting, and everyone wins.

The result? EC2 costs dropped from $4,200 per month to $1,100 per month. The Rust service uses a single c7i.xlarge. Python handles the logic on a c7i.large.

Sometimes the best architecture is a polyglot one.

The Bottom Line

The performance gap isn’t just embarrassing — it’s expensive.

Python is the turtle: slow, memory-hungry, but incredibly comfortable and fast to develop in. Go is the hare: quick, lean, pragmatic, and easy to pick up. Rust is the rocket: blazing fast, memory-efficient, but demands respect and patience.

Developer speed? Python wins. Production cost? Rust wins. Learning curve? Go wins.

If you’re running a scraper at scale, the data is unambiguous. Rust and Go aren’t just faster — they’re dramatically more cost-efficient. Python remains unbeatable for developer velocity and ecosystem richness.

The real question isn’t which language is best. It’s what’s the right tool for your scale.

At 1 million pages, the answer might surprise you. It sure surprised me.