The Web Scraper’s Cheat Code Nobody Talks About

I thought I was done. I was not done.

I was scraping a major quick-commerce platform, one of those 10-minute grocery delivery apps, and I was feeling pretty good about myself. I’d built a category crawler that walked their navigation tree, pulled subcategory pages, handled pagination, and collected product URLs. It ran overnight. When I checked the output in the morning: 5,287 products.

The platform’s own website says they carry 45,000+ products.

I had found 12% of the catalog. My crawler had essentially done the equivalent of walking into a supermarket, looking at the items on the endcap displays near the entrance, and declaring the inventory complete.

The naive fix is obvious: crawl harder. More categories, deeper pagination, more concurrency. But before I started that, I did what I should have done at the start. I checked robots.txt .

What I found there changed how I approach every scraping project now.

— -

Why Category Pages Are Lying to You

Here’s the thing about category pages on e-commerce sites: they’re merchandised. They’re not database dumps. A product manager decided what shows up on the “Dairy & Breakfast” landing page. An algorithm ranked those items by conversion rate. The pagination you’re crawling is a curated view of the catalog, not the catalog itself.

On this platform, a typical subcategory page surfaced 30–50 products. Some categories had “load more” buttons, some had pagination, some silently truncated at 48 items with no indication there was more. If you’re not paying close attention, you scrape those 48 items and move on, never knowing the category actually had 300 SKUs.

The other problem: subcategory depth. Navigation trees are designed for humans browsing at a high level. The “Snacks” category links to “Chips & Crisps”, “Biscuits”, “Namkeen”, but there’s no nav link to “Protein Bars > Whey-Based > 40g Single Serve”. That stuff exists in the catalog. It just isn’t surfaced in the nav tree because it would make the UI unusable.

Category crawling finds what marketing decided to show. Sitemaps find what the database actually contains.

— -

The Sitemap Discovery

robots.txt is the file at domain.com/robots.txt that tells web crawlers which paths they’re allowed to access. Every serious web developer knows it. Most scrapers check it to avoid getting blocked. What they often skip is the last section.

At the bottom of many robots.txt files, there’s a Sitemaps (sitemap.xml)directive that points to the XML sitemap. This is the file that sites submit to Google to say “here’s everything on my website, please index all of it.”

The platform’s robots.txt pointed to their sitemap index. The index contained dozens of shard files, each with up to 10,000 product URLs. The full catalog — every product the platform carries — was sitting there in static XML, no login required, no browser automation needed.

The numbers told a stark story. My overnight category crawler found a fraction of what the sitemap exposed in minutes — across two different platforms. The full comparison:

[embed]Category Crawling vs Sitemap — By the Numbers

No Playwright. No JavaScript rendering. No waiting for React to hydrate. Just HTTP requests to static XML files. Something was also clearly off about the total counts — the sitemap had far more entries than the platform claims to carry. Something was very wrong, or very interesting. (It turned out to be both. More on that later.)

— -

Why Companies Maintain Sitemaps So Carefully

Here’s the reason this works so well: sitemaps are for Google.

Getting indexed by Google is existential for an e-commerce company. If your product pages aren’t in Google’s index, nobody finds them via search. So there’s an entire SEO operation inside these companies whose job is to make sure the sitemap is comprehensive, accurate, and regularly updated. Product added to catalog? It goes in the sitemap within hours. Product gets a new image? Sitemap gets a <lastmod> update.

Category pages are maintained by product teams and UI designers. Sitemaps are maintained by people who are religiously accountable to Google Search Console. These are different standards of quality.

When I’m scraping a site’s category navigation, I’m at the mercy of whatever UI decisions got made. When I’m reading their sitemap, I’m reading the file they trust Google with. That’s a much better source.

— -

The Implementation

First, find the sitemap. Don’t hardcode it, parse robots.txt:

import re
from curl_cffi import requests as creqs

session = creqs.Session(impersonate="chrome110")

def find_sitemap(domain):
    r = session.get(f"https://{domain}/robots.txt")
    for line in r.text.splitlines():
        if line.lower().startswith("sitemap:"):
            return line.split(":", 1)[1].strip()
    return f"https://{domain}/sitemap.xml"

I’m using curl_cffi here instead of the standard requestslibrary because it impersonates a real browser’s TLS fingerprint. Some CDNs will block requests that look like Python’s default HTTP client. curl_cffi with impersonate="chrome110" gets through in almost every case I’ve encountered.

Next, the sitemap index gives you shard URLs:

def get_shards(index_url):
    r = session.get(index_url)
    return re.findall(r"<loc>(https://[^<]+)</loc>", r.text)

I’m using regex instead of an XML parser here deliberately. Sitemaps are supposed to be well-formed XML, but in practice they sometimes aren’t, mismatched namespaces, encoding issues, stray characters. A simple regex that grabs everything between <loc> tags is more resilient than a parser that throws on malformed input.

Then parse each shard:

def parse_shard(shard_url):
    r = session.get(shard_url)
    products = []
    for block in re.finditer(r"<url>(.*?)</url>", r.text, re.DOTALL):
        b = block.group(1)
        loc = re.search(r"<loc>(https://[^<]+)</loc>", b)
        img = re.search(r"<image:loc>([^<]+)</image:loc>", b)
        if loc:
            products.append({
                "url": loc.group(1),
                "image_url": img.group(1) if img else ""
            })
    return products

Run this across all shards with light concurrency, I used concurrent.futures.ThreadFuturePoolExecutor with 8 workers, and you’re done in 4 minutes with a CSV of every product URL and image URL on the platform.

— -

What You Get for Free

Two things come out of the sitemap that you’d normally have to scrape from product pages:

Image URLs: The <image:loc> extension tag is standard in Google’s image sitemap spec. Many platforms populate it for every product. So before you’ve made a single product page request, you already have the CDN URL for every product image. If your use case only needs images and slugs (say, you’re building a product catalog or doing image deduplication analysis), you might not need to scrape product pages at all.

Category from shard path: One of the platforms I scraped structured their shard URLs like this:

/sitemaps/products/dairy-breakfast/butter/sitemap-dairy-breakfast-butter.xml

The category and subcategory are right there in the path. You can extract them with a single regex split, which means you get category labels for every product in the catalog without touching a single product page. That’s the kind of information that would normally require either scraping the PDP’s breadcrumbs or maintaining a mapping table by hand.

The other platform’s shard URLs were just numbered, sitemap-products-1.xml, sitemap-products-2.xml, no category encoded. But the image URLs were clean, and the product slugs themselves contained useful category signals.

— -

The Twist: 237,760 Is Not 237,760 Products

Remember how the platform claims 45,000 products but the sitemap had 237,760 entries? I said this was both wrong and interesting. Here’s the interesting part.

When I started deduplicating by slug, I noticed something: tomato-local appeared 50 times. Not as duplicates, each entry had a different pvid (product variant ID) in the URL and a different image URL pointing to what was presumably a slightly different photo.

Quick-commerce platforms operate on a dark store model. They have hundreds of micro-warehouses across a city. Each dark store sources locally, the tomatoes at one location come from a different supplier than those at another. Each supplier relationship produces a distinct product entry with its own ID and its own product image.

When you deduplicate by slug, the sitemap inflation becomes obvious, the same product slug appears many times, each with a different variant ID and sometimes a different product image. Local produce is the worst offender; commodity products with supplier variants can appear dozens of times.

The math works out: the claimed catalog size × average dark-store variants per product ≈ sitemap entry count. It fits. (The full deduplication breakdown is in the table above.)

This isn’t a data quality problem. It’s a business architecture that leaked into the sitemap. Once you understand it, you can deduplicate by slug for product-level analysis, or keep the variant IDs if you need location-level granularity.

Combo packs are also counted separately, which inflates the number further. A “Butter 100g” and “Butter 3-Pack” are distinct entries. Reasonable. They’re different products, but worth knowing when you’re comparing catalog sizes.

— -

The Two-Phase Pipeline

Understanding this led me to restructure my entire scraping pipeline. The old approach was monolithic: crawl category pages, find product URLs, scrape PDPs, all in one pass. The new approach splits into two clean phases.

Phase 1: Discovery. Download all sitemap shards and build a catalog CSV. Columns: product_id, url, slug, image_url. This is pure HTTP, no browser, no rate limiting needed (these are static XML files served from CDN). Takes 4 minutes, produces a complete map of everything on the platform.

Phase 2: Extraction. Use the catalog CSV as input to a targeted PDP scraper. Now you’re making one request per product page to get price, stock status, ingredients, weight, and whatever else you need. This phase needs rate limiting and concurrency controls, but it’s operating on a known-good list of URLs rather than discovering them on the fly.

The separation matters for a few reasons. If Phase 2 fails halfway through (rate limited, IP blocked, whatever), you haven’t lost your discovery work. Restart from the CSV at the URL where you left off. Phase 1 can run daily to pick up new products without re-scraping all the PDPs. And the phases can be handed off: one person builds and maintains the catalog crawler, another works on the PDP extraction logic.

It also makes the product count question trivially answerable at any time. Want to know how many products a platform has today? Run Phase 1. Done in 4 minutes, no product page requests needed.

— -

The Caveats

Nothing this clean comes without gotchas.

Stale entries. Sitemaps lag reality. We found product IDs that returned 404s when we actually requested the page, products that had been discontinued but whose sitemap entries hadn’t been cleaned up. Not many, maybe 1–2% of entries, but enough to account for in your pipeline. Always check HTTP status before treating a sitemap entry as live inventory.

Not all sitemaps are product-level. Some sites have sitemaps that only list category pages or blog posts. A quick scan of the shard URLs or a sample of the <loc> values tells you immediately whether you’re dealing with product URLs or not. If the URLs look like /category/dairy-breakfast rather than /products/butter-100g/id/1234, you’re looking at a category sitemap and you’ll need to crawl from there.

Sitemap ≠ ground truth. The sitemap tells you what exists. It doesn’t tell you if it’s in stock, what the current price is, or whether the product is available in your delivery zone. For a live price/stock feed, you still need product page scraping. The sitemap gives you the universe of things to scrape, what it doesn’t give you is the state of those things right now. Think of it as a highly reliable table of contents, not a data extract.

— -

Check robots.txt First. Every Time

I’ve changed my personal scraping checklist based on this. Before I write a single line of crawler code, I do three things:

Open robots.txt. Read the whole thing. Note every Sitemap: directive. Then open the sitemap index and look at the shard structure before deciding on an approach.

This 10-minute investigation would have saved me the time building a category crawler that found 12% of the catalog. For other platforms, the payoff will be different, but you’ll know immediately whether the sitemap approach is viable before you’ve committed to anything.

The SEO team has already done the hard work of comprehensively enumerating every URL on the site. They did it for Google. You’re just reading the same file.

It’s the closest thing to a cheat code I’ve found in web scraping.

— -

All scraping was done for product catalog research purposes. The code snippets above use curl_cffi for HTTP and standard Python re for XML parsing. The full pipeline including PDP extraction and deduplication is a bit more involved, happy to cover that in a follow-up post if there’s interest.

Eightinity Engineering We share real-world AI and mobile engineering insights. Before you go:

👏 Show your support by clapping and following the author 🧠 Discover more AI, OpenAI, and SAM model articles 📱 Explore iOS, Android, and UI engineering blogs 🚀 Learn how we build AI-powered products at Eightinity 🔔 Follow us: **LinkedIn | [X (Twitter)](https://x.com/8inityStudio) | [Website](https://www.eightinity.in/)**