I Built a Scrapy Project for Superpages

Website Landing Page

Scraping a business directory sounds simple… until you actually try to do it properly.

At first, I thought this would be a quick script:

• Send requests
• Parse HTML
• Save data

But very quickly, it turned into something much deeper: A system design problem, not just scraping

This is the story of how I built a scalable, fault-tolerant Scrapy spider for scraping business listings from Superpages Australia.

🧠 The Real Challenge

The goal was simple:

Extract structured company data across:

• Multiple cities
• Multiple categories
• Paginated listings
• Individual business pages

But the real problems were:

• Duplicate data
• Broken or blocked requests
• Pagination loops
• Partial scraping runs
• Restart issues

This wasn’t just scraping anymore.

This was building a reliable data pipeline.

Designing the Crawl Like a System

Instead of writing a flat script, I designed a multi-stage crawl pipeline:

Homepage  
  → Cities  
    → Categories  
      → Listings (paginated)  
        → Company detail pages

Each stage became a dedicated function inside the spider.

Moving from Loops → Callbacks

The biggest shift when using Scrapy is:

You don’t loop. You chain requests.

Here’s how the crawl starts:

def start_requests(self):
    yield Request(url=self.base_url, headers=self.headers)

Then in parse():

def parse(self, response):
    cities = response.css('#container a')
    categories = response.css('.col-map')

    for city in cities:
        city_name = city.css('::text').get(default='').strip()
        city_url = urljoin(self.base_url, city.css('::attr(href)').get(default='').strip())

        for category in categories:
            category_name = category.css('::text').get('')
            category_url = urljoin(
                city_url.rstrip('/') + '/',
                category.css('a::attr(href)').get('').split('/')[-1]
            )

            yield Request(
                url=category_url,
                headers=self.headers,
                cookies=self.cookies,
                callback=self.parse_city,
                meta={'city': city_name, 'category': category_name}
            )

Instead of nested loops, each request carries its own context using meta.

🛡️ Eliminating Duplicate Data (Critical Step)

One of the biggest problems in scraping is duplication.

To solve this, I extracted a unique ID from each company URL:

company_id = company_url.split('/')[4]

Then I used a set:

if company_id not in self.seen_id:
    self.seen_id.add(company_id)

But I didn’t stop there.

I also loaded previously scraped IDs from existing files:

def load_existing_ids(self):
    output_dir = "output"

    if not os.path.exists(output_dir):
        return

    for file_name in os.listdir(output_dir):
        if file_name.endswith(".json"):
            file_path = os.path.join(output_dir, file_name)

            with open(file_path, 'r', encoding='utf-8') as f:
                data = json.load(f)

            for item in data:
                company_url = item.get("Url", "")
                parts = company_url.split('/')

                if len(parts) > 4:
                    company_id = parts[4].strip()
                    if company_id:
                        self.seen_id.add(company_id)

This made the scraper:

• Restart-safe
• uplicate-free
• Scalable

Handling Listings + Pagination

Once inside a category page:

def parse_city(self, response):
    city = response.meta.get('city', '')
    category = response.meta.get('category', '')

    company_urls = response.css(
        'h3.h6 a[data-yext-click="name"] ::attr(href)'
    ).getall()

Each company is processed:

for company in company_urls:
    company_url = urljoin(self.base_url, company)
    company_id  = company_url.split('/')[4]

    if company_id not in self.seen_id:
        self.seen_id.add(company_id)

        yield Request(
            url=company_url,
            headers=self.detail_headers,
            cookies=self.cookies,
            callback=self.parse_company,
            meta={'city': city, 'category': category}
        )

Then pagination:

next_page = response.css('.page-link:contains("Next") ::attr(href)').get('').strip()

if next_page:
    yield Request(
        url=urljoin(self.base_url, next_page),
        headers=self.headers,
        callback=self.parse_city,
        meta={'city': city}
    )

🧾 Extracting Structured Data

Inside parse_company():

def parse_company(self, response):
    category = response.meta.get('category')
    item = OrderedDict()

    item["Company / Business Name"] = response.css('title.h4::text').get('').strip()
    item['Phone'] = response.css('dt:contains("Phone Number") + dd::text').get('').strip()
    item['Mobile'] = response.css('dt:contains("Mobile Number") + dd::text').get('').strip()
    item["Company Website"] = response.css('[data-click="website"] ::attr(href)').get('')
    item['State'] = response.css('[data-address-county]::text').get('').strip()
    item['Address'] = ''.join(response.css('dt:contains("Home Address") + dd ::text').getall())

    item["Industry"] = category
    item['Country'] = 'Australia'
    item['Url'] = response.url
    item["Date Added / Updated"] = datetime.now().strftime("%b %d, %Y")

    yield item

This produces clean, structured JSON ready for:

• CRM systems
• lead generation
• analytics

🔁 Handling Retries & Stability

Instead of relying only on Scrapy defaults, I configured:

custom_settings = {
    'CONCURRENT_REQUESTS': 2,
    'RETRY_TIMES': 3,
    'RETRY_HTTP_CODES': [500, 502, 503, 504, 400, 403, 404, 408, 429, 401],
}

This helps handle:

• Rate limits
• Temporary server failures
• Blocked requests

Performance Strategy

I intentionally kept the crawler slow:

CONCURRENT_REQUESTS = 2

Why?

Because: Stability > Speed

A fast scraper that gets blocked is useless.

🧠 Key Lessons

1. Scraping is a system problem

Not just parsing HTML

2. Deduplication is mandatory

Always design for restarts

3. Scrapy shines with structure

Callbacks > loops

4. Reliability beats speed

Slow + stable wins long-term

Final Thought

Most beginners write scrapers like scripts.But real-world scraping?

It’s closer to building a resilient data pipeline

Once you think like that, your entire approach changes.

Note: Before Using correct Ensure your selectors.

If you’re building scraping systems or working with Scrapy, I’d love to hear your approach 👇