How to Extract E-Commerce Data with Crawl4AI and Pydantic

Stop wasting hours fixing broken BeautifulSoup scripts every time a competitor changes a CSS class. Here is how to extract perfect e-commerce data in 35 lines of Python using Crawl4AI.

Image generated with Google Gemini

HTML scrapers break. It is the most frustrating law of software development.

You spend hours writing the perfect Python script using BeautifulSoup or Selenium to monitor a competitor’s pricing page. You meticulously map out every HTML tag: find_all('div', class_='price-tag-v2-bold'). It works flawlessly on Friday.

By Monday morning, the competitor pushes a site update, changes their CSS classes, and your scraper instantly crashes. You are back to square one.

If you are still writing hardcoded CSS selectors in 2026 to extract data, you are fighting a losing battle. The web is too dynamic.

The new standard is the Resilient Scraper. Instead of targeting fragile HTML tags, we can use an LLM to read the raw Markdown of a page and extract exactly what we need into a strict, structured JSON format. Even if the website completely redesigns its frontend, the scraper still works.

Here is exactly how to build one in 35 lines of Python using the open-source library Crawl4AI.

The Build: A 35-Line Resilient Scraper

To build our resilient scraper, we need two components: a crawler that converts messy web pages into clean text, and an AI model that structures that text.

In 2026, Crawl4AI is the open-source champion for this. It is a lightweight Python framework that spins up a headless browser, bypasses basic bot protections, and automatically converts bloated HTML into highly optimized Markdown — saving you massive amounts of API tokens.

Let’s build a script that targets a competitor’s product page and extracts the product name, the current price, and whether the item is in stock.

First, install the required packages in your terminal:

pip install -U crawl4ai pydantic openai
crawl4ai-setup

(Note: The setup command automatically installs the required headless Chromium browser).

Next, create a file named resilient_scraper.py and paste the following code:

import os
import asyncio
from pydantic import BaseModel, Field
from crawl4ai import AsyncWebCrawler
from crawl4ai.extraction_strategy import LLMExtractionStrategy

# 1. Define the exact structure we want the AI to return
class ProductData(BaseModel):
    product_name: str = Field(description="The full name of the product")
    price: float = Field(description="The current price as a number, excluding currency symbols")
    in_stock: bool = Field(description="True if the item is currently in stock, False otherwise")

async def main():
    # Set your OpenAI API key (or use a local model like Ollama)
    api_key = os.getenv("OPENAI_API_KEY")
    target_url = "https://example.com/ecommerce/mock-product"

    # 2. Configure the LLM Extraction Strategy
    strategy = LLMExtractionStrategy(
        provider="openai/gpt-4o-mini", # Use a cheap, fast model for basic reading
        api_token=api_key,
        schema=ProductData.model_json_schema(),
        extraction_type="schema",
        instruction="Extract the core product details from this page. Ignore ads, footers, and reviews."
    )

    # 3. Launch the crawler and execute
    async with AsyncWebCrawler(verbose=True) as crawler:
        result = await crawler.arun(
            url=target_url,
            extraction_strategy=strategy,
            bypass_cache=True
        )

        if result.success:
            print("Extraction Successful! Here is the clean JSON:")
            print(result.extracted_content)
        else:
            print("Failed to crawl the page.")

if __name__ == "__main__":
    asyncio.run(main())

How the Code Works

1. The Pydantic Schema: The ProductData class acts as an unbreakable contract. By defining field types (like ensuring price is a float), we guarantee the LLM will not return a messy string like "$49.99 (On Sale!)". It will strictly return 49.99, which can be immediately inserted into a database.
2. The LLMExtractionStrategy: This tells Crawl4AI to take the Markdown it scraped and pass it directly to gpt-4o-mini along with our Pydantic schema. We use a cheap "mini" model because this is a simple reading task, costing a fraction of a cent per page.
3. The Asynchronous Crawler: AsyncWebCrawler handles the heavy lifting of rendering JavaScript, waiting for the page to load, and converting it to text before the LLM even sees it.

Decoupling Data from Design

The true power of the Resilient Scraper is its adaptability.

Think about what happens when the competitor redesigns their product page next month. The HTML <div> tags will change. The CSS class names will change. A traditional BeautifulSoup scraper will throw a fatal NoneType error and crash your monitoring pipeline.

But our Crawl4AI script will not break.

The LLM simply reads the new layout, recognizes the context, finds the new price location, and maps it back to our strict Pydantic schema perfectly. You have effectively decoupled your data extraction from the website’s visual design.

Stop maintaining fragile CSS selectors and spending your weekends fixing broken data pipelines. Upgrade to an LLM extraction strategy, and your web scrapers will finally become unbreakable.

If you found this guide valuable, don’t forget to 👏 clap and subscribe so you don’t miss the next deep dive into AI system architectures.

This story is published on How To Profit AI. Connect with us on LinkedIn to stay in the loop with the latest AI stories.

Subscribe to our Newsletter for the latest on AI. Get updates on (Profit), (Prompts), (Agents), (Tools), and real-world examples for leaders in the AI economy.