From Cron Jobs to Orchestration: Building Production-Grade Web Scraping Systems with Prefect

Most web scraping projects start the same way. A script is written, a cron job is configured, and the data starts flowing. The setup is clean, minimal, and entirely appropriate for the scale it was designed to serve. For a while, everything works exactly as expected.

Then the pipeline grows. More pages, more categories, more data sources, more business logic between extraction and storage. And with that growth comes a category of problems that a cron job was never built to handle.

This is not a criticism of cron. It is one of the most reliable, well-understood pieces of Unix infrastructure ever created. But its role is narrow: run a command at a specified time. What it cannot do is understand the state of what it is running, reason about whether a previous run completed successfully, retry a failed step without rerunning the entire script, detect that a browser crashed halfway through a crawl, or prevent two overlapping runs from writing duplicate data to the same database.

For simple scraping at modest scale, those limitations are invisible. For production scraping infrastructure handling thousands of pages daily across multiple domains with proxies, JavaScript rendering, and structured data extraction pipelines, they become the dominant source of engineering pain.

The Anatomy of a Broken Cron-Based Pipeline

The failure modes that emerge as a cron-based scraping setup scales follow a predictable pattern.

The most common starting point is the website blocking the requests. Proxies are introduced to distribute the traffic load, but proxies fail intermittently. A proxy goes down mid-run, and the scraper either crashes entirely or silently produces incomplete results. The cron job does not know the difference between a run that completed successfully and a run that extracted half the data before the proxy connection dropped.

Browser-based scraping adds another layer of fragility. JavaScript-rendered pages require a headless browser, and headless browsers are resource-intensive processes that crash under load, time out on slow pages, and accumulate memory over long runs until the system runs out of it. When the browser crashes, the run typically fails silently or with a generic error message that does not identify which pages were successfully scraped before the failure.

CAPTCHAs introduce nondeterministic failures. A page that scraped cleanly yesterday may present a CAPTCHA today, blocking the entire run without any clear signal about what changed or what state the scraper was in when it encountered the block.

Concurrency problems compound all of these. If a run takes longer than the scheduled interval, the next scheduled run begins while the previous one is still in progress. Both runs attempt to write to the same database tables, producing duplicate records that are expensive to detect and remove and sometimes impossible to prevent without adding locking logic that the cron scheduler cannot provide.

The deepest problem is observability. When something goes wrong in a cron-based pipeline, the typical discovery mechanism is noticing that the database has not been updated recently, or that a downstream report is missing data. The cron job ran, or it did not. The script produced output, or it did not. The intermediate states, which steps completed, which failed, which products were extracted and which were missed, are either logged to a text file that nobody reads or not recorded at all.

What Orchestration Actually Means

Workflow orchestration addresses these problems by introducing a layer of infrastructure that understands the state of every step in a pipeline. Rather than a scheduler that fires a command and forgets about it, an orchestrator tracks the execution of every task, records its outcome, manages retries, handles dependencies between steps, and provides a unified view of what is running, what succeeded, what failed, and why.

Prefect is one of the most capable and developer-friendly orchestration frameworks available for Python-based data pipelines, and it is particularly well suited to web scraping workflows because of how naturally its concepts map onto the structure of a real scraping operation.

The central mental model shift that Prefect enables is the move from “running a script” to “managing a system.” The scraping logic does not change. The data still needs to be extracted, cleaned, structured, and stored. But the infrastructure around that logic changes completely, and the change in what you can observe, control, and recover from is transformative.

Flows, Tasks, and State

In Prefect, a workflow is composed of flows and tasks. A flow is a Python function decorated with @flow that defines the top-level logic of a pipeline. Tasks are individual units of work decorated with @task that the flow coordinates.

The key capability that makes this different from a plain Python script is that every task execution has a state. That state is one of several possible values: running, completed, failed, retrying, or cached. Prefect tracks these states persistently, which means that when something goes wrong, the system knows exactly which task failed, what the error was, and whether it has been seen before.

A basic Prefect task with retry logic looks like this:

from prefect import flow, task
from prefect.tasks import task_input_hash
from datetime import timedelta
@task(retries=3, retry_delay_seconds=30, cache_key_fn=task_input_hash, cache_expiration=timedelta(hours=1))
def scrape_product_page(url: str) -> dict:
    # browser or requests-based scraping logic here
    response = fetch_page(url)
    return extract_product_data(response)
@flow(name="scrape-products")
def scrape_products_flow(category_urls: list[str]):
    results = scrape_product_page.map(category_urls)
    return results

The retries=3 parameter instructs Prefect to automatically retry the task up to three times if it fails, with a 30-second delay between attempts. The cache_key_fn and cache_expiration parameters tell Prefect to cache the result of a successful scrape for one hour: if the same URL is requested again within that window and the task is marked completed, Prefect returns the cached result without executing the scraping logic again.

These three behaviors, automatic retry, configurable delay, and result caching, address the three most common sources of scraping failures in a single decorator. The scraping function itself does not need to implement any of this logic.

Decomposing a Large Scraping Pipeline into Flows

One of the most practically valuable aspects of Prefect for production scraping is the ability to decompose a large, monolithic script into a hierarchy of focused flows and subflows. A scraping pipeline that previously lived in a single file with hundreds of lines of sequential logic can be restructured into a set of independently observable, independently retryable components.

A production scraping pipeline organized as Prefect flows might look like this:

from prefect import flow, task
@flow(name="discover-urls")
def discover_urls_flow(base_url: str) -> list[str]:
    sitemap = fetch_sitemap(base_url)
    return parse_product_urls(sitemap)
@flow(name="scrape-products")
def scrape_products_flow(urls: list[str]) -> list[dict]:
    raw_pages = scrape_product_page.map(urls)
    return raw_pages
@flow(name="clean-html")
def clean_html_flow(raw_pages: list[dict]) -> list[dict]:
    return [clean_page(page) for page in raw_pages]
@flow(name="extract-data")
def extract_data_flow(clean_pages: list[dict]) -> list[dict]:
    return [extract_structured_data(page) for page in clean_pages]
@flow(name="save-database")
def save_database_flow(products: list[dict]):
    upsert_products(products)
@flow(name="send-alerts")
def send_alerts_flow(products: list[dict]):
    new_products = filter_new(products)
    if new_products:
        notify_team(new_products)
@flow(name="main-pipeline")
def main_pipeline(base_url: str):
    urls = discover_urls_flow(base_url)
    raw = scrape_products_flow(urls)
    clean = clean_html_flow(raw)
    structured = extract_data_flow(clean)
    save_database_flow(structured)
    send_alerts_flow(structured)

Each subflow in this structure has its own monitoring, its own retry configuration, its own logs, and its own state history in the Prefect UI. If the scrape_products_flow fails on a specific batch of URLs because a proxy goes down, that failure is isolated to that subflow. The discover_urls_flow that ran before it has already completed and its result is available. When the issue is resolved and the pipeline is retried, it can resume from the failed subflow rather than re-running the entire pipeline from the beginning.

This is the structural difference that transforms a fragile script into a resilient system. The pipeline can fail at any point, recover from that point, and produce complete results without duplicating work or losing data from earlier stages.

Proxy Rotation and Browser Crash Recovery

Two of the most practically useful applications of Prefect’s state management in scraping contexts are proxy failure handling and browser crash recovery.

For proxy rotation, a task that wraps a proxy-based request can be configured to detect proxy-specific failure modes and retry with a different proxy from a pool:

@task(retries=5, retry_delay_seconds=10)
def scrape_with_proxy(url: str, proxy_pool: list[str]) -> str:
    proxy = select_proxy(proxy_pool)
    try:
        return fetch_with_proxy(url, proxy)
    except ProxyConnectionError:
        mark_proxy_failed(proxy)
        raise  # Prefect will retry with a fresh proxy on the next attempt

For browser-based scraping, Playwright or Selenium sessions can be managed within a task context that ensures proper cleanup on failure:

@task(retries=2, retry_delay_seconds=60)
def scrape_js_page(url: str) -> dict:
    with sync_playwright() as p:
        browser = p.chromium.launch()
        try:
            page = browser.new_page()
            page.goto(url, timeout=30000)
            content = page.content()
            return parse_content(content)
        except Exception as e:
            raise e
        finally:
            browser.close()

When the browser crashes or times out, the task is marked as failed, the browser process is cleaned up by the finally block, and Prefect schedules a retry after the configured delay. The failure is recorded with the full exception traceback, the URL that was being scraped when the failure occurred, and the timestamp, giving the operator complete information about what happened and when.

Scheduling, Deployment, and the Prefect UI

Prefect provides its own scheduling mechanism that replaces the cron job for orchestrated pipelines. Deployments can be configured with cron-style schedules, interval-based schedules, or event-driven triggers:

from prefect.deployments import Deployment
from prefect.server.schemas.schedules import CronSchedule
deployment = Deployment.build_from_flow(
    flow=main_pipeline,
    name="daily-product-scrape",
    schedule=CronSchedule(cron="0 6 * * *"),
    parameters={"base_url": "https://example-shop.com"}
)
deployment.apply()

What this deployment provides beyond a traditional cron job is the full Prefect observability layer. The Prefect UI shows every run of this deployment, its current state, the duration of each subflow and task, any retries that occurred, the logs from each task, and the history of all previous runs. When a run fails, the failure is visible immediately in the UI with the full error context, not discovered hours later when someone notices the database has not been updated.

The concurrency controls that prevent overlapping runs are also built into the deployment configuration, eliminating one of the most damaging failure modes of cron-based pipelines without requiring any custom locking logic in the scraping code itself.

The Mindset Shift: From Running Scripts to Managing Systems

The technical capabilities that Prefect adds to a scraping operation are significant, but the more important change is conceptual. There is a meaningful difference between the mental model of someone who runs a scraping script and someone who manages a scraping system.

Running a script means caring about whether the script produced the expected output. The script either ran and the data is there, or it did not and the data is missing. The internal states, the progress, the partial completions, the transient failures and recoveries, are invisible.

Managing a system means having a continuous, observable picture of the entire pipeline. Which categories have been scraped today and which have not. How many products were extracted, how many failed, and why. Whether the failure rate is higher than usual, which might indicate a website change or a proxy configuration issue. Whether the data extraction logic is producing anomalies that suggest the HTML structure of the source has changed.

This shift from script-runner to system-manager is what makes the difference between a scraping setup that works until it breaks and nobody knows why, and a scraping infrastructure that can handle thousands of pages daily, recover from failures automatically, alert the team when human intervention is needed, and scale incrementally as the data requirements grow.

Conclusion

Cron jobs are the right starting point for web scraping. They are simple, reliable, and require almost no infrastructure to operate. But they are not the right ending point for any scraping operation that needs to run reliably at scale in production.

Workflow orchestration with Prefect provides the state awareness, retry logic, result caching, observability, and pipeline decomposition that production scraping infrastructure requires. The migration from a cron job to a Prefect flow does not require rewriting the scraping logic. It requires wrapping that logic in a framework that understands it, tracks it, and manages it with the same discipline that any other production system deserves.

The result is not just more reliable scraping. It is a fundamentally different relationship with the system, one where failures are visible, recoverable, and informative rather than silent, destructive, and mysterious.