Scrapy Playwright Tutorial: How to Scrape JavaScript Pages With Proxies

A practical guide to using Scrapy Playwright for JavaScript scraping, dynamic pages, proxy setup, and cleaner browser-based crawling.

Scrapy Playwright lets Scrapy render JavaScript-heavy pages by sending selected requests through Playwright instead of Scrapy’s default downloader. It is useful when content loads after page render, scrolling, clicks, or API calls. For proxy-backed scraping, pair each Playwright session with consistent cookies, headers, browser settings, and clean HTTP proxies.

Why Use Scrapy With Playwright?

Scrapy is fast, asynchronous, and built for structured crawling. It is excellent when the data you need is available in the raw HTML response. Scrapy’s official documentation describes the basic workflow as creating a project, writing spiders, extracting data, and exporting results.

But not every website works that way.

Many modern websites render content with JavaScript. Product cards, prices, reviews, availability, search results, comments, and pagination may appear only after the browser loads scripts. A normal Scrapy request may return HTML, but the important data might be missing.

That is where Scrapy Playwright helps.

The scrapy-playwright package is a Scrapy download handler that uses Playwright for Python to process requests that need JavaScript while still keeping the normal Scrapy workflow for scheduling, parsing, and item pipelines.

In plain English: Scrapy still manages the crawl, but Playwright renders the pages that need a browser.

When Should You Use Scrapy Playwright?

Use Scrapy Playwright when the data is not available in the initial HTML.

Good use cases include:

• JavaScript-rendered product pages
• Infinite scroll pages
• Search result pages loaded by client-side scripts
• Pages requiring button clicks before data appears
• Sites where content loads after network requests
• Pages where Scrapy sees placeholders instead of real data

Do not use Playwright for every page by default.

A real browser is heavier than a normal HTTP request. If the data is already available in HTML, standard Scrapy is usually faster and cheaper.

The best setup is selective: use Scrapy for normal pages and Playwright only for pages that need JavaScript rendering.

Basic Installation

A typical setup starts with Scrapy, Playwright, and Scrapy Playwright.

pip install scrapy scrapy-playwright
playwright install

Then create a Scrapy project:

scrapy startproject demo_scraper
cd demo_scraper

Inside your project settings, enable the Playwright download handler.

# settings.py

DOWNLOAD_HANDLERS = {
    "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
    "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
}

TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"

This tells Scrapy how to process requests that use Playwright.

A Simple Scrapy Playwright Spider

Here is a basic spider structure:

import scrapy

class ExampleSpider(scrapy.Spider):
    name = "example_playwright"

    def start_requests(self):
        yield scrapy.Request(
            "https://example.com",
            meta={
                "playwright": True,
            },
            callback=self.parse,
        )

    def parse(self, response):
        yield {
            "title": response.css("title::text").get(),
            "url": response.url,
        }

The important part is:

meta={"playwright": True}

That tells Scrapy to use Playwright for that request.

Without it, Scrapy handles the request normally.

This selective control is one of the biggest advantages of using Scrapy Playwright. You do not have to turn your whole crawler into a browser automation project. You can use browser rendering only where it is needed.

Waiting for JavaScript Content

Many JavaScript pages need time before the target data appears.

You can wait for a selector before parsing the page.

from scrapy_playwright.page import PageMethod
import scrapy

class ProductSpider(scrapy.Spider):
    name = "products"

    def start_requests(self):
        yield scrapy.Request(
            "https://example.com/products",
            meta={
                "playwright": True,
                "playwright_page_methods": [
                    PageMethod("wait_for_selector", ".product-card")
                ],
            },
            callback=self.parse,
        )

    def parse(self, response):
        for product in response.css(".product-card"):
            yield {
                "name": product.css(".product-title::text").get(),
                "price": product.css(".price::text").get(),
            }

This is better than using random sleep delays.

Waiting for a selector is more reliable because the spider continues only when the expected content is actually present.

Handling Infinite Scroll

Some websites load more content as the user scrolls.

Scrapy Playwright can simulate that with page methods.

from scrapy_playwright.page import PageMethod
import scrapy

class ScrollSpider(scrapy.Spider):
    name = "scroll_products"

    def start_requests(self):
        yield scrapy.Request(
            "https://example.com/products",
            meta={
                "playwright": True,
                "playwright_page_methods": [
                    PageMethod("wait_for_selector", ".product-card"),
                    PageMethod("evaluate", "window.scrollBy(0, document.body.scrollHeight)"),
                    PageMethod("wait_for_timeout", 2000),
                ],
            },
            callback=self.parse,
        )

    def parse(self, response):
        for product in response.css(".product-card"):
            yield {
                "name": product.css(".product-title::text").get(),
                "price": product.css(".price::text").get(),
            }

This is a simple example. In production, you may need repeated scrolling until no new items load.

But keep it controlled. Endless scrolling loops can trigger bot detection, waste resources, and create unstable crawls.

Using Proxies With Scrapy Playwright

Proxy handling is one of the main reasons developers combine Playwright with Scrapy carefully.

Scrapy Playwright supports proxies through Playwright launch options. Older package documentation shows proxy support through PLAYWRIGHT_LAUNCH_OPTIONS using a proxy object with server, username, and password fields.

A simplified settings example looks like this:

# settings.py

PLAYWRIGHT_LAUNCH_OPTIONS = {
    "proxy": {
        "server": "http://proxy-host:port",
        "username": "proxy_user",
        "password": "proxy_pass",
    }
}

For production scraping, use clean HTTP proxies that match your target workflow. A proxy that connects is not automatically a good scraping proxy. You need stable sessions, reasonable latency, clean IP reputation, and authentication that works with your Playwright setup.

For more on Playwright proxy workflows, read this guide on how developers use proxies with Puppeteer and Playwright.

Common Proxy Mistakes

The biggest proxy mistake is rotating too aggressively.

A browser session should usually have continuity. If the same cookies appear from different IPs every few seconds, the session looks suspicious. If every Playwright request uses a new proxy but the same fingerprint, that also creates a pattern.

Avoid these mistakes:

• Rotating IPs on every page during one session
• Reusing cookies across unrelated proxies
• Pairing a U.S. proxy with a mismatched timezone
• Using one browser fingerprint across every proxy
• Retrying blocked pages too aggressively
• Assuming HTTP 200 means the page was scraped correctly

A better setup keeps proxy identity, browser profile, cookies, and session timing aligned.

Scrapy Playwright vs Plain Playwright

Why not just use Playwright alone?

You can.

Plain Playwright is excellent for browser automation. But Scrapy gives you stronger crawling structure.

Scrapy helps with:

• Request scheduling
• Link following
• Item extraction
• Pipelines
• Exporting data
• Retry logic
• Middleware
• Crawl organization
• Large spider projects

Playwright helps with:

• JavaScript rendering
• Browser interactions
• Waiting for dynamic content
• Clicking and scrolling
• Browser state

Together, they work well when you need both crawling structure and browser rendering.

Best Practices for Scrapy Playwright

Use Playwright only where needed. Wait for selectors instead of relying on fixed sleeps. Keep browser sessions consistent. Use separate cookie jars for separate identities. Monitor response content, not just status codes. Throttle requests on sensitive websites. Do not scrape pages faster than a real user would browse. Log proxy errors, timeouts, CAPTCHAs, and soft blocks. Validate extracted data before assuming the crawl worked.

Scrapy Playwright is powerful, but it is not a magic anti-detection tool.

Final Thoughts

Scrapy Playwright is useful when normal Scrapy cannot see JavaScript-rendered content. It lets you keep Scrapy’s crawl structure while using Playwright for pages that need browser rendering.

The best use case is selective rendering. Do not run every request through a browser unless you need to.

For proxy-backed scraping, focus on consistency. Match the proxy, browser profile, cookies, headers, timezone, and request timing. Scrapy Playwright can make JavaScript scraping easier, but your session design still determines whether the scraper runs smoothly or gets flagged.