Async Web Scraping in PHP with Guzzle

A sequential scraper waits for each response before sending the next request. On a 50-page crawl at 500ms per page, that’s 25 seconds of sitting idle. Guzzle’s async Pool cut that to ~3 seconds in my testing.

But async in PHP has a catch: memory. Every in-flight request holds its response body, and DOM parsing compounds fast. I saw scrapers balloon past a gigabyte on a few thousand pages because nobody thought about cleanup.

Concurrent Requests with Guzzle Pool

Pool controls how many requests are in flight at any time. Here’s a complete example scraping all 50 pages of Books to Scrape:

$client = new Client([
    'timeout' => 15,
    'headers' => ['User-Agent' => 'Mozilla/5.0'],
]);

$requests = function () {
    for ($page = 1; $page <= 50; $page++) {
        yield new Request('GET', "https://books.toscrape.com/catalogue/page-{$page}.html");
    }
};

$allBooks = [];

$pool = new Pool($client, $requests(), [
    'concurrency' => 10,

    'fulfilled' => function ($response, $index) use (&$allBooks) {
        $crawler = new Crawler($response->getBody()->getContents());
        $crawler->filter('article.product_pod')->each(
            function ($node) use (&$allBooks, $index) {
                $allBooks[] = [
                    'page'  => $index + 1,
                    'title' => $node->filter('h3 a')->attr('title'),
                    'price' => $node->filter('.price_color')->text(),
                ];
            }
        );
    },

    'rejected' => function ($reason, $index) {
        echo "Page " . ($index + 1) . " failed: {$reason->getMessage()}\n";
    },
]);

$pool->promise()->wait();
echo "Scraped " . count($allBooks) . " books from 50 pages\n";

Scraped 1000 books from 50 pages in 2.87s

The generator function creates Request objects lazily. If you’re crawling 10,000 pages, you don’t want 10,000 objects in memory upfront. I start concurrency at 5–10 and adjust based on how the server responds.

Scraping JavaScript-Rendered Pages

When Guzzle returns <div id="app"></div> with nothing inside, the content is rendered client-side. You need Symfony Panther, which controls a real Chrome instance through ChromeDriver:

$client = PantherClient::createChromeClient(null, [
    '--headless=new',
    '--disable-gpu',
    '--no-sandbox',
]);

try {
    $crawler = $client->request('GET', 'https://quotes.toscrape.com/js/');
    $client->waitFor('.quote');

    $quotes = $crawler->filter('.quote')->each(function ($node) {
        return [
            'text'   => $node->filter('.text')->text(),
            'author' => $node->filter('.author')->text(),
        ];
    });

    echo "Found " . count($quotes) . " quotes\n";
} finally {
    $client->quit();  // skip this and Chrome processes pile up
}

The waitFor() method blocks until the CSS selector matches at least one element. Without it, you’re racing against JavaScript execution and getting empty results intermittently.

For infinite scroll pages, scroll the viewport and wait for new content:

for ($i = 0; $i < $maxScrolls; $i++) {
    $client->executeScript('window.scrollTo(0, document.body.scrollHeight);');
    usleep(1500000);

    $currentCount = $crawler->filter('.quote')->count();
    if ($currentCount === $previousCount) break;
    $previousCount = $currentCount;
}

Memory Leak Prevention

PHP’s garbage collector handles most cleanup, but long-running scrapers expose edge cases. A naive scraper on 500 pages climbs to ~85 MB. With explicit cleanup, it stays flat at ~4 MB:

for ($page = 1; $page <= 500; $page++) {
    $response = $client->get($url);

    $html = $response->getBody()->getContents();
    $response->getBody()->close();       // release stream immediately

    $crawler = new Crawler($html);
    $titles = $crawler->filter('h3 a')->each(fn($n) => $n->attr('title'));

    $crawler->clear();                   // detach DOM nodes
    unset($html);                        // free the string

    if ($page % 50 === 0) {
        gc_collect_cycles();             // force cycle collection
    }
}

The key: $crawler->clear() detaches DOM nodes with circular references that PHP’s reference-counting GC can’t free on its own. Apply the same pattern inside Pool’s fulfilled callback.

Full code with streaming responses and more patterns is in the complete PHP web scraping guide.