PHP cURL for Web Scraping: Sessions, Cookies, and Headers

The difference between a scraper that works on day one and one that still works a month later comes down to how you handle cookies, redirects, and headers. I show each concept in raw cURL and in Guzzle so you can see where the cutoff is for your project.

Basic Page Fetch: cURL vs Guzzle

cURL with minimum browser headers:

$ch = curl_init();

curl_setopt_array($ch, [
    CURLOPT_URL            => 'https://news.ycombinator.com/',
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_FOLLOWLOCATION => true,
    CURLOPT_MAXREDIRS      => 5,
    CURLOPT_TIMEOUT        => 15,
    CURLOPT_HTTPHEADER     => [
        'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36',
        'Accept: text/html,application/xhtml+xml,application/xml;q=0.9',
        'Accept-Language: en-US,en;q=0.9',
    ],
]);

$html = curl_exec($ch);
$statusCode = curl_getinfo($ch, CURLINFO_HTTP_CODE);
curl_close($ch);

Same thing in Guzzle:

$client = new Client([
    'timeout'         => 15,
    'allow_redirects' => ['max' => 5],
    'headers'         => [
        'User-Agent'      => 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36',
        'Accept'          => 'text/html,application/xhtml+xml,application/xml;q=0.9',
        'Accept-Language'  => 'en-US,en;q=0.9',
    ],
]);

$response = $client->get('https://news.ycombinator.com/');

Identical output. The difference shows up when you need cookies, retries, or concurrent requests. Guzzle scales, cURL’s procedural API doesn’t.

Cookie Handling and Session Persistence

Most sites track state through cookies. In cURL, you manage them by pointing to a file:

$cookieFile = tempnam(sys_get_temp_dir(), 'scraper_cookies_');

$ch = curl_init();
curl_setopt_array($ch, [
    CURLOPT_URL            => 'https://quotes.toscrape.com/login',
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_COOKIEJAR      => $cookieFile,
    CURLOPT_COOKIEFILE     => $cookieFile,
]);

$html = curl_exec($ch);
preg_match('/name="csrf_token" value="(.+?)"/', $html, $matches);

curl_setopt_array($ch, [
    CURLOPT_URL        => 'https://quotes.toscrape.com/login',
    CURLOPT_POST       => true,
    CURLOPT_POSTFIELDS => http_build_query([
        'csrf_token' => $matches[1],
        'username'   => 'admin',
        'password'   => 'admin',
    ]),
]);

$html = curl_exec($ch);
curl_close($ch);
unlink($cookieFile);

Guzzle’s CookieJar keeps everything in memory:

$jar = new CookieJar();
$client = new Client(['cookies' => $jar]);

$response = $client->get('https://quotes.toscrape.com/login');
$crawler = new Crawler($response->getBody()->getContents());
$csrfToken = $crawler->filter('input[name="csrf_token"]')->attr('value');

$client->post('https://quotes.toscrape.com/login', [
    'form_params' => [
        'csrf_token' => $csrfToken,
        'username'   => 'admin',
        'password'   => 'admin',
    ],
]);
// Every subsequent request carries the authenticated session

Notice the CSRF extraction uses DomCrawler’s filter() instead of a fragile regex. Use the parser, not string matching.

Headers That Pass Inspection

Bare-minimum headers get you blocked on most serious sites. Anti-bot systems check whether your header combination is consistent with a real browser. A Chrome User-Agent with missing sec-ch-ua headers is an instant red flag.

Headers that match what Chrome 146 actually sends:

$client = new Client([
    'headers' => [
        'User-Agent'         => 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/146.0.0.0 Safari/537.36',
        'Accept'             => 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8',
        'Accept-Language'    => 'en-US,en;q=0.9',
        'Accept-Encoding'    => 'gzip, deflate, br',
        'sec-ch-ua'          => '"Chromium";v="146", "Google Chrome";v="146", "Not-A.Brand";v="24"',
        'sec-ch-ua-mobile'   => '?0',
        'sec-ch-ua-platform' => '"Windows"',
        'Sec-Fetch-Dest'     => 'document',
        'Sec-Fetch-Mode'     => 'navigate',
        'Sec-Fetch-Site'     => 'none',
        'Sec-Fetch-User'     => '?1',
    ],
]);

Your headers should tell a consistent story. If the User-Agent says Chrome on Windows, every other header should match what Chrome on Windows actually sends.

The full guide covers redirect handling, proxy rotation, and anti-bot evasion with complete code: PHP Web Scraping: The Complete Guide.