Why My Headless Browser Kept Getting Logged Out: Debugging Session Persistence, Cloudflare, and User-Agent Pitfalls

A real debugging story behind a browser-based MCP server — and five lessons that apply to any Playwright automation

TL;DR

A browser-automation MCP server kept returning empty results and asking me to log in on every run. It turned out to be five overlapping bugs, not one:

1. A persistent browser profile silently drops session cookies on restart.
2. Cloudflare’s cf_clearance cookie is bound to the exact User-Agent — mixing a real Chrome UA (login) with a HeadlessChrome UA (background) breaks it.
3. Headless browsers trip bot detection that headed browsers pass.
4. Cookie presence ≠ authenticated. The platform sets uid/sid for anonymous visitors too.
5. The code was scraping the wrong URL.

If you automate any cookie-gated, Cloudflare-protected site with Playwright, these will bite you. Here’s how I isolated each one.

The setup

The Model Context Protocol (MCP) lets AI assistants call external tools. One popular community MCP server automates a publishing platform through a real browser (Playwright) instead of an API. I wanted it to:

• log in once,
• run headless in the background afterward,
• and reliably fetch my published articles.

Instead I got three frustrating symptoms:

• A browser window popped up on almost every call.
• It demanded a fresh login repeatedly, even though the session was “saved.”
• get-my-articles returned an empty list even though I had published posts.

The “login succeeds but the next call acts logged-out” contradiction was the thread I pulled on.

Root cause #1: Persistent profiles drop session cookies

The server used a persistent browser context (a Chrome user-data directory on disk), which should keep you logged in like a normal browser. So why did the session evaporate?

I wrote the smallest possible reproduction: open a persistent context, set two cookies — one with an expiry, one without — close, reopen, and check what survives.

const ctx = await chromium.launchPersistentContext(dir, { headless: true });
await ctx.addCookies([
  { name: 'persistent_c', value: 'P', domain: '.example.com', path: '/',
    expires: Math.floor(Date.now()/1000) + 3600, secure: true, sameSite: 'Lax' },
  { name: 'session_c', value: 'S', domain: '.example.com', path: '/',
    secure: true, sameSite: 'Lax' }, // no expires → session cookie
]);
// → BEFORE close: persistent_c, session_c
await ctx.close();

const ctx2 = await chromium.launchPersistentContext(dir, { headless: true });
// → AFTER reopen: persistent_c   (session_c is GONE)

There it was. A persistent context preserves cookies with an expiry but discards pure session cookies on reopen — and the platform’s auth relied on exactly those session cookies. Every time the server closed and relaunched the browser (e.g., switching from a visible login window to headless background mode), it logged itself out.

The fix: snapshot the full storage state right after login — storageState() captures session cookies that live only in memory — and re-inject it on every launch:

// Right after a successful login, while cookies are live in memory:
const state = await context.storageState();
fs.writeFileSync(sessionFile, JSON.stringify(state));

// On every subsequent (headless) launch:
const saved = JSON.parse(fs.readFileSync(sessionFile, 'utf8'));
await context.addCookies(saved.cookies); // restores session cookies the disk profile dropped

Lesson: A persistent profile alone is not enough for session-cookie-based auth. Pair it with a storageState snapshot you re-seed on launch.

Root cause #2: cf_clearance is bound to your User-Agent

With cookies restored, the background runs hit a wall: a Cloudflare “Just a moment…” interstitial that never cleared. Headless navigation was stuck on bot verification, while the visible login window had sailed through.

Cloudflare issues a cf_clearance cookie once you pass its challenge — but that cookie is tied to the exact User-Agent string that earned it. My login window ran with a normal Chrome/... UA, while the headless runs reported the default HeadlessChrome/... UA. Different UA → cf_clearance rejected → endless re-challenge.

The fix: pin one consistent User-Agent for both the login window and the headless runs (and roughly match your bundled Chromium’s major version):

const USER_AGENT =
  'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 ' +
  '(KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36';

await chromium.launchPersistentContext(dir, { headless, userAgent: USER_AGENT, args });

Lesson: If a clearance/anti-bot cookie “mysteriously” stops working in headless mode, suspect a User-Agent mismatch before anything else.

Root cause #3: Headless trips detection headed passes

Even with a matching UA, you still have to wait the challenge out. Cloudflare’s JS challenge resolves after a few seconds and then navigates. The original code waited a fixed 3 seconds and scraped whatever was on screen — usually the challenge page itself.

async function waitForChallenge(page, timeoutMs = 45000) {
  const start = Date.now();
  while (Date.now() - start < timeoutMs) {
    const title = (await page.title()) || '';
    if (!/just a moment|checking|verify you|security verif/i.test(title)) break;
    await page.waitForTimeout(1500);
  }
  await page.waitForTimeout(1500); // let the SPA hydrate
}

I also kept the standard hardening (--disable-blink-features=AutomationControlled, patching navigator.webdriver) — and, importantly, removed a --disable-web-security flag the old code shipped, which was an unnecessary security downgrade.

Lesson: Don’t scrape on a fixed timer. Poll for a real signal (title changed, content present) that the page is actually ready.

Root cause #4: Cookie presence ≠ authenticated

This one was sneaky. The code decided “logged in” by checking whether sid and uid cookies existed. But the platform sets those for anonymous visitors too. So the code happily proceeded while the page rendered a logged-out view — complete with “Sign in” buttons and an empty content list.

The reliable signal isn’t a cookie; it’s the rendered page:

async function isPageLoggedIn(page) {
  return page.evaluate(() =>
    !document.querySelector('[data-testid="headerSignInButton"]') &&
    !!document.querySelector('[data-testid="headerUserIcon"]')
  );
}

A screenshot made the “mixed” state obvious in seconds — the header showed sign-in buttons and a generic avatar. When in doubt, look at the actual pixels.

Lesson: Verify authentication from the UI state, not from the existence of a cookie that may not mean what you think it means.

Root cause #5: The wrong URL

The smallest bug, found last: get-my-articles navigated to /me/stories/public, which now redirects to the Drafts tab. The published list lives at ?tab=posts-published. One URL change, plus a more resilient extractor that collects story links by pattern instead of brittle selectors, and the list finally came back full.

The architecture that fixed it

Putting it together:

• Persistent Chrome profile + storageState re-seeding → login survives restarts and the headed→headless switch.
• One consistent User-Agent + an explicit Cloudflare wait → headless navigation actually reaches content.
• Page-based auth detection → no more logged-out scraping.
• Headless by default; the visible window appears only for the one-time login, and other tools trigger login on demand.

End result: log in once in a visible window, then everything runs silently in the background — no repeated windows, no re-logins, and the full list of articles returned headless.

A nice bonus: this also solved a limitation the project had documented as unsolved — persisting Google-based logins. Because the persistent profile + storage snapshot captures the whole session, a Google login is reused silently afterward.

[embed]Fix session persistence, Cloudflare challenge, and empty get-my-articles by emregulerr · Pull… Browser-based MCP server for Medium content management - no API tokens required! Built with AI in hours. - Fix session…github.com

The real lesson: debug empirically, not by guessing

I didn’t fix this by reading the code harder. I fixed it by isolating each assumption with the smallest possible experiment:

• A 15-line script proved session cookies don’t survive a profile reopen.
• A UA-matched headed→headless test proved the cf_clearance theory.
• A full-page screenshot proved the page was logged out despite “valid” cookies.
• A DOM dump revealed the correct tab URL and selectors.

Each test turned a hypothesis into a fact. Stacked bugs like these are almost impossible to reason about in your head — but trivial to observe one experiment at a time.

Key takeaways

1. Persistent profile ≠ persistent session. Re-seed storageState for session-cookie auth.
2. Anti-bot clearance cookies are User-Agent-bound. Keep the UA identical across modes.
3. Wait for a real readiness signal, never a fixed timer.
4. Authentication is a UI fact, not a cookie fact.
5. Reproduce before you fix. One assumption, one experiment.

A final, responsible note: browser automation should always respect a site’s Terms of Service and rate limits. Treat any saved session file as a secret — it holds your login cookies — and keep it out of version control.