Mail App #2x02: Deciding How Far To Automate Medium (And Where To Stop)

Using 403 Forbidden As A Signal To Limit Medium To Email Summaries, While Letting Login-Free Sites Go All The Way To Full Article Summaries

Image generated with Midjourney.

In Mail App #2x00, I shared the overall story: I wanted to auto-summarize the articles behind Medium Daily Digest links, hit Cloudflare’s 403 Forbidden, and ended up revising my design. In Mail App #2x01, I went deeper into the technical details, showing how I filtered article URLs with is_medium_article_url, tried requests and a Session with copied cookies, and still got 403 for every Medium article URL.

In Mail App #2x02, I want to use those results to make some explicit design decisions: how far should the mail app try to automate for Medium, at what point should I stop and hand control over to the browser or external services, and how should I treat other newsletters differently.​

What 403 Forbidden Told Me

As I confirmed in Mail App #2x01, Medium article URLs returned 403 Forbidden across the board, both when I used a simple requests.get and when I tried a more advanced pattern with a requests.Session and cookies copied from the browser. The same URLs loaded fine in a logged-in browser, so it is reasonable to assume that Medium (via Cloudflare or another WAF) is distinguishing between real browser traffic and automated HTTP clients and is actively blocking the latter.

In the scraping world, the usual playbook for dealing with 403 Forbidden includes a series of escalations: adjusting User-Agent and headers, adding rate limiting, using proxies and IP rotation, driving a headless browser, tweaking TLS fingerprints, and so on. These steps make sense if your goal is to systematically harvest data from a site. In my case, however, my goal is much more modest: I want a nicer reading workflow for myself, not a full-scale scraping infrastructure.

Setting an Upper Bound for Medium Automation

Because of that, I decided to explicitly define an “upper bound” for what the mail app will and will not do for Medium. For now, my policy is:

What the mail app will do for Medium

• Connect to Yahoo.com via IMAP and extract only Medium Daily Digest emails.
• Convert the email HTML body to plain text with BeautifulSoup and generate a simple summary of the email.
• Extract Medium article URLs from the email and store “that day’s article list” in JSON.

What the mail app will not do for Medium

• Send HTTP requests from Python to Medium article URLs to fetch full article HTML.
• Directly summarize Medium article bodies from within the mail app.

How I actually use it

• When I want to read an article, I open the URL in a browser from the mail app.
• For article-level summaries, I rely on browser-based tools like Comet or Perplexity instead of trying to do it inside the mail app.

In other words, for Medium, the mail app is intentionally “just an email summarizer and link hub.” It stops before the full-article step, on purpose, so I do not need to fight 403 errors or spend time building Cloudflare workarounds.​

Treating Login-Free Newsletters as a Separate Category

At the same time, there are many newsletters that are delivered by email just like Medium Daily Digest, but whose links lead to free, login-free articles. Think of Morning Brew, various AI newsletters, and many technical blogs that send out updates via email.​

For those, I plan to apply a different “second stage” and let the mail app go further:

Second stage for login-free sites

• Extract URLs from the email body in the same way as for Medium.
• Use requests.get to fetch HTML from each URL.
• Extract the main content with BeautifulSoup or similar.
• Run that content through an LLM to generate summaries.
• Save per-email or per-day article summaries into JSON.

For these sites, the risk of hitting 403 Forbidden is generally much lower than with Medium, and a simple HTTP client with reasonable headers is often enough. The second-stage pipeline that failed on Medium can potentially be reused as-is for these newsletters, so the work I did is not wasted; it just needs to be applied to more cooperative targets.

The Idea of Per-Site Policies

Putting all this together, I find it helpful to explicitly think in terms of “per-site policies” for automation. Instead of assuming that every site will support the same level of automation, I define, per site, how far the mail app should try to go.

Medium

• Automatically fetch and summarize emails.
• Extract and list links.
• Do not touch article bodies from Python.

Login-free newsletters and blogs

• Automatically fetch and summarize emails.
• Extract and list links.
• Also fetch article bodies via requests.get and generate automatic summaries.

Everything else (strong login requirements, aggressive WAF, restrictive terms)

• Do not attempt automated full-article fetching by default.
• Consider exceptions only case by case if there is a strong reason.

Defining “how far the automation goes” for each site keeps the mail app’s design simpler and easier to reason about. If I later decide to introduce Selenium or Playwright, I can add a special path like “for Medium only, use real browser automation” without having to rewrite the entire pipeline.

What Mail App #2x02 Fixes

The main purpose of Mail App #2x02 is to turn the 403 Forbidden experience into clear design decisions: what I decided to give up on, what I decided to shift to other sites, and where the boundaries of the mail app are for now.​

In short:

• For Medium, the mail app will handle email summaries and link lists, but it will not try to fetch or summarize article bodies from Python.
• For login-free newsletters and blogs, the second stage (fetching article HTML and summarizing it) is still on the table and will be actively explored.
• Instead of applying one uniform workflow everywhere, I will define per-site automation policies that make these boundaries explicit.

With these policies in place, it becomes much easier to organize the code base: some modules are “Medium first-stage only,” while others are “second-stage enabled for login-free sites.”

In Mail App #2x03, I plan to pick specific newsletters or sites as candidates for this second stage and design concrete workflows for them: from email retrieval to URL extraction, article fetching, summarization, and storage.​