The Agent-Ready Web: A Working Guide to Cloudflare’s New Score

I pointed Cloudflare’s new scanner at my own portfolio website.

Level 0 of 5. Zero of thirteen checks passing.

Not one. No robots.txt rules for AI crawlers. No sitemap. No llms.txt. No MCP server card. No agent skills index. No API catalog. No OAuth discovery. No markdown content negotiation.

Almost all of these were new to me.

This is the story of what fixing it actually looked like, and what the score is really measuring underneath the green checkmarks.

A new audience arrives

The web has had to learn new languages before. It learned to speak to browsers. Then it learned to speak to search engines. Each shift quietly redefined what it meant to ship a website. Mobile-responsive used to be a nice-to-have. Then it was the difference between getting indexed and disappearing.

There’s a third species showing up now. Agents.

Not the chatbot in your sidebar. The autonomous one your customer is letting answer questions on their behalf. The one your reader pointed at “find me a writer who covers AI infrastructure.” The one a buyer’s procurement agent uses to vet your company before a human ever sees your homepage.

That agent doesn’t read your navbar. It doesn’t watch your hero animation. It hits your domain, looks for a few specific files at predictable URLs, and decides in milliseconds whether your site is worth its tokens.

If those files aren’t there, you don’t get cited. You don’t get found. You don’t get an explanation.

Cloudflare just shipped the first real scoreboard for this. It’s called isitagentready.com, and it published on April 17th alongside a deeper case study showing what happens when a site actually optimizes for this audience: 31% fewer tokens consumed, 66% faster answers from the agent. Same questions. Same model. Different site structure.

I ran mine the next day. Zero of thirteen.

The following screenshots show the status after I ran fixes.

The scoreboard

The scanner checks four dimensions.

Discoverability: can an agent find your content at all? robots.txt, sitemap, HTTP Link headers pointing at your metadata. The agent equivalent of "can a search crawler reach you in the first place."

Discoverability section, post-fixes. Shows robots.txt, Sitemap, and Link Header requirements

Content Accessibility: can an agent read what it found without rendering a browser? This is mostly markdown content negotiation: when an agent sends Accept: text/markdown, do you serve markdown back, or do you make it parse a JavaScript-rendered HTML page? The token bill for those two paths is not close.

Content section. Ensures Markdown for Agents

Bot Access Control: what’s your policy toward this traffic? Named rules for which AI companies you allow. A new directive called Content-Signal that lets you separately permit or deny three things: training, real-time inference, search indexing. These are not technical settings. They're business decisions that look like config.

Bot access control section, showing AI bot rules and content signals in robots.txt

Capabilities: can an agent do something on your site, not just read it? Agent Skills index (a menu of available actions), API catalog, OAuth discovery, MCP server card. This is where your site stops being a document and starts being something an agent can operate.

Note: In this case, I didn’t need any of these for my portfolio site. I inserted stubs to get the tests passing, but that’s more of a personal decision.

Agent discoverability section, showing APi and OAuth

Additional agent discoverability: MCP server card, agent skills index, WebMCP tools exposed

A fifth dimension, Commerce, gets checked but not scored: x402, UCP, ACP. Standards for letting agents pay you directly. They're early enough that Cloudflare flagged them as preview, not requirement. My personal portfolio isn’t a commerce site, so I didn’t add any of these.

Commerce section for e-commerce site specific agent discoverability

More commerce specific requirements, like Universal Commerce Protocol and Agentic Commerce Protocol (ACP)

The best part?

Every failing check returns a fix recipe pulled from Cloudflare’s own Agent Skills index. Exact prompts to copy for fixes.

The scanner is itself an MCP server, exposed at /.well-known/mcp.json, so any agent can run it programmatically. The whole tool dogfoods what it measures. That part lands.

What zero of thirteen actually means

Zero is louder than a low score. It means the site, as far as an agent is concerned, doesn’t exist as anything other than HTML to be parsed at maximum cost.

A few of those failures were vanity bruises. A portfolio doesn’t need an MCP server card. A writer’s site doesn’t need an OAuth provider. Some of the score assumes a building I’m not constructing. I added bare-bones implementations just to fulfill the scanner.

But most of them were real.

No Content-Signal meant I had no recorded position on whether my writing should be absorbed into training data. The default was "everything's fair game," which is the worst kind of policy: the one you make by accident.

No llms.txt meant any agent reading me had to guess what was worth fetching from a sitemap, if it found the sitemap, which it couldn't, because there wasn't one. No markdown negotiation meant every agent that did find me paid the full HTML tax.

Zero of thirteen wasn’t a configuration problem. It was a posture problem. I had no posture toward agents at all.

Fixing it

The first scan was a punch. The second one was a lesson.

I didn’t fix everything at once. I built a loop (a /agent-ready command) and ran it against the site repeatedly. Scan, read the gap list, patch, rescan. The iteration mattered because fixes regress other fixes. Add a malformed Content-Signal line, silently invalidate your robots.txt. Add named-bot rules, override your wildcard allow. Ship a sitemap, discover the old one was broken. You need a feedback loop for this, not a checklist.

Wave 1: twenty minutes, eight policy decisions disguised as files.

The mechanical part went fast. What was actually happening underneath was a sequence of decisions I’d never made consciously.

ai-train=no in my Content-Signal directive. I do not want my essays vacuumed into the next training run. That's a position. It might be unenforceable. It's still mine to declare.

ai-input=yes. When an agent is answering someone's question right now and my writing is relevant, I want to be quoted. Different question, different answer.

search=yes. Obviously. Don't break Google to fix something else.

A named-bot allowlist. User-agent: * is the old web's handshake. Naming GPTBot, ClaudeBot, PerplexityBot, Bytespider, Applebot-Extended individually is the new one. Not because the wildcard doesn't work, but because the wildcard is a refusal to take the question seriously. Each AI company is a different counterparty. Treating them as one mass is a policy by default.

An llms.txt and llms-full.txt at the root. The first is a curated reading list pointed at LLMs. The second is the full concatenated content. The difference between handing someone a syllabus and making them grep your bookshelves.

Eleven of thirteen passing after this wave. Twenty minutes of work, nine green boxes I didn’t have an hour earlier.

Wave 2: the standards that don’t fit.

Two checks didn’t yield to the recipe.

oauthDiscovery and oauthProtectedResource expect a real OAuth server. A portfolio site doesn't have one. There were three options: publish stub metadata pointing at GitHub as an external identity provider (technically passes, slightly dishonest), accept 11 of 13 and move on (real but feels like giving up), or stand up Cloudflare Access in front of a /private/ route so I legitimately have a protected resource (real, but a lot of scaffolding for a writer's site).

I took the stub. The honest reading is that this standard was designed for SaaS, and personal sites are in a weird spot the spec didn’t account for.

mcpServerCard is the same shape. The file passes the check. "Truly passing" means a live MCP endpoint with actual tools an agent can invoke. That's a Cloudflare Worker on a subdomain, not a JSON blob in /public. It's on the list. It just isn't this week.

Wave 3: the wins the score doesn’t see.

This is the wave that mattered most and showed up least on the scoreboard.

Markdown content negotiation. When an agent requests Accept: text/markdown, my origin now serves a markdown variant of the same page. Cloudflare measured this single change as a major contributor to the 31%/66% improvement they saw on their own docs. One check on the score. Most of the actual cost reduction lives behind it.

Then the hygiene the scanner doesn’t grade. Stripping site chrome from the markdown variants. Cutting directory-listing pages from llms.txt because they waste tokens on no information. Adding a hidden agent directive at the top of every HTML page telling agents to fetch the markdown version instead. None of these are scored. All of them compound.

Four things the iteration taught me that a checklist wouldn’t have:

1. Standards churn weekly. MCP Server Card is a draft. Content-Signal is a proposal. Web Bot Auth is an IETF draft. A snapshot rots inside a month. A loop doesn’t.
2. Half the checks are policy. The scanner asks technical questions, but the answers are business decisions. ai-train=yesor no is not compliance. It's you choosing whether you want your work to be input or training set.
3. Regressions hide in syntax. A bad Content-Signal line silently invalidated my robots.txt the first time. The rescan caught it. Without the loop I would have shipped broken.
4. The score is a floor, not a ceiling. The hygiene below the scored layer (markdown chrome stripping, llms.txt curation, semantic HTML) is where Cloudflare’s measured 31%/66% actually lives.

What Cloudflare proved

The point was never the score. The point was the throughput.

Cloudflare pointed an agent at other large technical documentation sites, then at their own rebuilt docs. Same agent. Same questions. 31% fewer tokens consumed. 66% faster to the right answer.

The mechanism is named in their post: the grep loop. An agent hits a documentation site. The llms.txt is too big for the agent's context window. The agent grep-searches for keywords. Misses. Refines. Retries. Each iteration burns thinking tokens. Cost climbs. Accuracy drops.

Fix the structure (curated llms.txt, markdown variants, a clean directory the agent can fit in one context window) and the loop collapses to a single linear path. Identify the right page. Fetch it as markdown. Answer.

That’s the prize. Not a gold star on a scoreboard. A site that’s genuinely cheaper and faster for an agent to reason over, which means more agents will choose it as a source, which means more citations, which means everything compounds the right direction instead of the wrong one.

This is the part the audience for this article should care about most: agent readiness isn’t a vanity metric. It’s an operating cost line item, and right now the agents are paying it. Soon they’ll start routing around the sites that make them pay too much.

Final score

The command

This is the loop I built, drop-in:

---
description: Audit + iteratively fix a site for AI agent readiness until 100%. Wraps isitagentready.com's MCP scanner and extends with every adjacent standard. Triggers: agent-ready, ai-optimize, agent audit, llms.txt, isitagentready.
argument-hint: <url> [--commerce] [--extended-only] [--max-iterations N] [--dry-run]
---

# /agent-ready — Comprehensive AI Agent Readiness Loop

**Goal:** Drive a site to 100% agent-readiness against every known AI/agent standard. Benchmark is isitagentready.com's 4 scored dimensions + commerce + extended standards not in their checker.

**Ground truth:** wraps `https://isitagentready.com/.well-known/mcp.json` (tool: `scan_site`) as the authoritative auditor. Extended checks run locally.

**Reference:** `_meta/reference/agent-readiness/criteria.md` — full check inventory with fix recipes.

---

## Inputs

- `$1` — target URL (required). Example: `https://example.com`
- `--commerce` — include x402 / UCP / ACP in scoring (default: on, ask user for preference)
- `--extended-only` — skip isitagentready.com scan, run only local extended checks
- `--max-iterations N` — cap the fix loop (default: 8)
- `--dry-run` — audit only, emit fix plan, do NOT apply

---

## Protocol

### Phase 0 — Preflight

1. Parse `$ARGUMENTS`. If no URL: **STOP, ask**.
2. Verify the target is writable (local working copy / repo). If remote-only, switch to `--dry-run`.
3. Read `_meta/reference/agent-readiness/criteria.md` for the check inventory.
4. Init state file: `_meta/plans/agent-ready-{{timestamp}}.md` — records each iteration's delta.

### Phase 1 — Baseline Audit

Run in **parallel** (independent checks):

**Task A — Scan via isitagentready.com MCP:**

POST https://isitagentready.com/.well-known/mcp.json (Streamable HTTP) Tool: scan_site Args: { "url": "$1" }

If MCP unavailable, fall back to their web form or HTTP-level re-checks per `criteria.md`.

**Task B — Fetch their Agent Skills index (fix recipes):**

GET https://isitagentready.com/.well-known/agent-skills/index.json

Cache locally at `_meta/reference/agent-readiness/skills-cache.json`. For each failing check, the matching skill doc is the fix source of truth.

**Task C — Extended checks not covered by isitagentready.com:**
Run locally against `$1`. Run iterative research subagents to build `additional-criteria.md` § "Extended" — covers afdocs, llms-full.txt, /index.md fallback, hidden agent directives, ai-training redirects, per-directory llms.txt, structured data, caching headers, security.txt, etc.

**Task D — Commerce checks:**
x402 (HTTP 402 + machine-readable payment payload), Universal Commerce Protocol, Agentic Commerce Protocol manifests.

Emit: unified gap list grouped by dimension (Discoverability / Content / Bot Access / Capabilities / Commerce / Extended).

### Phase 2 — Fix Plan

For each gap:
1. Pull the fix recipe (from Agent Skills index for core checks, from `criteria.md` for extended).
2. Determine file(s) to create/modify and exact content.
3. Flag any that require human decision (e.g., Content-Signals policy choice, OAuth flow config).

Write plan to `_meta/plans/agent-ready-{{timestamp}}.md`. If `--dry-run`, **STOP here and report**.

### Phase 3 — Apply Fixes

**Parallelize:** independent files → parallel agents, shared files → serial.

Typical parallel groups:
- Group 1 (well-known endpoints, independent files): `robots.txt`, `sitemap.xml`, `/.well-known/agent-skills/index.json`, `/.well-known/api-catalog`, `/.well-known/mcp/server-card.json`, `/.well-known/http-message-signatures-directory`, `/.well-known/oauth-authorization-server`, `/.well-known/oauth-protected-resource`, `llms.txt`, `llms-full.txt`, `security.txt`.
- Group 2 (server config — often shared): Markdown content negotiation rules, /index.md rewrite rules, hidden agent directive middleware, AI-training redirect rules. Serial.
- Group 3 (page content — shared templates): frontmatter enrichment, directory-listing pruning. Serial.

Each sub-agent **must write files** and return a short summary.

### Phase 4 — Rescan Loop

After applying fixes:
1. Re-run Phase 1 Task A + Task C + Task D.
2. Compare: did any previously-passing checks regress? Block and diagnose if so.
3. Compute score. If `score < 100%` AND `iterations < max-iterations` AND `gaps_closed_this_iteration > 0`: loop to Phase 2.
4. If `gaps_closed_this_iteration == 0` AND `score < 100%`: **stall** — remaining gaps require human decision. Report and stop.
5. If `score == 100%`: celebrate, commit, push.

### Phase 5 — Report

Append to `_meta/plans/agent-ready-{{timestamp}}.md`:
- Final score, dimension breakdown
- Every check: PASS / FAIL / HUMAN-REQUIRED
- Files changed, commits made
- Benchmark delta: token-count + latency comparison vs baseline (borrow the Cloudflare docs methodology — point an agent at the site and measure tokens-to-answer)

Update `_meta/_learnings.md` if new patterns/gotchas surfaced.

---

## Invariants

- NEVER fabricate a fix. If a check's recipe isn't in the Agent Skills index or in `criteria.md`, mark HUMAN-REQUIRED — don't guess.
- NEVER skip the rescan loop. Fixes can regress other checks (e.g., adding `Content-Signal` can break a malformed `robots.txt`).
- NEVER apply Content-Signals, AI-training redirects, or bot-access rules without surfacing the policy choice. These are business decisions, not compliance.
- NEVER write to `.claude/` (vault invariant). Plans → `_meta/plans/`. Docs → `_meta/docs/`. Reference → `_meta/reference/`.
- ALWAYS commit per dimension (atomic commits per `preferences.md`), not one giant "agent-ready" commit.

---

## Exit Conditions

| Condition | Action |
|-----------|--------|
| Score = 100% | Commit remaining, push, close plan. |
| Stall (no progress + HUMAN-REQUIRED gaps) | Report, surface decisions to user, pause. |
| Regression detected | Halt, diagnose, surface before proceeding. |
| Max iterations hit | Report partial score + remaining gap list. |

---

## First-run Bootstrap

If `_meta/reference/agent-readiness/skills-cache.json` does not exist, fetch it before Phase 1. If `criteria.md` is missing or older than 30 days, flag for refresh against the live Cloudflare blog post + isitagentready.com's current skill index.

Read what it plans to do. Make the critical decisions yourself. Repeat.

Where this is going

I got 11 of 13. The two I didn’t are OAuth-shaped problems: standards designed for SaaS infrastructure, applied to a site that doesn’t have users to authenticate or APIs to gate. That’s a real gap in the spec, not a gap in my site

But 11 of 13 from zero of 13 in a few hours of iteration is the actual point. The scoreboard didn’t exist a month ago. Now it does. Most of the web hasn’t run it yet. The window where being agent-ready is a differentiator instead of table stakes is open right now and won’t be open long.

The sites that learn to speak back first get cited first.

Run yours.