How AI Agents Actually See the Web

Almost every autonomous web agent faces the same fundamental challenge: it need to convert a web page into something an LLM can read. For sure, the brute-force approach is obvious — just take a screenshot, throw it at a vision model, ask it where to click. And it works sometimes. In the same way that bubble sort works: technically correct, practically painful, and you’ll regret it at scale.

Vision-based observation has real costs:

Every single screenshot can burn 1,000+ tokens just for encoding. Multiply that by every turn in a multi-step workflow. But this is not the worst thing. The main problem — it is not accurate. GPT-5.2 reached 86% accuracy in ScreenSpot Pro, but the devil is in the details. For some elements it reached 100% recognition, for some not. It’s pitiful when an agent is unable to select a date from a calendar for 10 attempts. And, finally, here’s no built-in notion of “what changed” — every observation is a fresh, full-resolution image.

This doesn't mean vision is useless. It has its place , like interacting with desktop applications. But as the primary observation channel for web interaction it's like using OCR to read a database. There's a better representation. And — in one of those beautiful accidents of engineering history — someone already built it.

ARIA: The solution

In 2014, the W3C published WAI-ARIA 1.0. The goal was pragmatic: make the increasingly JavaScript-heavy web usable for screen reader users. The Web 2.0 era had flooded the internet with custom dropdown menus, tab panels, and modal dialogs — all built from <div>` soup, semantically indistinguishable from a parking lot.

ARIA introduced a formal vocabulary: roles (what is this element?), names (what is it called?), and states (what’s happening to it right now?). Just add ARIA- properties

`role="listbox"`, `aria-expanded="true"`, `aria-label="Select your country"`

to your custom widget, and assistive technology knows what it’s looking at.

Nobody involved in the original spec was thinking about LLMs. They were thinking about screen readers. But the abstraction they defined — a tree of typed, named, stateful UI elements — turns out to be exactly the state representation an autonomous agent needs.

The accessibility community solved our observation problem a decade before we knew we had one. I’d buy them all a beer if I could.

The ARIA Snapshot: Your Page, Decompiled

Playwright (Microsoft’s open-source browser automation framework) serializes the accessibility tree into what it calls an ARIA snapshot — a YAML-like text representation optimized for LLM consumption.

Here’s what an account settings page looks like through this lens:

- navigation [ref=e1]:
  - link "Home" [ref=e2]:
    - /url: /
  - link "Settings" [ref=e3] [active]:
    - /url: /settings
- main [ref=e4]:
  - heading "Account Settings" [level=1]
  - group "Profile":
    - textbox "Display name" [ref=e5]: John Doe
    - textbox "Email" [ref=e6]: john@example.com
      - /placeholder: you@example.com
  - group "Preferences":
    - checkbox "Email notifications" [ref=e7] [checked]
    - checkbox "Dark mode" [ref=e8]
  - button "Save Changes" [ref=e9] [cursor=pointer]

No CSS. No layout. No <div class=”sc-bZQynM kRmOoG”>. Just a clean tree of semantics.

Three things make this useful as an LLM observation:

Roles map directly to interaction affordances. A button can be clicked. A textbox can be filled. A checkbox can be toggled. The model doesn’t need to infer interactivity — it’s declared.

Names provide human-readable identifiers computed via the Accessible Name spec — a priority chain: from aria-labelledby down through aria-label, <label>, element content, title, and finally placeholder. This is why <button><svg class=”icon-save”/></button> is invisible to both screen readers and AI agents — there’s literally nothing to compute a name from. Add aria-label=”Save” and the problem disappears.

States ([checked], [expanded], [disabled], [selected], [pressed]) capture dynamic UI conditions. The model knows a checkbox is already checked without needing to visually inspect a tiny square for a tick mark. (If you’ve ever debugged a vision-based agent failing on checkboxes, you know the pain.)

Element References: Stable Addressing That Actually Works

Each interactive element gets a ref — a stable identifier that maps directly to a DOM element. When the model wants to act, it issues:

{
"name": "browser_click",
"arguments": { "element": "Save Changes button", "ref": "e9" }
}

No XPath. No CSS selectors. No “click at coordinates (834, 612) and pray.” The ref system gives you deterministic addressing that survives layout changes, theme switches, and responsive breakpoints. It’s the difference between “tell me your address” and “I’ll just follow you home and hope you go to the same place tomorrow.”

Incremental Diffs: The Token Budget Trick

Here’s where it gets clever.

A complex web app can have thousands of semantic elements. Sending the full snapshot after every agent action is wasteful — and with API pricing per token, “wasteful” has a very specific dollar amount attached to it.

Playwright implements incremental diffs: after the initial full snapshot, subsequent observations contain only what changed.

- <changed> main [ref=e4]:
  - ref=e5 [unchanged]
  - textbox "Email" [ref=e6]: john.doe@newdomain.com
  - ref=e7 [unchanged]
  - checkbox "Dark mode" [ref=e8] [checked]
  - ref=e9 [unchanged]

The [unchanged] markers tell the model: “this subtree is identical to what you already have in context.” In practice, this produces a ~94% reduction in payload size — from ~5KB down to under 300 bytes on pages with 100+ elements.

This is the feature that makes multi-step agent workflows economically viable. Without it, a 20-step form-filling task would burn tokens re-describing the entire page at every turn. With it, the model maintains a mental model and processes only the deltas. Classic engineering: the fastest data to process is the data you don’t send.

Under the Hood: Why This Is Harder Than It Looks

Serializing an accessibility tree sounds straightforward until you actually try it. The Playwright implementation (in packages/injected/src/ariaSnapshot.ts, if you’re the type who reads source code for fun) handles a parade of edge cases:

Slot resolution: <slot> elements project content from light DOM into shadow DOM. The snapshot shows resolved content, not raw placeholders.

Aria-owns: ARIA allows accessibility relationships that contradict DOM hierarchy. A combobox’s dropdown might live across the document but be logically “owned” by the input.

Pseudo-elements: ::before and ::after content is visible to users and screen readers.

Visibility: display: none, visibility: hidden, aria-hidden=”true”, zero-dimension elements — all filtered out. The snapshot represents what a user would actually perceive.

Each of these is a rabbit hole. The aria-owns case alone requires building a virtual tree that doesn’t match the real DOM structure — you’re essentially doing graph surgery on the fly.

The Blind Spots

Let’s be honest about what this representation cannot do:

Visual design — Colors, spacing, typography, icons? Gone. An angry red error message and a friendly green success message look identical in YAML.

Spatial relationships — “The button is below the form” vs. “next to the form”? The tree doesn’t encode layout. (Some agents are experimenting with bounding-box annotations, but it’s not standard yet.)

Canvas / WebGL — Games, charts, complex visualizations — they’re opaque unless the developer provided ARIA fallbacks (spoiler: they usually didn’t).

Implicit visual semantics — “The highlighted row” only works if “highlighted” is expressed as an ARIA state, not just a CSS class.

This is the fundamental trade-off: you get a clean, cheap, structured observation at the cost of visual context. For most web automation tasks (forms, navigation, data entry), it’s overwhelmingly sufficient. For visual verification or spatially complex UIs, you need eyes. We’ll get to that.

Build vs. Buy: Chrome Has an Accessibility Tree. Why Not Use It?

This is the question every engineer asks. Chrome already maintains a full accessibility tree, accessible via the Chrome DevTools Protocol:

await cdpSession.send('Accessibility.getFullAXTree');

You can even poke at it manually: open DevTools → Elements → Accessibility pane, or visit chrome://accessibility. It’s right there. Why would Playwright spend engineering effort reimplementing this in JavaScript?

Five reasons, roughly in order of importance:

Cross-browser portability. CDP is Chromium-only. Firefox has a completely different protocol. WebKit has yet another. Playwright supports all three. The only way to get identical snapshot behavior across engines is to run the same JavaScript code inside the page. It’s the “write once, run everywhere” promise that Java made and JavaScript accidentally delivered.

LLM-optimized output. Chrome’s AX tree is designed for screen readers — it includes ignored nodes, platform-specific IDs, bounding boxes, and color information. Playwright’s snapshots are designed for a context window: clean YAML, only visible/meaningful nodes, human-readable refs. Different consumers, different formats.

Incremental diffs. The diff mechanism described in section 4 requires owning the serialization layer. You can’t efficiently diff an opaque blob returned by a browser API. The custom engine makes the stateful diffing possible.

Fine-grained traversal control. Shadow DOM piercing, slot resolution, aria-owns graph surgery — Playwright controls exactly how these are handled and represented. With a native API, you get whatever the browser decided to expose.

The Agent Landscape: Who’s Seeing What, and How

Not all agents read the web the same way. The industry has converged on three architectures, and the ratio is shifting fast.

Vision-First

Take a screenshot, feed it to a vision model, click at coordinates. This is the approach you use when you don’t have (or don’t want) access to page structure — general desktop automation, native apps, or legacy web apps so badly built that their DOM is more misleading than helpful.

Used by: Anthropic Computer Use (general desktop), early ChatGPT browsing, OpenAdapt, SeeAct

Good for: Universal applicability, visual content understanding, spatial reasoning.

Bad for: Your token budget, your latency requirements, and your sanity when debugging why the agent clicked 3 pixels left of the button.

Semantic-First

Extract the ARIA snapshot / accessibility tree, present it as structured text, let the model reason over semantics and issue actions by reference. This is the approach that dominates for web-specific agents in 2025–2026.

Used by: Claude via MCP + Playwright, Cursor, GitHub Copilot (agent mode), Browser-Use, AgentQL, Stagehand, OpenAI Operator, enterprise platforms (UiPath, Automation Anywhere), Playwright MCP and Playwright agents

Good for: Speed, cost, precision, stability across UI changes, incremental updates.

Bad for: Visual content, canvas/WebGL, sites with dumpster-fire accessibility (which, honestly, is its own punishment).

The MCP Effect

One reason semantic approaches are consolidating so quickly: Model Context Protocol(MCP) is standardizing the interface between AI models and tools. The Playwright MCP server provides ARIA snapshots as the canonical observation format, and every MCP-compatible client (Claude, Cursor, and a growing ecosystem) speaks the same semantic language.

This is creating a virtuous cycle:

• Agents standardize on ARIA snapshots → developers fix their accessibility → snapshots get better → agents get more reliable → more agents adopt the standard.

The accessibility community spent 20 years pushing developers to write semantic HTML. AI economics might accomplish in 2 years what advocacy couldn’t in 20. I’m not sure whether to be inspired or annoyed.

What This Means (The Part Where I Pretend to Have Business Insights)

The accessibility tree is no longer a compliance checkbox. It’s becoming the de facto API for programmatic interaction with the web. Some implications worth noting:

For web developers: Your next most demanding “user” is a robot. A properly labeled <button> isn’t just good accessibility — it’s AI compatibility. An icon-only button with no aria-label is now a bug in two dimensions.

For the industry: The convergence on ARIA snapshots + MCP is creating genuine interoperability. The same page representation works for Claude, Copilot, Browser-Use, and any future agent that plugs into the protocol. That’s rare in tech, and it happened because the underlying standard (ARIA) was already battle-tested.

For testing: ARIA snapshots double as assertions. Instead of brittle CSS selector checks, you can write semantic tests that survive redesigns. “The page should have a checked checkbox labeled ‘Email notifications’” is both a test and a human-readable spec.

The web’s accessibility layer has quietly become its machine-readability layer. Turns out the engineers who spent years arguing for semantic HTML were building the future’s most important API. They just didn’t know it yet.

Curious what your site looks like to an AI? Run Playwright’s snapshot feature and read the YAML. If it doesn’t make sense to you, it doesn’t make sense to an agent either.