I made the same AI compete against itself in SEO tasks

The only difference was the live Google data connection.

The problem with “AI vs SEO tools” comparisons

Most of them change two things at once. They run one model on one data source, a different model on a different source, then declare a winner. When both the model and the inputs change, you learn nothing about either. Was the difference the reasoning, the training data, or the retrieval layer? You can’t tell.

So I changed only one thing.

The method

Both sides were the same model — claude-opus-4-8 (fable5 was my first choice, but…):

• same weights,
• same prompts,
• same niche,
• same market.

The only difference:

Side A (tools) could pull live Google data through NodesHub’s SERP API.

Side B (naked) only has its own memory.

Since both sides are the same model, any gap comes from the data, not from one being smarter.

I ran the naked side first, before pulling any live SERP. The tools side is live Google data, so it’s correct by definition and acts as the answer key.

Scoring was strict: a naked keyword counts as verified only if the exact phrase (or a close variant) actually appears in the live data.

The setup

Nodeshub SEO OS with claude code in terminal

• Niche: SERP data APIs.
• Seed keyword: *"serp data api" .*
• Market: US, English.
• Tasks: keyword research, SERP intent, People Also Ask, query fan-out.
• Verification: live Nodeshub data, plus manual US/incognito Google autocomplete for fan-out queries the API couldn’t confirm.

[Task 1] Keyword research

Naked model: 40 keywords. Tools side: 411 keywords from live data.

The naked model gave me 40 keywords, and the tools side mined 411 from live data. When I graded the 40 against the 411 with strict matching, only 17 of them held up, which works out to 42%.

The misses weren’t nonsense. Keywords like “featured snippet api” or “local pack api” sound completely reasonable, but nobody actually types them. The model also threw in vendor names like zenserp, valueserp and oxylabs as if they were keywords. Those are real companies, but they aren’t real searches.

The live data also caught something I’d have missed. “SERP” is also a finance term, short for Supplemental Executive Retirement Plan, and 24 of the 411 keywords turned out to be about 401ks, pensions and Roth IRAs, with queries like “is serp better than 401k?”. The seed keyword is a homonym that points at a completely different audience, and only the live data showed me that.

[Task 2] SERP intent

I asked both sides to predict the intent, the SERP features and the top domains for three keywords.

On intent, the naked model got all three right, reading them correctly as commercial-investigation queries. On features, it fell apart. It predicted Google Ads on all three SERPs, but real Google showed no ads at all, and instead displayed AI Overviews and a Perspectives Carousel, neither of which the model mentioned.

SERP results without Ads.

The domains had the same problem. The model named 9 real ones, but it also made up 6 of them (zenserp, valueserp, oxylabs, GitHub, G2 and scrapingbee), and it missed the actual leaders completely: nimbleway, trajectdata, scrapingdog, searchapi.io, serper.dev, scrapfly, proxyway and olostep.

It did get one thing right, correctly guessing that Reddit ranks for “best serp api”, where it sits at #2.

[Task 3] People Also Ask

Naked model: 15 questions. Tools side: 8 real ones from live PAA boxes.

The naked model generated 15 questions, while the tools side returned 8 real ones from the live PAA boxes. The exact seed “serp data api” returned no PAA at all, because it’s a sparse term, which is something the naked model had actually flagged itself.

Of the 15 questions, only 3 matched the real 8 in theme, covering the free, legal and what-is angles. The other 12 were made up. And the real PAA had questions the model never came up with, like “What does SERP API stand for?”, “What is a serper API?” and “What is the fastest SERP API?”. So the verification rate here was 3 out of 15, or 20%.

[Task 4] Query fan-out (where the naked model wins)

Naked model: 30 queries. Tools side: 18 queries.

The naked model produced 30 queries, while the tools side produced 18, made up of live AI variants plus confirmed Related Searches. When I checked the 30 through both channels, against live data and manual autocomplete, the naked model scored 24 of 30, or 80%.

This is the one task the model is actually good at, and it’s worth understanding why. Autocomplete tends to follow predictable patterns, with modifiers like pricing, free, python, docs and alternatives, and the model has those down cold. Its 5 misses were the brand-on-brand specifics, things like “zenserp vs serpapi” and “scraperapi serp” that nobody really types.

The raw count hides the real difference, though. Yes, the model produced more queries (30 against 18), but 6 of them were wrong, and the catch is that you can’t tell which 6 without checking them against live Google first. The tools side starts from 100% certainty instead, because every query it returns comes straight from real SERP data, so there’s nothing left to verify afterwards. And that lower count isn’t a hard limit, it’s just a setting: the 18 came from a fixed number of mining loops, and running a few more gives you more confirmed queries on demand, all of them still real.

The scorecard

Full per-task scorecard: https://docs.google.com/spreadsheets/d/13-_sQhvtHVk_qQ2qtUpo8gFZbci1gYTw3DiyH1Lvjxk/edit?usp=sharing

The cost

Tools side: $1.93 in Claude tokens (full session, from /usage) plus $0.255 in Nodeshub tokens (102 × $0.0025) = $2.185.

Naked side: $0.146 in Claude tokens, $0 in data.

Per verified item, the naked model is cheaper: $0.0031 vs $0.0050 but 47 verified outputs sit next to 49 hallucinations, and you can’t tell them apart by reading: both are fluent and confident.

To separate them you have to check every claim against live Google, which is exactly what the tools side does for you. The per-item price only looks cheaper after you’ve already paid for that checking in time. The tools side gives you 440 items you can trust immediately.

What this means for using AI in SEO

A model without live access is fluent and confident, but it can describe a Google that no longer exists.

So match the task to the weakness. For a first draft of query shapes, the model alone is fine. For what really ranks, which features render, or what users really ask, you need live data.

And the only number that counts is verified items (anyone can generate 1000 plausible keywords).

Methodology

Same model claude-opus-4-8 confirmed on both sides; only data access changed.

Tooling. The tools side pulled live data through NodesHub SEO Skills, an open-source Claude Code setup (github.com/Senuto/nodeshub-seo-skills) that calls the NodesHub SERP API straight from the terminal. Every live SERP, PAA and related-search lookup in the test ran through that workflow.

Blinding Naked side ran before any live data was pulled.

Ground truth Tools side is live Google data, used as the answer key.

Keyword scoring Strict matching against the 411 keywords.

Dual verification Fan-out checked against live data, then manual US/incognito autocomplete; queries with no autocomplete presence counted as hallucinations.

Cost Claude figure is the full-session /usage total (input 174, output 46.1k, cache read 2.8m, cache write 109.1k); cache reads dominate, so a re-run costs less. NodesHub at $0.0025/token.

How to connect the SERP API to Claude Code

If you want to reproduce this setup, here’s how to wire NodesHub’s SERP API into Claude Code. Everything runs locally on your machine, with no server in the middle.

You’ll need Claude Code, Python 3.9 or newer, and a NodesHub API key. You get 100 free tokens when you sign up at nodeshub.io, which is enough to run a full session like this one.

1. Clone the skills repo and open the folder in your terminal:

git clone https://github.com/Senuto/nodeshub-seo-skills.git
cd nodeshub-seo-skills

2. Copy your API key from the API Playground on nodeshub.io (the “Copy to clipboard” button).

3. Connect the key. The simplest way is to open Claude Code in this folder and type:

/connect-nodeshub

then paste your key when asked. If you’d rather skip the slash command, you can save it directly with a script, or just export it for the session:

python3 .claude/skills/nod-nodeshub-api/scripts/save_key.py YOUR_KEY
# or
export NODESHUB_API_KEY=YOUR_KEY

The key is stored in .claude/settings.local.json, which is already gitignored.

4. Check that everything is wired up:

python3 .claude/skills/nod-nodeshub-api/scripts/check_setup.py

5. Run a skill. The live SERP analysis I used in this test is a single command inside Claude Code:

/nod-serp-analysis "your keyword"

Claude Code reads the skill instructions, the Python script calls the NodesHub API for live Google data, and you get a formatted report back. The same scripts also run standalone from any terminal, CI pipeline, or other tool (Cursor, Aider, Gemini CLI) if you’d rather not use Claude Code, as long as the API key is set.