When Google Search Console Could Not Fetch My Sitemap

A debugging story about Cloudflare, Vercel, domain state, and an HTML sitemap fallback

Quick summary: Google Search Console kept showing a vague Could not fetch error for a technically valid XML sitemap. After checking Cloudflare DNS, bot rules, static XML files, Vercel, Workers/OpenNext, and finally a temporary-domain test, the strongest signal pointed to a production-domain or Google-side state issue. The practical fix was not to keep rewriting the XML, but to add a crawlable HTML sitemap fallback.

I recently spent a surprising amount of time debugging a sitemap issue that looked simple at first.

The site had a valid XML sitemap. The endpoint returned 200 OK. The response type was application/xml. The XML validated. robots.txt referenced the sitemap correctly. Browser access worked. Command-line checks worked.

But Google Search Console kept saying:

Could not fetch

There was no useful detail. No clear HTTP error. No XML parsing error. Just a generic failure message.

This is the story of how I tried to isolate the problem, what I ruled out, and why I eventually added an HTML sitemap as a practical SEO fallback.

The site setup

The project is a Next.js tool site for browser-side video and image processing.

It is not a complex SaaS app. There is no login system, no database, no payments, and no server-side user data flow. Most of the site is made up of SEO pages and browser-based tools. The heavier media-processing runtime assets are served through the Cloudflare ecosystem.

The production deployment was on Cloudflare Pages.

The site also had localized routes, so the sitemap needed to expose the main tool pages across multiple languages.

In theory, this should have been a straightforward sitemap setup.

It was not.

The first symptom

In Google Search Console, both the generated XML sitemap and a static XML sitemap showed the same status:

Could not fetch

The discovered page count stayed at zero.

That was confusing because direct checks were normal:

/sitemap.xml -> 200 application/xml
/sitemap-static.xml -> 200 application/xml
/robots.txt -> 200 text/plain

Both XML files passed validation.

The static sitemap mattered a lot. It was just a plain file under public/, not a Next.js metadata route. If both the dynamic sitemap and a static XML file failed in Search Console, the issue was probably not limited to the Next.js sitemap generator.

Checking the obvious things

I started with the standard checks.

The robots.txt file was simple:

User-Agent: *
Allow: /
Sitemap: /sitemap.xml

There was no rule blocking the sitemap or the main pages.

The sitemap response had the correct content type. The XML was valid. The URLs were present. The localized routes were present. The endpoint returned 200.

At this point, there was no obvious evidence that the XML itself was broken.

Going deep into Cloudflare

Because the site was deployed on Cloudflare Pages, I spent a long time checking Cloudflare-specific configuration.

The production domain and the www domain were active in Cloudflare Pages. SSL was enabled. DNS records pointed to the Pages deployment and were proxied through Cloudflare.

I checked the custom domain status, old verification records, DNS records, and basic routing. Nothing looked wrong.

Then I checked the bot and security layer.

Cloudflare AI Crawl Control showed no obvious robots issue. Googlebot was recognized as a search engine crawler. Cloudflare Security Events showed requests to the sitemap path from verified crawlers, including Google-related user agents.

There was also a custom verified-bots rule using:

cf.client.bot

The rule skipped security products for verified bots.

I did not find evidence of:

• WAF block
• Managed Challenge
• JS Challenge
• Interactive Challenge
• Googlebot being denied access

Googlebot-style requests could receive 200 responses.

That made a simple Cloudflare is blocking Googlebot explanation unlikely.

HTTP-level checks

I kept testing the sitemap from different angles.

I checked:

• normal browser requests
• command-line requests
• Googlebot-style user agents
• HTTP/1.1 behavior
• compressed responses
• XML validation with xmllint

The result stayed the same: the sitemap looked technically valid outside of Search Console.

One detail was interesting: the generated sitemap had Next.js route headers, as expected. But the static sitemap also failed in Search Console.

That made the Next.js metadata route less likely to be the sole cause.

Removing build noise

During the investigation, I also found a separate build reliability issue.

The Cloudflare build could fail because next/font/google tried to fetch fonts during build time. This was not the sitemap bug, but it made deployment verification noisy.

I removed the Google Fonts dependency and switched to a system font stack.

After that, both the regular Next build and the Cloudflare build completed successfully.

That gave me a more stable baseline for the sitemap investigation.

The Vercel diagnostic test

Next, I deployed the same project to Vercel as a diagnostic comparison.

The goal was not to move production to Vercel. The goal was narrower:

Is the sitemap XML or project output fundamentally broken?

The deployment itself worked.

But the key detail is this: when the production domain was used, Google Search Console still could not fetch the sitemap.

That was important.

If the same project still failed under the same production domain, then simply changing hosting platforms was probably not enough.

Later, I tested the project with a different temporary domain. Under that different domain, Google Search Console could fetch the sitemap successfully.

That changed the interpretation.

The issue was probably not just Cloudflare vs Vercel.

The stronger signal was that the failure was tied to the production domain itself, or to Google Search Console’s state for that domain.

Here is the simplified control-variable summary:

Cloudflare Pages

Domain used: Production domain

GSC status: Could not fetch

What it suggested: Not explained by basic Cloudflare DNS, SSL, WAF, or robots settings.

Vercel diagnostic deployment

Domain used: Production domain

GSC status: Could not fetch

What it suggested: Moving the same project to Vercel did not fix the production-domain problem.

Same project on a different temporary domain

Domain used: Temporary domain

GSC status: Success

What it suggested: The sitemap output was likely valid; the production domain or GSC state became the stronger suspect.

Cloudflare Workers + OpenNext

Domain used: Production domain

GSC status: Could not fetch

What it suggested: Swapping backend infrastructure did not fix the production-domain problem.

That comparison was the turning point. The pattern no longer looked like a framework bug. It looked much more like a domain-level anomaly.

Testing Cloudflare Workers and OpenNext

Because the old Cloudflare Pages build chain used *@cloudflare/next-on-pages*, and that adapter is deprecated, I also tested Cloudflare Workers with OpenNext.

This was not just a theoretical check. I went through the actual deployment path:

• added Workers/OpenNext configuration
• configured wrangler
• tested a Workers custom domain
• confirmed the Worker was serving traffic
• checked the x-opennext response header
• tested the homepage
• tested robots.txt
• tested sitemap.xml
• tested runtime routes needed by the app

At first, the Worker test domain worked.

Then I switched the production domain from Pages to Workers. That required removing the production custom domains from the Pages project and adding them to the Worker, because Cloudflare would not allow the same hostname to be managed by both at once.

After the switch, the production domain was served through Workers/OpenNext.

The key routes still returned valid responses:

/ -> 200
/sitemap.xml -> 200 application/xml
/robots.txt -> 200 text/plain

The response headers confirmed the traffic was going through OpenNext Workers.

Then I submitted the sitemap again in Google Search Console.

It still failed.

I also tried a cache-busting sitemap URL with a query string. That URL returned valid XML outside of Search Console. GSC still reported Could not fetch.

That result was important because it disproved my earlier suspicion.

This was not simply a Cloudflare Pages or next-on-pages issue.

Even after changing the backend delivery path to Workers/OpenNext, the production domain still had the same GSC failure.

The strongest conclusion

After all of this, the most likely problem area was not the XML file itself.

It was also not clearly one hosting provider.

The strongest clue was the domain test:

• same project
• same kind of sitemap
• different domain
• Search Console could fetch it

That points toward domain-level or Google-side state.

Possible explanations include:

• historical crawl state for the production domain
• Google Search Console state for the domain property
• DNS or routing history associated with the domain
• Google-side host classification or cache

I cannot prove exactly which one it is.

But the evidence pointed away from endlessly rewriting a valid XML sitemap.

The practical problem

Even if an XML sitemap is valid, it is not very useful if Search Console refuses to process it.

The important pages still need to be discoverable.

For a multilingual tools site, that matters.

So I stopped treating the XML sitemap as the only discovery mechanism.

I added an HTML sitemap.

Why an HTML sitemap helps

An HTML sitemap is just a normal page with internal links.

Googlebot can crawl it like any other page.

That gives the site another discovery path:

normal page -> footer link -> HTML sitemap -> localized tool pages

This does not directly fix the XML sitemap failure.

It reduces the risk of relying on only one discovery mechanism.

That was the practical goal.

Designing the HTML sitemap

I kept the page intentionally simple.

The HTML sitemap:

• returns plain HTML
• is linked from the footer
• uses index, follow
• has a canonical URL
• groups links by language
• lists only the core tool pages
• excludes privacy policy and terms pages
• avoids duplicate homepage entries

It is not trying to be a fancy user interface.

It is a reliable crawl hub.

In the current setup, it contains:

9 languages x 11 core tools = 99 tool links

For the homepage converter, the link is represented by each language’s localized tool title, instead of repeating a generic brand link.

That keeps the page focused and avoids unnecessary duplicates.

What I learned

Sitemap debugging is not always about the sitemap file.

Sometimes:

• the XML is valid
• the headers are correct
• the route is public
• bots are not blocked
• multiple hosting paths work technically

and Google Search Console still reports a fetch failure.

At that point, adding another discovery mechanism can be more useful than continuing to tweak a valid XML file.

For this case, the final strategy was:

• keep the XML sitemap
• keep monitoring Search Console
• add an HTML sitemap
• link it from the footer
• make the important localized pages discoverable through ordinary internal links

Final setup

The live site discussed in this post is:

• Main site: https://videosnap.cc/
• HTML sitemap fallback: https://videosnap.cc/html-sitemap

The HTML sitemap is not a replacement for the XML sitemap.

It is a crawl-discovery fallback.

And in this case, that was the most practical solution.