Every Content Importer I Tried Broke on the Same Thing: Cloudflare

I run a few WordPress sites. Nothing glamorous — content sites, the kind that live or die on how much material you can get published and how consistently you can keep doing it.

About a year ago I needed to migrate roughly 2,800 articles off a dying custom CMS and into WordPress. No export function. No API. No RSS feed worth the name. Just a public archive with pagination that went to page 94.

I did what anyone would do. I searched for a WordPress content importer.

The part where every importer failed

There is no shortage of scraping and importing plugins. I bought three of them. Here is what actually happened.

The first one asked me to be a developer. It opened with a form full of empty text fields: title selector, content selector, image selector, date selector, pagination selector. Fine — I opened DevTools, dug through the DOM, and after about forty minutes I had it working on that one site. Then I pointed it at a second site and had to do the whole thing again. Every new source meant a fresh archaeology session in someone else’s HTML.

That is not automation. That is manual labour with extra steps.

The second one imported empty posts. It ran, it reported success, it created 200 published posts — and every single one of them was blank. It took me an embarrassingly long time to figure out why. The source site sat behind Cloudflare. The plugin was firing a plain wp_remote_get(), receiving a JavaScript challenge page instead of the article, finding no matching selectors in it, and cheerfully saving nothing.

This is not an edge case. A very large slice of the modern web sits behind Cloudflare or something like it. If your importer can’t handle a protected page, it can’t handle a meaningful portion of the internet. Most of them can’t, and most of them don’t tell you — they just fail quietly, which is the worst way for software to fail.

The third one timed out. It worked, technically. It also ran in the browser, which meant PHP hit its execution limit somewhere around post 60 and the job died. Reload, resume, die again. Ninety-four pages of pagination at that rate is a weekend gone.

What I actually wanted

At some point I stopped looking and wrote down what the tool should do. It was a short list:

1. Give it a URL. That’s it. The plugin should read the page and figure out the structure itself. If I have to open DevTools, the tool has failed.
2. Handle protected sites. If I can read the page in Chrome, the importer should get the same content Chrome gets.
3. Finish the job without me. Point it at a category, walk away, come back to a populated site. Page 1 to page 94, on server cron, no browser tab required.
4. Don’t care what the source runs. WordPress, Drupal, Blogger, a hand-rolled PHP site from 2009, a Next.js magazine — irrelevant. It renders in a browser, it imports.

That list turned into a plugin. I called it Scrape Gate.

How it ended up working

Field detection. You paste one article URL. The detection engine reads the rendered page and works out where the title lives, where the article body starts and stops, which image is the featured one, where the date is, and how the category pagination is structured. It strips the furniture — navigation, sidebar, cookie banner, share buttons, related-posts widget — and keeps the article.

This is the part I spent the most time on, and it is a custom engine rather than an off-the-shelf readability library, because real publishing sites have genuinely awful markup and generic extractors give up on them constantly. Every detected field stays editable if you want to override it. On most sites you never open that panel.

The protected-site layer. The request layer negotiates challenge pages properly instead of pretending they’re articles. The practical effect: sites that produced empty posts in my other three plugins import cleanly. This turned out to be the feature people actually buy it for.

Cron-driven category crawls. Give it a category URL, set an interval — every five minutes, every hour, whatever your server is comfortable with — and it walks the pagination until the category is exhausted. It runs through WordPress cron, so nothing depends on your browser being open and nothing gets killed by a PHP timeout. My 2,800-post migration ran overnight.

Find and replace, applied on import. Source branding, phone numbers, leftover shortcodes, tracking parameters, inline styles — the rules run on every post as it comes in, including the ones imported at 4am. No cleanup pass afterwards.

Optional everything else. Featured images: download to your media library, hotlink, or skip. In-content images: same choice. Links inside the content: keep, strip, or convert to external links pointing back at the source — which is what you want if you’re building an aggregator rather than a content library.

The honest part

A tool like this is a loaded gun, and I’d rather say so plainly than pretend otherwise.

Scrape Gate will copy content from any site you point it at. That does not make it legal for you to publish that content. Copyright applies exactly the same way it always did, and “a plugin did it” has never been a defence.

What it’s genuinely good for: migrating your own content between platforms, consolidating sites you own, aggregating sources that permit republication, building research archives, pulling in feeds you’re licensed to use, and moving material out of a CMS that’s about to be switched off. That last one is why it exists.

If your plan is to clone someone else’s magazine and run ads on it, this plugin will technically do it and you will technically deserve the DMCA notice.

Where it is now

It’s a standard WordPress plugin — a .zip you install like any other. WordPress 5.6+, PHP 7.4+, and a working server cron for the long unattended jobs. One-time payment, no subscription, lifetime use, with a year of updates and a year of support that comes directly from me rather than a ticket queue.

I built it because the alternative was three weeks of copy-paste, and because every existing option broke on the same wall. If you’ve hit that wall too, it’s here:

**scrapegate.site**

And if you’ve solved this problem some other way, I’d honestly like to hear it. I spent four months on this and I’m still not convinced there wasn’t a shortcut.