One Missing Slash

How a twelve-year-old website generated 38.5% of its own traffic — and why the fix came down to a single SQL query

When I watched lately on my log, a sentence in my thought came on top: “There’s a bot with rotating IPs. Nothing we can do, right?” After that I took a coffee.

By the end of the day, all three parts of that sentence had turned out to be false. There was no bot. The IPs were not rotating. And there was a great deal I could do — because the site was generating the traffic itself.

This is a write-up of that day: a self-inflicted crawler trap on a legacy platform, the measurement that made the fix safe, and the three theories of mine that did not survive contact with the data. The system is a Drupal 7 installation running six domains off one virtual machine, in continuous production since 2014. Details are anonymised; every figure is real.

The log was drowning in pairs

The pattern was unmistakable once you looked at more than one line. Every entry came twice: a page not found warning, immediately followed by a PHP notice from the page-layout module. Thousands of them, all night, on all three live domains.

974446  00:31  page not found  holiday-homes/islands/coastal/apartments/
                               beaches/villas/seaview-cottage.htm
974445  00:31  php notice      Undefined index: pipeline in
                               panels_panel_context_render()
974444  00:31  page not found  holiday-homes/islands/beaches/coastal/
                               islands/finca-with-pool.htm

Those paths look like someone ran the sitemap through a blender. That impression is exactly backwards, and it is the whole key to the case: the paths were not random at all.

Every segment was a real page

Look at the segments individually. holiday-homes, islands, beaches, coastal — each one is a genuine URL slug from the site. They had simply been concatenated. And each request went exactly one segment deeper than the one before it.

That growth pattern has one common cause: a relative link.

Somewhere in the templates sat href="islands.htm" instead of href="/islands.htm". A browser or crawler standing on /holiday-homes.htm resolves that against the current directory and requests /holiday-homes/islands.htm. That URL does not exist, so it 404s.

Here is what turned an ordinary bug into an unbounded one. On this site, a 404 is not a stub page. It renders the full layout — header, navigation, breadcrumb, footer. Including the same relative links. So the error page for a wrong URL hands the crawler a fresh set of even wronger URLs, one level deeper. Every miss produced the next miss.

There is no natural end to this. With 168,087 real slugs recombining freely, the address space is effectively infinite, and it is generated entirely by the site’s own HTML.

Not an attacker. Four polite crawlers.

Before building any defence, it is worth four minutes to read the user agents. All of them identified themselves. All of them obey robots.txt.

Meta AI crawler          ~80%
Google secondary crawler  ~8%
ClaudeBot                 ~7%
Googlebot                 ~2%

No spoofing, no evasion, no botnet. Which immediately changes the shape of the fix: you do not block a crawler that reads instructions. You give it instructions.

The “rotating IPs” turned out to be an artifact of the infrastructure. A reverse proxy terminates TLS in front of the application, so every request in the web server log carries the same private address — the proxy’s. Blocking by IP would have hit either nothing at all or every visitor at once. The real client addresses were never in that log to begin with.

Where to draw the line

The obvious countermeasure is a rule that refuses deeply nested paths. The interesting question is how deep is safe, and that is where this kind of work usually goes wrong. It feels like a judgement call. It is not. It is a query.

Path depth of real URL aliases

  1 segment        71
  2 segments  168,087
  3 segments    2,364
  4 or more         0
                -------
  total       170,526

Zero. Not “probably none”, not “I can’t think of any” — zero out of 170,526. A rule that refuses four or more segments cannot touch a legitimate page, by construction.

A rule at three segments would have destroyed 2,364 of them. And the first draft of that rule, written before anyone ran the query, was set at two. That draft would have taken 170,451 URLs offline: essentially the entire catalogue.

A limit you measured is a different object than a limit you chose. Four was safe because a query returned zero, not because four felt generous.

The client’s own instinct, incidentally, was to go one level more conservative than proposed. That instinct plus one query is the entire distance between a safe rule and an outage.

Why it started four days ago

The trap was old. The traffic was not. Something had changed, and the git log said what.

Four days earlier, an unrelated repair had brought the site’s static page cache back to life. It had been silently broken since a server migration — writing cache files diligently, never serving them, so every anonymous page view triggered a full application bootstrap. Fixing the read path took one line in a config file. The same page went from 590 milliseconds to single-digit milliseconds.

Crawlers scale their request rate to your response time. The site had, in effect, invited them to crawl two orders of magnitude faster, and they accepted within a day.

Every performance win is also a load test of everything latent in the system. Watch the error rate after a speed improvement, not just the latency graph.

The cache fix was correct and should stay. But it is worth internalising that “we made it faster” and “we changed how much of it gets requested” are the same event.

Refuse, then redirect

The containment came in two layers.

First, a 410 Gone for any request with four or more path segments, evaluated by the web server before the application boots. No PHP process, no page rendering, no log entry. The choice of 410 over 404 matters more than it looks: a 404 tells a search engine “not here right now” and earns the URL months of re-checking, while a 410 says “permanently gone” and gets it dropped.

Four categories of request legitimately have four or more segments, and all four were found by asking the access log which deep paths currently answer 200 — not by imagining them:

/sites/...            static assets (CSS, JS, images)
/admin/...            signed-in administration
/map/ajax/...         map viewer endpoint      1,604 hits
/search/ajax/...      region picker endpoint

Guessing that list would have taken the map viewer offline. It appeared in the log 1,604 times in one rotation period and in nobody’s mental model of the system.

Then the client raised a fair objection: a refusal is a dead end for the crawler too. If the last segment of a mangled URL happens to name a real page, why not send it there?

So a second layer went in front of the refusal. A nightly job exports every URL alias into a lookup map keyed by the lastpath segment. A request that matches the depth rule is looked up in that map; a unique hit gets a 301 to the page it was actually after, and only what the map cannot resolve falls through to the 410.

Ambiguous keys are deliberately dropped from the map. Sixty aliases share a final segment with at least one other, and for those the correct target is genuinely unknowable — so they get the refusal rather than a guess.

The trap moved twice before it stopped

Within an hour of the first rule going live, the same behaviour reappeared on paths the rule no longer matched.

round 1   /a/b/c/d/coastal.htm             rule required a .htm suffix
round 2   /a/b/c/villas/reviews            suffix dropped — rule rewritten
round 3   /node/holiday-homes/islands/...  used an exempted framework path

Round three was self-inflicted. /node/ had been added to the exception list to protect framework routes — a reasonable-sounding precaution that turned out to buy nothing, because genuine /node/ paths are at most three segments deep and the deeper ones require a login, which the rule already handled. The exemption was pure attack surface. It came back out.

None of these rounds took more than a single line to fix. All three were caught because someone kept reading the log after declaring victory.

The one change that broke something

Extending the redirect from four segments down to three looked safe, and there was a guard for the obvious failure: a genuine three-segment URL would resolve to itself in the map, so a self-reference check would leave it alone.

It went live and immediately redirected a real page to a different real page.

Three aliases ended in the same final segment. The map builder was supposed to discard ambiguous keys, and it did — in every case anyone had spot-checked. Its deduplication compared adjacent lines in a sorted stream, which meant it depended on sort order in a way that quietly failed for some groupings. Rewritten to count keys in a hash instead, it discarded 60 aliases where it had previously reported 42.

The guard was not wrong. It was defending against the wrong failure. No amount of guarding at the point of use saves you from data that is already wrong upstream.

Rolled back in four minutes; the depth rule stayed at four, where the redirect targets are junk URLs and a collision costs nothing.

Three hypotheses that died on contact with a measurement

All three were mine. All three were plausible. Two of them would have cost real work.

“Compression is disabled for every cached page.” The cache’s rewrite rule sets an environment flag that switches compression off — and reading the config, that is exactly what it appears to do. Measured: 20,591 bytes delivered where the uncompressed page is 133,146. The flag does not survive the internal redirect that the very same rule performs, so compression was working the whole time. The rule is dead code, not a bug.

“There is a web shell on the production server.” A WordPress file-manager path appeared to answer 200 on a Drupal site, which is the classic signature of a dropped backdoor. It was a log-parsing error of my own making: scanners write junk into the referrer and user-agent fields, and I had grepped whole log lines instead of the request field. Every one of those requests was a 404. No such file exists. Parse the request field, not the line.

“Cached pages are being browser-cached for two weeks.” This one survived, and it was worse than I expected. The cache module writes its own header rules into a file per host directory; on one of three domains that file was simply absent, so the docroot’s default two-week expiry applied to pages carrying prices and availability. Live for four days before anyone noticed, and not revocable — headers already sent will keep those pages “fresh” in visitors’ browsers for up to fourteen days.

Writing down the two refutations is not bookkeeping. It is what stops the next engineer — quite possibly me — from re-investigating them in six months.

None of this fixed the bug

The relative links are still in the templates.

Everything above is containment. The trap can no longer be exploited at depth, and what does get through is redirected rather than dropped. But the mechanism that creates the URLs is untouched, which is precisely why it kept finding new shapes to appear in. Each round of whack-a-mole was cheap, and each round was a reminder that we were treating a symptom.

The real repair is queued where it belongs: inside a template rewrite that will open those files anyway. Fixing it separately would have meant touching the same code twice for one outcome — and on a twelve-year-old system with no documentation, every unnecessary edit is its own risk.

Five things worth carrying to the next system

A limit you measured is a different object than a limit you chose. “Four segments” was safe because a query returned zero. The same rule two levels down would have been an outage.

Check who is calling before you build defences. Four minutes of reading user agents turned an assumed attack into four compliant crawlers, and turned the fix from blocking into instructing.

Speed improvements are load tests. Making the site two orders of magnitude faster changed external behaviour within days. The latency graph looked wonderful. The error log told the real story.

Keep reading the log after you declare victory. The trap relocated twice. Both times it was caught within the hour, for the sole reason that nobody closed the terminal.

Write down what you were wrong about. The refuted hypotheses are documented next to the fixes. That is the part that saves someone a day — including when that someone is you.

Martin Grellmann is an independent engineer and architect who untangles systems that have grown for years — SAP landscapes and twelve-year-old web platforms alike. A clear target picture, a step-by-step roadmap, honest trade-offs instead of one more quick enhancement. CV and focus areas: grellmann.app

All figures come from production measurements taken during the work described: a single access log of 710,263 lines, the site’s own URL alias table, and header probes taken directly against the web server. Client details and URL slugs have been changed.