The Night Docker Hub Changed — and Exposed our Containerd Misconfiguration

On an ordinary evening, our pods refused to start. What followed was the discovery of our containerd misconfiguration, which had been working fine for years, until that night.

It’s been a while since I wrote about our incident, and a few months ago, we faced another interesting one, so I am here today to share it with you. Let’s jump right into it.

On August 7th, around 20:00, our on-call engineers got paged because there were pods that failed to start.

Back-off pulling image "prom/blackbox-exporter:v0.23.0": ErrImagePull:
failed to pull and unpack image "docker.io/prom/blackbox-exporter:v0.23.0":
failed to unpack image on snapshotter overlayfs: unexpected media type text/html

At a glance, it clearly indicates something was wrong with the container image pulling.

The first question that came to our mind was “Was there a problem with Dockerhub?” This was discarded easily by trying to pull the same images locally, and it worked just fine

~ docker pull docker.io/prom/blackbox-exporter:v0.23.0                                                                                                                                                                            (⎈ |kind-harbor:default)
v0.23.0: Pulling from prom/blackbox-exporter
22b70bddd3ac: Pull complete 
5c12815fee55: Pull complete 
2369ae062f60: Pull complete 
e563d9e460c8: Pull complete 
Digest: sha256:xxxxxx
Status: Downloaded newer image for prom/blackbox-exporter:v0.23.0
docker.io/prom/blackbox-exporter:v0.23.0

Across multiple clusters, the number of pods in ImagePullBackOff rose steadily.

Number of pods in pending (with ImagePullBackOff) grows gradually

Only images from docker.io seemed cursed—Artifactory-hosted, our internal registry images were fine. Trying to pull the imaged from docker.iodirectly inside the node also showed the issue.

Failed to pull image "alpine:latest": rpc error: code = NotFound desc = failed to pull and unpack image "docker.io/library/alpine:latest": failed to unpack image on snapshotter overlayfs: unexpected media type text/html for sha256:xxxx: not found

From here on, we unfolded the investigation, step by step.

1) The obvious places: credentials and endpoints

Hypothesis: Docker Hub auth or endpoint regression

We inject Docker Hub credentials directly into our nodes (with our USERDATA script) to avoid anonymous rate limits. Maybe the password expired? So, we rotated the password and rolled it out.

From a node shell, the credentials worked with the docker CLI… yet containerd still failed. A mismatch between how humans pull images and how the runtime authenticates had entered the chat.

We experimented with the registry endpoint. Changing the containerd hosts.toml for docker.io from:

server = "https://docker.io"

to:

server = "https://registry-1.docker.io"

…suddenly errors reduced. The dashboard began to slope downward. We rolled out this change/patch via a DaemonSet so every node would get the same patch.

Then we noticed something odd in the logs: some pulls were “working” by… not authenticating at all…

You have reached your unauthenticated pull rate limit. https://www.docker.com/increase-rate-limit" mediatype=application/vnd.docker.distribution.manifest.list.v2+json size=529 FATA[0003] failed to copy: httpReadSeeker: failed open: unexpected status from GET request to https://registry-1.docker.io/v2/bitnami/kubectl/manifests/sha256:d4397a782dcc1e9495c9632a3c7eef1c8b081af357261c5dc25c3c80c5e3649c: 400 Bad Request

We spotted an auth header quirk and massaged the config with a tiny-looking change:

Change the Authorization header

Everything seems calm, so we called it a night at that point.

2) The patch that helped — and quietly broke something else

This is where a seemingly unrelated failure surfaced.

By morning, users were reporting pods stuck in CrashLoopBackOff on the wrong CPU architecture. Our internal component **Noe*, which injects a nodeSelector when an image is x86-only or ARM-only, had stopped adding that preference. Noe was built using a TOML v1 parser that expected the key to be lowercase (toml:”authorization”).*. Now, AMD64-only images were happily landing on ARM nodes…and crashing.

After the patch, Noe was crashing hard

3) The rate-limit wall

At exactly the time you’d expect clusters to be most chatty — 7 a.m. — errors flared back up, loudly and steady:

429 Too Many Requests - toomanyrequests: You have reached your unauthenticated pull rate limit.

That’s when our hearts dropped. Our “fix” had incidentally steered pulls into the anonymous path.

The temporary improvement we’d seen overnight was simply because the unauthenticated rates hadn’t been exhausted yet.

By morning traffic, we’d burned the quota.

4) A thorough look into the pipeline

To get deterministic answers, we reproduced the problem in a dev environment behind a man-in-the-middle proxy and watched what containerd actually did when pulling from Docker Hub:

1. HEAD to registry-1.docker.io → 401 Unauthorized, with WWW-Authenticate: Bearer realm=… auth.docker.io … (this is normal).
2. GET token from auth.docker.io.
3. GET image manifest from registry-1.docker.io with the freshly minted Bearer token.

All standard — except our configuration also stapled a Basic header to requests going to registry-1.docker.io. So the registry received two Authorization headers: one Bearer (correct) and one Basic (ours).

For a long time, Docker Hub had apparently ignored the extra header. On August 7th, it started rejecting such requests—sometimes as 400 Bad Request, sometimes acting as if we were anonymous and rate-limiting us with 429.

5) When the map is not the territory (how containerd matches config)

We double-checked our mental model of containerd’s registry matching:

• Image docker.io/library/nginx:latest selects /etc/containerd/certs.d/docker.io/hosts.toml.
• Inside that file, server = "https://registry-1.docker.io" tells containerd which endpoint to talk to.
• Any [host."https://registry-1.docker.io"].header values are injected into those HTTP requests.

We injected the authorization header with Basic directly into the header configuration

Two consequences fell out:

• If you hard-attach Authorization here, it rides along in addition to the Bearer token containerd gets from auth.docker.io.
• If you rename authorization → Authorization, components expecting a lowercased key (like Noe) may fail to parse credentials.

6) Dead ends and discarded ideas

• Kyverno mutate-all-the-things: At some point, we considered rewriting docker.io → Artifactory at admission time. But Kyverno must inspect the image to make the decision—ironically requiring working registry auth. Bootstrapping failure.

Culprit / Gotcha

In hindsight, the smoking gun was small enough to fit in a single line of TOML: We were attaching a Basic Authorization header at the registry host level, meaning every request to Docker Hub carried both Bearer and Basic. When Docker Hub started enforcing stricter semantics, our requests went from "tolerated" to "rejected" or "treated as anonymous".

The midnight capitalization tweak (authorization → Authorization) then broke Noe’s parser, preventing it from retrieving credentials to inspect image metadata (like architecture). With Noe blind, the nodeSelector never got injected, and x86-only pods landed on ARM nodes.

This wasn’t one incident — it was a combination of two:

1. Registry auth semantics changed → duplicate Authorization headers are no longer tolerated by DockerHub.
2. Tooling assumption → header key case mattered to Noe.

Steps We Took (and why they worked)

1) Stop arguing with Docker Hub, go through a proxy we control

We redirected docker.io pulls at the containerd layer to our Artifactory proxy by overwriting docker.io’s hosts.toml with the content from our internal mirror:

# DaemonSet snippet (conceptual)
  target="/host/etc/containerd/certs.d/docker.io/hosts.toml"
  source="/host/etc/containerd/certs.d/<our-internal-proxy>/hosts.toml"
  cat "$source" > "$target"

From that point on, docker.io/... pulls were authenticated via Artifactory—not directly against Docker Hub—sidestepping the duplicate-header quirk and the anonymous rate limits. Recovery was immediate and visible in the Pending-pod metrics.

Full view of the timeline (number of pods pending)

2) Make Noe resilient

We patched Noe to treat authorization/Authorization case-insensitively when parsing containerd credentials. With that, Noe could once again read credentials, inspect images, and inject architecture selectors correctly.

*https://github.com/adevinta/noe/pull/145*

3) Clean up and make it stick

We baked the redirection and auth approach into node user-data so that every new node comes up with the right registry wiring. Where possible, we moved credentials away from raw headers in hosts.toml and into containerd’s supported auth blocks, avoiding header injection entirely.

What Changed Permanently

• All Docker Hub pulls go through Artifactory for authentication, caching, and control. No more double headers.
• Noe now tolerates header-key case-sensitivity when parsing TOML.
• Fleet bootstrap ensures the correct hosts.toml is in place from node birth.

What We Learned (the hard way)

• A system that “works” might be relying on undefined behavior. The day the other side tightens the rules, you discover the contract you were actually on.
• Tiny details (like header case in a TOML file) are not tiny when another component’s behavior depends on them.
• Fixes that look cosmetic can degrade the observability of the real problem by changing timing (e.g., delaying anonymous-rate-limit failures until morning traffic).
• When you are running out of idea, fall back to the basic and getting deep into how things work always reveal some hidden facts.