When Container Images Fail to Pull on AKS: The Hidden Zstd + containerd Mismatch

Over the past week, I’ve been testing DHI on Azure Kubernetes Service (AKS). During this work, I ran into a particularly interesting container runtime issue — one that at first looked like a corrupted image but ultimately turned out to be a compression format mismatch between container images and AKS’s container runtime.

This post breaks down what happened, why it happened, and how to fix it.

The Problem: Pods Failing to Start

Several pods suddenly started failing during image pull with the following error:

Failed to pull and unpack image "<image_name>":
number of layers and diffIDs don't match: 3 != 9

At face value, this error suggests an inconsistency in the image manifest — usually a sign of a corrupted or incorrectly built image. But the image worked perfectly fine on my minikube cluster. So, the problem had to be somewhere else.

Inspecting the Image Manifest

Running:

docker manifest inspect <image>@<sha256>

revealed the layer structure:

• 9 total layers
• 6 layers compressed using zstd
• 3 layers compressed using gzip

Example media types:

application/vnd.docker.image.rootfs.diff.tar.zstd
application/vnd.docker.image.rootfs.diff.tar.gzip

Now here’s the interesting part:

The container runtime successfully pulled the 3 gzip layers but failed to pull the 6 zstd layers.

Hence:

Pulled layers = 3
Expected diffIDs = 9
→ number of layers and diffIDs don't match: 3 != 9

So, the image itself was valid — the runtime simply did not support zstd-compressed layers.

Root Cause: containerd Without Zstd Support in AKS

AKS uses containerd as its container runtime.

Support for pulling zstd-compressed image layers was:

• Added natively in containerd v2.0.1
• Backported to containerd v1.7.29

Relevant upstream PRs:

• https://github.com/containerd/containerd/pull/12018
• https://github.com/containerd/containerd/pull/11068

However — and this is the key point — AKS’s containerd v1.7.29 was built and released before the zstd backport was merged.

So, although AKS reports containerd v1.7.29, the binary shipped in AKS does not include zstd support.

Result:

• gzip layers → pulled successfully
• zstd layers → failed to unpack
• manifest mismatch error → pods fail to start

Why This Matters

Modern container build tools (like BuildKit, Docker, Kaniko, etc.) increasingly use zstd compression because:

• Faster decompression
• Smaller layer sizes
• Better image pull performance

If your runtime doesn’t support zstd — your images simply won’t run.

The Solution: Move to Ubuntu 24.04 Node Pools

AKS recently introduced Ubuntu 24.04 (ubuntu2404) OS SKU, which ships with:

• containerd v2.0
• Full zstd layer support

Ubuntu 24.04 node pools are currently in preview and will become the default OS SKU in AKS Kubernetes v1.35 GA.

Azure tracking issues:

• https://github.com/Azure/AKS/issues/4700
• https://github.com/Azure/AKS/issues/4314

Enabling Ubuntu 24.04 Node Pools on AKS

Step 1 — Enable the preview feature

az feature register \
  --namespace Microsoft.ContainerService \
  --name Ubuntu2404Preview

Wait for registration to complete:

az feature show \
  --namespace Microsoft.ContainerService \
  --name Ubuntu2404Preview \
  --query properties.state

Step 2 — Update your node pool OS SKU

az aks nodepool update \
  --resource-group <resource_group> \
  --cluster-name <cluster_name> \
  --name <nodepool_name> \
  --os-sku Ubuntu2404

Once the new nodes are created, your zstd-compressed images will pull and unpack correctly.

Final Takeaway

If your container images use zstd compression, your AKS nodes must run a container runtime that supports it.

Today, that means:

• Ubuntu 22.04 node pools → containerd v1.7.x → no zstd support
• Ubuntu 24.04 node pools → containerd v2.0 → zstd supported

As container image tooling continues moving toward zstd by default, upgrading your runtime environment is no longer optional — it’s essential.