3000+ Clusters Part 4: How not to DDoS your internal registry

Deploying containers in environments with limited internet access or in big scale can be challenging. Talos Linux simplifies this with a local image cache that stores container images on the node itself. By avoiding repeated downloads from external registries, Talos image cache reduces deployment times, conserves bandwidth, and ensures a smoother, more predictable workflow.

In the recent months we have been using time on daily operations and things like upgrading clusters. In that process we identified some critical issues with having Talos placed at the edge, where resources might be limited or uptime is crucial. Besides that, we also figured out that having 3000 clusters pulling from the same registry at once may be too much and will result in a DDoS of the registry.

Here we can see what happens with the availability of the registry, when 3000 clusters pulls at the same time.

That made us look into if we could eliminate the process of downloading images when deploying a cluster — This is where Talos Image Cache comes in.

How does image caching works

In an earlier article we wrote about how the image creation works with Packer. We now incorporate the cache creation into that process. When creating the Packer image, you will need to prepare the cache to be included in the VM image, as it then works a built-in part of Talos disk layout.

When the VM boots first time and starts in maintenance mode it will not be aware of the cache. This is because it has to be enabled in the machineconfig. To enable the cache you will need to patch the machineconfig with this:

machine:
  features:
    imageCache:
      localEnabled: true

After patching the machineconfig file you must bootstrap the VM with talosctl bootstrap, which will get the config applied. In that process it will create a new registryd service in Talos, that will serve as a local image registry and is tied to the image cache partition.

Creating the cache

For creating the cache you will need dependencies like Talosctl Docker in the environment used for creating the image. To see which images Talos needs you can list them by running the command below.

talosctl images default

ghcr.io/siderolabs/flannel:v0.26.5
registry.k8s.io/coredns/coredns:v1.12.0
gcr.io/etcd-development/etcd:v3.5.19
registry.k8s.io/kube-apiserver:v1.32.3
registry.k8s.io/kube-controller-manager:v1.32.3
registry.k8s.io/kube-scheduler:v1.32.3
registry.k8s.io/kube-proxy:v1.32.3
ghcr.io/siderolabs/kubelet:v1.32.3
ghcr.io/siderolabs/installer:v1.9.5
registry.k8s.io/pause:3.10

Those images need to be parsed to a file, which will be used as a reference list by docker.

talosctl images default > images.txt

If you need additional images like Flux Cilium ExternalDNS you can create a new file e.g. extra-images.txt — which contains those images like the example below. Make sure to include the both tag-based and digit-based reference to each image needed.

ghcr.io/external-secrets/external-secrets:v0.8.5
ghcr.io/external-secrets/external-secrets@sha256:7fa397bec94e05d4de27d8c6b892933b592e64b7a439f1abc985bf3ca26b97b2
ghcr.io/fluxcd/helm-controller:v1.1.0
ghcr.io/fluxcd/helm-controller@sha256:4c75ca6c24ceb1f1bd7e935d9287a93e4f925c512f206763ec5a47de3ef3ff48
ghcr.io/fluxcd/kustomize-controller:v1.4.0
ghcr.io/fluxcd/kustomize-controller@sha256:e3b0cf847e9cdf47b19af0fbcfe22786b80b598e0caeea8b6d2a5f9c26a48a24
ghcr.io/fluxcd/source-controller:v1.4.1
ghcr.io/fluxcd/source-controller@sha256:3c5f0f022f990ffc0daf00e5b199548fc0fa6e7119e972318f0267081a332963
quay.io/cilium/cilium-envoy@sha256:0287b36f70cfbdf54f894160082f4f94d1ee1fb10389f3a95baa6c8e448586ed
quay.io/cilium/cilium@sha256:d55ec38938854133e06739b1af237932b9c4dd4e75e9b7b2ca3acc72540a44bf
quay.io/cilium/operator-generic@sha256:c55a7cbe19fe0b6b28903a085334edb586a3201add9db56d2122c8485f7a51c5

You will then need to add the list into images.txt .

cat extra-images.txt >> images.txt

You can now prepare the cache.

cat images.txt | talosctl images cache-create --image-cache-path ./image-cache.oci --images=-

It is now possible to create the docker image either as an ISO or disk-image — here we will be creating a disk-image. Depending on the amount of images you need, it can be nessesary to extend the disk size for the image. The disk size of the cache image can be changed by adding following option to the command below --image-disk-size=3GB .

mkdir -p _out/
docker run --rm -t -v $PWD/_out:/secureboot:ro -v $PWD/_out:/out -v $PWD/image-cache.oci:/image-cache.oci:ro -v /dev:/dev --privileged ghcr.io/siderolabs/imager:v1.9.5 metal --image-cache /image-cache.oci

There is now a ZST compressed disk-image located in _out/ which contains the whole Talos installation and cached images. It can now be used a the base for the Packer build, which wont be done here.

Activate caching

When the Packer build is done and Proxmox template created. Clone the VM and boot it up. After boot and it goes into maintenance mode, you will need to patch the VM with config for enabling the cache — if you don´t enable it, Talos will automatically remove the image cache partition.

The configuration you need to parse to Talos is as below.

machine:
  features:
    imageCache:
      localEnabled: true

When the config is parsed to the controlplane config and bootstrap is done. We can verify if the cache is working.

First, we can check if the image cache partition is created. Look for the partition with label IMAGECACHE .

talosctl get discoveredvolumes --talosconfig talosconfig

NODE            NAMESPACE   TYPE               ID      VERSION   TYPE        SIZE     DISCOVERED   LABEL        PARTITIONLABEL
192.168.1.79   runtime     DiscoveredVolume   loop0   1         disk        684 kB   squashfs
192.168.1.79   runtime     DiscoveredVolume   loop1   1         disk        77 MB    squashfs
192.168.1.79   runtime     DiscoveredVolume   vda     1         disk        22 GB    gpt
192.168.1.79   runtime     DiscoveredVolume   vda1    2         partition   105 MB   xfs          STATE        STATE
192.168.1.79   runtime     DiscoveredVolume   vda2    1         partition   1.0 MB                             BIOS
192.168.1.79   runtime     DiscoveredVolume   vda3    1         partition   1.0 GB   xfs          BOOT         BOOT
192.168.1.79   runtime     DiscoveredVolume   vda4    1         partition   1.0 MB   talosmeta                 META
192.168.1.79   runtime     DiscoveredVolume   vda5    1         partition   2.8 GB   ext4         IMAGECACHE   IMAGECACHE
192.168.1.79   runtime     DiscoveredVolume   vda6    2         partition   16 GB    xfs          EPHEMERAL    EPHEMERAL
192.168.1.79   runtime     DiscoveredVolume   vdb     1         disk        11 GB

After verifing the parition is created, let´s list the services and see if registryd is running.

talosctl get services --talosconfig talosconfig

NODE           NAMESPACE   TYPE      ID           VERSION   RUNNING   HEALTHY   HEALTH UNKNOWN
192.168.1.79   runtime     Service   apid         2         true      true      false
192.168.1.79   runtime     Service   auditd       2         true      true      false
192.168.1.79   runtime     Service   containerd   2         true      true      false
192.168.1.79   runtime     Service   cri          2         true      true      false
192.168.1.79   runtime     Service   dashboard    1         true      false     true
192.168.1.79   runtime     Service   etcd         2         true      true      false
192.168.1.79   runtime     Service   kubelet      2         true      true      false
192.168.1.79   runtime     Service   machined     2         true      true      false
192.168.1.79   runtime     Service   registryd    2         true      true      false
192.168.1.79   runtime     Service   syslogd      2         true      true      false
192.168.1.79   runtime     Service   trustd       2         true      true      false
192.168.1.79   runtime     Service   udevd        2         true      true      false

So everything looks to be running, but let´s just check that Talos actually is pulling from the local registryd .

talosctl logs -f registryd --talosconfig talosconfig

192.168.1.79: 2025-09-01T11:47:13.826Z INFO image request {"method": "GET", "url": "/v2/kube-proxy/blobs/sha256:a1ae78fd2f9d8fc345928378dc947c7f1e95f01c1a552781827071867a95d09c?ns=registry.k8s.io", "remote_addr": "127.0.0.1:41478", "name": "kube-proxy", "digest": "sha256:a1ae78fd2f9d8fc345928378dc947c7f1e95f01c1a552781827071867a95d09c", "is_blob": true, "registry": "registry.k8s.io"}
192.168.1.79: 2025-09-01T11:47:13.839Z INFO image request {"method": "GET", "url": "/v2/siderolabs/flannel/manifests/sha256:ffbc360bd071f093138a026d23e35b8b759e153badaae351a931a238d3c80987?ns=ghcr.io", "remote_addr": "127.0.0.1:41496", "name": "siderolabs/flannel", "digest": "sha256:ffbc360bd071f093138a026d23e35b8b759e153badaae351a931a238d3c80987", "is_blob": false, "registry": "ghcr.io"}
192.168.1.79: 2025-09-01T11:47:13.858Z INFO image request {"method": "GET", "url": "/v2/kube-proxy/blobs/sha256:4157173e30dd7c3fb793c3d8b3c7c36957097c55541ffede2871e11d3864d2e4?ns=registry.k8s.io", "remote_addr": "127.0.0.1:41478", "name": "kube-proxy", "digest": "sha256:4157173e30dd7c3fb793c3d8b3c7c36957097c55541ffede2871e11d3864d2e4", "is_blob": true, "registry": "registry.k8s.io"}
192.168.1.79: 2025-09-01T11:47:13.867Z INFO image request {"method": "GET", "url": "/v2/kube-proxy/blobs/sha256:b3e9d7d2007338bdeb8a4a6b1bcfaa85be948fe1a259aced775691903c0d8c99?ns=registry.k8s.io", "remote_addr": "127.0.0.1:41512", "name": "kube-proxy", "digest": "sha256:b3e9d7d2007338bdeb8a4a6b1bcfaa85be948fe1a259aced775691903c0d8c99", "is_blob": true, "registry": "registry.k8s.io"}
192.168.1.79: 2025-09-01T11:47:13.879Z INFO image request {"method": "GET", "url": "/v2/siderolabs/flannel/blobs/sha256:aed85bb568e158c63c120be3dc0f2894b3ae56d057c516f21e467778f1f362b3?ns=ghcr.io", "remote_addr": "127.0.0.1:41496", "name": "siderolabs/flannel", "digest": "sha256:aed85bb568e158c63c120be3dc0f2894b3ae56d057c516f21e467778f1f362b3", "is_blob": true, "registry": "ghcr.io"}
192.168.1.79: 2025-09-01T11:47:13.936Z INFO image request {"method": "GET", "url": "/v2/siderolabs/flannel/blobs/sha256:afe4beee6ae0524e783966cf4d6625d85bff2a485f8f770cf6ce4919c4f430fa?ns=ghcr.io", "remote_addr": "127.0.0.1:41496", "name": "siderolabs/flannel", "digest": "sha256:afe4beee6ae0524e783966cf4d6625d85bff2a485f8f770cf6ce4919c4f430fa", "is_blob": true, "registry": "ghcr.io"}
192.168.1.79: 2025-09-01T11:47:13.981Z INFO image request {"method": "GET", "url": "/v2/siderolabs/flannel/blobs/sha256:d62bb7eb03b5936dc5a5665fd5a6ede7eab4a6bd0ed965be8c6c3c21e1e53931?ns=ghcr.io", "remote_addr": "127.0.0.1:41522", "name": "siderolabs/flannel", "digest": "sha256:d62bb7eb03b5936dc5a5665fd5a6ede7eab4a6bd0ed965be8c6c3c21e1e53931", "is_blob": true, "registry": "ghcr.io"}
192.168.1.79: 2025-09-01T11:47:14.000Z INFO image request {"method": "GET", "url": "/v2/siderolabs/flannel/blobs/sha256:bac69ade4f604293095657fde91c3bac6b5563ea2aa7f5f5c33ae1be64de180b?ns=ghcr.io", "remote_addr": "127.0.0.1:41538", "name": "siderolabs/flannel", "digest": "sha256:bac69ade4f604293095657fde91c3bac6b5563ea2aa7f5f5c33ae1be64de180b", "is_blob": true, "registry": "ghcr.io"}
192.168.1.79: 2025-09-01T11:47:14.130Z INFO image request {"method": "GET", "url": "/v2/siderolabs/flannel/blobs/sha256:7105b7cfd4a2b27f2597c6ed14792241915524daef063c4aac782873ba704d8e?ns=ghcr.io", "remote_addr": "127.0.0.1:41496", "name": "siderolabs/flannel", "digest": "sha256:7105b7cfd4a2b27f2597c6ed14792241915524daef063c4aac782873ba704d8e", "is_blob": true, "registry": "ghcr.io"}
192.168.1.79: 2025-09-01T11:47:14.141Z INFO image request {"method": "GET", "url": "/v2/siderolabs/flannel/blobs/sha256:211abe45b3872c3bcbcf72be3fdecf2ece3feebb62f246b02efcb62f6abdd5b2?ns=ghcr.io", "remote_addr": "127.0.0.1:41496", "name": "siderolabs/flannel", "digest": "sha256:211abe45b3872c3bcbcf72be3fdecf2ece3feebb62f246b02efcb62f6abdd5b2", "is_blob": true, "registry": "ghcr.io"}
192.168.1.79: 2025-09-01T11:47:14.169Z INFO image request {"method": "GET", "url": "/v2/siderolabs/flannel/blobs/sha256:9b3b3a025c7289260e7f8aae6fbf35cbf03d3ac1083e797d3d330aeba7dbe483?ns=ghcr.io", "remote_addr": "127.0.0.1:41522", "name": "siderolabs/flannel", "digest": "sha256:9b3b3a025c7289260e7f8aae6fbf35cbf03d3ac1083e797d3d330aeba7dbe483", "is_blob": true, "registry": "ghcr.io"}
192.168.1.79: 2025-09-01T11:47:14.179Z INFO image request {"method": "GET", "url": "/v2/siderolabs/flannel/blobs/sha256:b7418df0b7850e1076ff1c0782755cb3ccbf72787d3a24e2dd3e6b5b827316ca?ns=ghcr.io", "remote_addr": "127.0.0.1:41522", "name": "siderolabs/flannel", "digest": "sha256:b7418df0b7850e1076ff1c0782755cb3ccbf72787d3a24e2dd3e6b5b827316ca", "is_blob": true, "registry": "ghcr.io"}
192.168.1.79: 2025-09-01T11:47:14.201Z INFO image request {"method": "GET", "url": "/v2/siderolabs/flannel/blobs/sha256:af7ae5dd5a38bfc9278937ccc3c7badad2d230db4dcde52ed51b7642cdd58464?ns=ghcr.io", "remote_addr": "127.0.0.1:41522", "name": "siderolabs/flannel", "digest": "sha256:af7ae5dd5a38bfc9278937ccc3c7badad2d230db4dcde52ed51b7642cdd58464", "is_blob": true, "registry": "ghcr.io"}
192.168.1.79: 2025-09-01T11:47:14.266Z INFO image request {"method": "GET", "url": "/v2/siderolabs/flannel/blobs/sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1?ns=ghcr.io", "remote_addr": "127.0.0.1:41522", "name": "siderolabs/flannel", "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1", "is_blob": true, "registry": "ghcr.io"}
192.168.1.79: 2025-09-01T11:47:14.274Z INFO image request {"method": "GET", "url": "/v2/siderolabs/flannel/blobs/sha256:ca3adac18af57418e1221de048ef762eea12685706c0861d0db6b5e7e7c8d077?ns=ghcr.io", "remote_addr": "127.0.0.1:41522", "name": "siderolabs/flannel", "digest": "sha256:ca3adac18af57418e1221de048ef762eea12685706c0861d0db6b5e7e7c8d077", "is_blob": true, "registry": "ghcr.io"}

Here is the important part remote_addr . That should be set to 127.0.0.1 as that is the local registryd service.

Conclusion

So, if you are running Talos in an air-gapped environment, where resources like bandwidth is very limited or you deploy to a large scale, image cache might a worth looking into. It is both easy to configure, optimize deployment time and wont disturb your local registry, as it will not need external sources for pulling images.