ECR-creds-refresher

kubernetes operator

Generally, the following are the main methods to authenticate to a private registry in order to be able to pull images from it:

• Configuring the container runtime on each node, i.e. k3s checks if /etc/rancher/k3s/registries.yaml file exists, to retrieve the registry configuration when generating the containerd configuration to authenticate to the private registry.
• Using a kubelet credential provider plugin to dynamically fetch credentials for private registries. (configure the kubelet to invoke a plugin binary to dynamically fetch registry credentials)

In this particular case, the motivation was that I had to authenticate to AWS Elastic Container Registry private repositories, and couldn’t modify the aforementioned cluster configurations. Therefore ecr-creds-refresher operator was the natural workaround.

Pre-requisites

Before obtaining the ECR authentication authorization token ( aws ecr get-login-password --region REGION | docker login --username AWS --password-stdin AWS_ACCOUNT_ID.dkr.ecr.REGION.amazonaws.com) ensure that you have an AWS user/role with the required ECR permissions and valid AWS credentials (such as AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, or temporary credentials when assuming a role).

Demo

When attempting to spin up a pod based on an image from the private ECR repo :

kubectl run test \
--image=255656399702.dkr.ecr.us-east-1.amazonaws.com/os/alpine:latest \
--image-pull-policy=Always -- sleep 5

The pod will fail with ImagePullBackOff

ImagePullBackOff

To fix this, we need:

• A secret that holds the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY
• A mechanism to obtain the authentication token. ⚠️ Important note: An authentication token is used to access any Amazon ECR registry that your IAM principal **has access to and is valid for 12 hours**.

The second point is exactly what the ecr-creds-refresher will do for us, more concretely:

• Upon startup, it will read the AWS credentials from the configured secret (which can be in any namespace).
• Watches for ECRPullSecret custom resources (CRs), on CR Creation/Update/Resume, will fetch the ECR token from AWS, update the secret that holds the token, and patch the default ServiceAccount in the desired namespaces.
• Last but not least, the operator does a periodic refresh of the ECR token and updates all secrets in the desired namespaces.

By desired namespace, I mean the namespaces in which we want to run pods that use images from private ECR repositories (this is easily configurable using the operator CustomResource).

CR

Operator demo and repo here 🔄 : ecr-creds-refresher

[embed]