ARgocd — Automatic image updates using helm

Introduction:

In my previous blogs, I explained how ARgocd app of apps works and how we can deploy the application in kubernetes without using command line and pipelines. In this blog I will explain how we can use the latest images automatically in our application manifests whenever there is an image update in the Azure Container Registry

What we are trying to achieve?

Lets assume we are using Azure Container Registry for our application images. There will be regular updates to the images and new version will be pushed by the application team regularly and as platform engineers, we might need to update the images in the kubernetes manifests.

In below example, if the image v2 is pushed to ACR, we might need to follow the github process to update the image in our kubernetes manifest by raising PR. So this requires some efforts to follow git process to update the image everyime there is a push to ACR. Instead the argocd offers an option to automatically udpate the image in our manifest and raise a PR whenever there is a new image in ACR. The argocd keeps listening to the Azure Container Registry and when there is update, it creates a branch and updates the image and optionally raises a PR as well.

Argocd Image updater step by step

Lets see how this image updater works step by step.

Firstly, we need to install the image updater. There are different ways to install image updater. We can directly apply the official yaml manifest from the official site by running the command kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj-labs/argocd-image-updater/stable/config/install.yaml.

But lets follow the ideal approach. Lets place the required manifests in our repository and let argocd detect the manifests and install the image updater.

We already have a repository that is holding the argocd application and the real application manifests (refer to my previous article “app of apps).

Now lets introduce a folder to place the argocd image updater related files. We are going to use the official helm repository of argocd and use the chart for argocd image updater.

Basically, this helm chart is going to install the deployment/pods required for the image updater functionality. Lets pass the input values for this helm . I have refered to the values.yml in the official helm repo and created this values.yaml

# values.yaml
serviceAccount:
  create: true
  name: argocd-image-updater
  annotations:
    azure.workload.identity/client-id: "e10b4cd3-bea5-4e92-bae1-228b03dc54a7"

podLabels:
  azure.workload.identity/use: "true"

# Argo CD Image Updater config
config:
  registries:
    - name: acr
      prefix: acrargotest.azurecr.io
      api_url: https://acrargotest.azurecr.io
      # Uses external auth script for Workload Identity
      credentials: ext:/scripts/acr-login.sh

# Enable auth scripts
authScripts:
  enabled: true
  scripts:
    acr-login.sh: |
      #!/bin/sh

      set -eo pipefail

      AAD_ACCESS_TOKEN=$(cat $AZURE_FEDERATED_TOKEN_FILE)

      ACCESS_TOKEN=$(wget --output-document - --header "Content-Type: application/x-www-form-urlencoded" \
      --post-data="grant_type=client_credentials&client_id=${AZURE_CLIENT_ID}&client_assertion_type=urn:ietf:params:oauth:client-assertion-type:jwt-bearer&scope=https://management.azure.com/.default&client_assertion=${AAD_ACCESS_TOKEN}" \
      https://login.microsoftonline.com/${AZURE_TENANT_ID}/oauth2/v2.0/token \
      | python3 -c "import sys, json; print(json.load(sys.stdin)['access_token'])")

      ACR_REFRESH_TOKEN=$(wget --quiet --header="Content-Type: application/x-www-form-urlencoded" \
      --post-data="grant_type=access_token&service=acrargotest.azurecr.io&access_token=${ACCESS_TOKEN}" \
      --output-document - \
      "https://acrargotest.azurecr.io/oauth2/exchange" |
      python3 -c "import sys, json; print(json.load(sys.stdin)['refresh_token'])")

      echo "00000000-0000-0000-0000-000000000000:$ACR_REFRESH_TOKEN"

# Git commit credentials for write-back
# git:
#   username: "<git-username-or-bot>"
#   password: "<git-personal-access-token>"

# Optional: RBAC for Image Updater
rbac:
  create: true

Understanding the input for image updater.

When we install the image updater using helm chart, in the background it creates deployments and pods for image updater. This image updater pod is the one listening to the Azure Container Registry .So this image updater pod should be able to authenticate to ACR to listen to image updates. So we use workload identity to authenticate to our container registry in Azure. We are going to tell the image updater to use a managed identity to authenticate to ACR. This managed identity will have a federated credential configured for the service account and the kubernetes cluster where the image updater pod is going to get installed.

So the image updater helm chart from official argocd creates the image updater pod, service account etc.

azure.workload.identity/client-id: "e10b4cd3-bea5-4e92-bae1-228b03dc54a7" # this is the managed identity created
 in azure and has acrpull permission
api_url: https://acrargotest.azurecr.io # ACR created in Azure

Configure FEDCred for Managed Identity

Go to managed identity →fed cred

PAss the namespace where the argocd service account is created. The cluster issuer url can be obtained from the AKS security configuration.

The service account is the service account we are creating in the values.yml as part of argocd image updater

Cluster issuer url:

Ensure workload identity and OIDC is enabled in the cluster.

Once we have federated credentials, we will push the changes.

Will the image updater get installed now?

No…

Because my argocd child apps are not listening to the new folder imageupdater. The child app is listening only to the real app folder sap-prod-app (refer to my previous article). So let us create another child application inside the argocd-childapp folder and target the directory “imageupdater”.

After I pushed changes, my new argocd app (imageupdaterapp) got created

Now Image updater argocd app starts listening to the files inside directory imageupdater and syncs back to AKS

When I checked the app, Initially I got below error during sync. This is because I had used a chart version that does not exist. I modified the chart version to 1.2.0 by checking the official argocd helm repo readme file.

Things started syncing and it created lot of things

The important one is the argocd image updater pod which is responsible for checking the applications and updating the image when there is latest image in ACR.

Install Image updater

Install image updater? again? Not really. So far we installed the required pods, service account etc for image updater, but we had not defined which application it has to track for image updates. To do this, we need a kubernetes Custom resource of kind “ImageUpdater” . This is where we define which application to target and which github repository to be updated with the latest image.

So we define a yaml manifest of kind “ImageUpdater”

This image updater tracks the child argocd application which manages the real application.

I am placing the custom resource manifest actualimageupdater.yml inside the imageupdater folder so that it can be applied. Notice that this image updater is going to track the argocd application “childapp1” defined in namepattern. So the image used in real application managed by childapp1 will be monitored for the changes. The image needs to be defined as below.

Let us push changes.

Now the imageupdater manifest should have got installed, but it didnt. The reason is because the files inside my imageupdater are helm based ones which is calling the official helm repo of argocd. And I had introduced a plain yaml, its not going to detect it because its listening for the helm based templates/files. So we need to create a templates folder and put this file inside that so that it follows the helm format. There is already templates folder inside the official helm and our new templates folder gets merged . Also there are multiple ways, we can create create another directory and put this yaml there and have another argocd child app manage this yaml.

Official argo

Hope it makes sense. Initially, We installed the image updater deployment/pods by calling this official argo helm chart which has got the templates defined to create deployment and pods etc for imageupdatercontroller.. Additionally, we created another templates folder defining the manifest of type “Imageupdater” . So both templates get combined and gets installed.

Image Updater Architecture Flow

1. ArgoCD Application (imageupdaterapp —https://github.com/anandambitionv2/appofapps/blob/main/argocd-childapp/imageupdaterapp.yml)

• ArgoCD monitors the folder containing the Helm chart for Argo CD Image Updater.
• When synced, ArgoCD deploys the Image Updater components, including the controller deployment, service account, RBAC resources, and supporting Kubernetes objects.

2. Image Updater Custom Resource

• After the Image Updater controller is running, the ImageUpdater custom resource (actualimageupdater.yml —https://github.com/anandambitionv2/appofapps/blob/main/imageupdater/templates/actualimageupdater.yml) is applied.
• This resource defines which ArgoCD applications should be monitored for container image updates.

3. Monitoring the ArgoCD Child Application

• The ImageUpdater resource references the ArgoCD application childapp1 through the applicationRefs section.
• The Image Updater controller continuously watches the images used by the application managed by childapp1.

4. Tracking the Actual Workload

• childapp1 is itself an ArgoCD Application resource that deploys the real business application.
• The Image Updater follows the deployment manifests managed by childapp1 and monitors the configured container images in the container registry.

5. Image Update Detection

• When a newer image tag is detected in the container registry, the Image Updater controller updates the Git repository according to the configured write-back strategy.
• We submit a PR and merge code back to main branch
• ArgoCD detects the change in main branch and automatically synchronizes the application, deploying the new image version to the cluster.

ArgoCD Application (imageupdaterapp) │ ▼ Deploys Image Updater Controller │ ▼ Applies ImageUpdater Custom Resource │ ▼ Monitors ArgoCD Application (childapp1) │ ▼ Tracks Actual Application Images │ ▼ Detects New Image Tags in ACR │ ▼ Updates Git Repository │ ▼ ArgoCD Syncs Changes │ ▼ New Image Deployed to AKS

Troubleshooting

Now let us check the logs of the image updater

We could see its skipping the app because the normal manifests are not supported, only kustomize and helm are supported. The real application that we used is normal deployment manifests, So lets update our actual real application and introduce kustomize files.

Introduce a kustomization.yml and call the manifests sap.yml and ns.yml and overwrite image in the kustomization

After we pushed and monitored logs for some time, we got new error

The error is authentication to ACR related. We have a managed identity which is configured with federated credentials but we have not given any access to ACR.

Given the ACRPULL role

ACR issue is fixed.

Now again we got new error which is related to github

The error is because our argocd should be able to update images by writing to the github repository . By default the write access is restricted to the repository. For simplicity, am using argocd ui and configure the credentials to connect to github repository

Create a PAT token for the specific github repository in github

Configure the repository from argocd ui as below. Username can be skipped. Only repo url and pat token is sufficient

Now we have connected successfully to repo

After all issue fixed, Now we could see a branch got created and asks for creating a pullrequest. This happened because there is a new image available in ACR.

Its creating a yaml file to overwrite the image in the kustomization. The image I had in kustomization is 2173 and its updating to the latest (2178)available in ACR

Things to note:

In commit, We could see a new yaml file(sap-prod-app/.argocd-source-childapp1.yaml) getting created and overwriting the values from kustomization.

We can also update the kustomization file directly. To do this, we need to add the writebacktarget as kustomization in our custom resource.

Once this is updated, we could see a commit updating the image directly on kustomization

The previous commit can be removed.

Once PR is raised and merged, our main branch is updated with the latest image 2178

Before PR:

RAising a PR:

PR Merged:

Main branch using latest image

Argocd app got synced

Pods updated with latest image

How ARgocd app synced changes?

The argocd app is targeting main branch of the real application, so since we merged the latest image update into main branch it detected changes and synced it

Note:

The image updater authentication to ACR failed after a day, I recreated the image update pod by just deleting it and letting it auto recreate. This fixed the issue. Seems there is token expiry issue.

Conclusion

Image updater is a really useful feature and it avoids manual code update process. Since this involves multiple steps, it might be quite confusing but hope I made it clear.

My github repository: https://github.com/anandambitionv2/appofapps