Promote by Digest, Gate Production, and Operate: GitOps Release Day on Azure

Part 3 of 3. Part 1 — Architecture. Part 2 — Deploy through dev HTTPS.

You have a dev serving traffic over HTTPS and an image digest recorded in Git. Engineering gets exciting when you stop rebuilding for staging and prod. It’s interesting when a green Git commit doesn’t trigger production automatically.

This article covers:

• Promotion ( az acr import + GitOps PRs)
• Why is Argo CD sync in production manual
• The role of Observability and Runbooks
• How to roll back while keeping the GitOps contract intact

Repository: https://github.com/btilki/micser-apps-Azure-Terraform-Helm-GitOps-monitoring

The release contract (recap)

Three steps, three tools:

Steps and Tools

The digest string in Git is the contract. Promotion copies manifest, not tags you might retag later.

Import by digest across ACRs; GitOps PRs are the audit trail

Promote to the stage

Pipelines live under pipelines/promote/. Shared logic is in pipelines/templates/promote-image.yml:

1. RBAC pre-check: It fails if the service principal can’t read the source ACR. It also fails if it can’t push to the target ACR, including the required resource groups
2. **az acr import** — copies the image manifest from dev ACR to stage ACR (or stage → prod in the prod pipeline)
3. GitOps edit + GitHub PR—updates gitops/envs/stage/values-<service>.yaml with the same sha256:... digest.

Register the promote-to-stage.yml in Azure DevOps like this:

• Point to the YAML path
• Run it manually
• Set the parameter: start with the frontend, then other V1 services

Azure DevOps environments: Set up **promote stage** with checks that suit your team. You can add optional approvers and branch filters. The stage is where you rehearse “upper env” behavior without production risk.

After the PR merge, Argo CD syncs stage apps (auto-sync is typical for stage). Verify:

az acr manifest list-metadata --registry <STAGE_ACR> --name frontend -o table 
kubectl get pods -n stage 
curl -sS -o /dev/null -w https://stage.boutique.example.com/

Confirm the digest in stage values match dev; promotion; parity is the point.

Recent Promotions of pipelines

Promote to prod (stricter on purpose)

Prod differs by design:

Controls and Why

Run promote-to-prod.yml with the set. Then, complete the promote-prod approval. Review the GitHub PR. Only prod values should change, and the digest must match what you tested in the stage.

Attach the prod ACR to the cluster if you have not already:

az aks update -g rg-boutique-shared-weu -n aks-boutique-weu --attach-acr acrboutiqueprodweu

Merge the prod values PR—then, in the Argo CD UI or CLI, sync each prod application (frontend-prod, etc.). Prod manifests exclude the automated sync policy by design.

Verify using the release-verification checklist: pods are ready, ingress is healthy, and there are no TLS errors.

Default URLs (replace example.com):

Envs and Storefronts

Identity and prod gates (practical view)

Three RBAC layers often get conflated:

• Azure RBAC — pipeline SP and kubelet identities on ACRs, DNS, and Key Vault
• Kubernetes RBAC—who can kubectl enter which namespace (optional AAD groups)
• Git + Argo — who can merge prod GitOps and who may click Sync

Prod secrets belong in Key Vault via Secrets Store CSI — not in Git. Platform controllers use Workload Identity (Cert-Manager, External-DNS). CI uses **promotion-azure-connection** and a library group secret for the prod ACR.

Longer treatment: SECURITY.md and docs/gitops/prod-branch-protection.md. Maintain a prod known-good digest table in your runbook repo.

Observability on release day

kube-prometheus-stack in monitoring scrapes Ingress and workload metrics. Grafana dashboards (see docs/runbooks/grafana-dashboards.md) help you answer the question, “Is this a bad deployment or a platform issue?”

Alertmanager routes worth configuring before production traffic:

• Ingress 5xx ratio bursts (BoutiqueHighHttp5xxRatiostyle rules)
• Pod crash loops after a new digest
• Certificate expiry — Cert-Manager metrics; DNS-01 failures show up as TLS errors first

Alerts don’t replace verification. They help focus your search when promotions and sync work well. However, users may still face issues.

Rollback without abandoning GitOps

Preferred path: Use Git to revert to the previous **image.digest** in gitops/envs/prod/values-<service>.yaml. Merge through the prod branch protection. This rule needs two reviewers. Then, sync the prod app(s) in Argo CD.

Avoid using kubectl set image unless it’s an emergency. The cluster needs to match Git, or the next sync will cause issues.

Runbook flow ( docs/runbooks/prod-rollback.md):

1. Note the time of the last prod values merge or manual sync
2. Check the current digest against a known-good version. This could come from a table, a past commit, or the last good stage
3. Open rollback PR; merge; sync prod app(s)
4. kubectl rollout status and curl the storefront

If Sync fails often or pods are in CrashLoop with the right digest, escalate to the platform. Check for outages in the node, CNI, or registry. Don’t promote without verification.

Other runbooks in the repo:

Runbooks

Hardening after the first prod release

Phases 8–9 in the repo include:

• Policy bundles
• Trivy in CI
• Budgets
• Dashboard polish
• Optional smoke steps in promoting pipelines

You don’t need everything on day one. But NetworkPolicies, PodSecurityPolicy, and CertificatePolicy should come soon after.

Full loop (what you built)

From a whiteboard perspective, you now have:

• Reproducible Azure Foundation (Terraform layers)
• Build once in dev, import the same digest upward
• GitOps PRs as audit trail; prod protected by review and manual sync
• HTTPS, DNS automation, and metrics for day-two operations

That’s a solid platform engineering demo. It’s not about trendy tools. The digest and Git contract stay aligned across different environments.

Clone the repo. If any step differs from the docs, open an issue. Then extend v1 with upstream Online Boutique services to complete the checkout path.

Repository: https://github.com/btilki/micser-apps-Azure-Terraform-Helm-GitOps-monitoring

Thanks for following the series! Operational reality lives in the runbooks and phase guides. I see Medium as the story and GitHub as the source of truth.