“Synced” Doesn’t Mean What You Think It Means

The ArgoCD drift that only existed in 2 of my 3 clusters, and what it taught me about admission webhooks.

There’s a kind of bug in GitOps that doesn’t break anything. The pipeline is green. The Helm chart applied cleanly. The workload is healthy. The secret it depends on is synced and in use. And yet ArgoCD insists, against all visible evidence, that the resource is “OutOfSync.”

There’s a more uncomfortable variant. The same bug shows up in two of your three clusters, and the third sits there looking perfectly clean. Same ApplicationSet, same git revision, same chart, same manifests. One environment innocently disagrees with the others, and the disagreement isn’t because dev is special. It’s because dev is old.

This is the story of that bug: how a single field called nullBytePolicy: Ignore silently appeared in my live manifests in stage and prod but not in dev, why the fix was easy once I stopped guessing and read the diff, and why "dev was fine" turned out to be the most misleading part of the entire investigation.

If you run External Secrets Operator (or any operator with a defaulting webhook) on a managed Kubernetes platform with strict admission, you have probably already shipped this exact bug. You just haven’t noticed.

The setup

The platform is a multi-tenant AI chat service deployed to three GKE Autopilot clusters: dev, stage, and prod. Everything is managed via ArgoCD, with an ApplicationSet templating per-environment Applications from a shared spec. Each Application uses ArgoCD’s multi-source feature to combine two repositories:

1. A small repo of resource manifests, including the ExternalSecret that pulls the Datadog API key from GCP Secret Manager.
2. The upstream Datadog Helm chart, with environment-specific values inlined.

The ApplicationSet skeleton looks like this:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: datadog
spec:
  generators:
    - list:
        elements:
          - env: dev
            cluster: https://...
          - env: stage
            cluster: https://...
          - env: prod
            cluster: https://...
  template:
    spec:
      sources:
        # Source 1: ExternalSecret manifest
        - repoURL: <git-repo>
          path: argocd/resources/datadog
          targetRevision: main
        # Source 2: Datadog Helm chart
        - repoURL: https://helm.datadoghq.com
          chart: datadog
          targetRevision: "3.*"
          helm:
            values: |
              # ... datadog config ...
              providers:
                gke:
                  autopilot: true
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
          - ServerSideApply=true
        # ...

The ExternalSecret itself is unremarkable:

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: datadog-api-key
  namespace: datadog
spec:
  refreshInterval: 1h
  secretStoreRef:
    kind: ClusterSecretStore
    name: gcp-secret-store
  data:
    - secretKey: api-key
      remoteRef:
        conversionStrategy: Default
        decodingStrategy: None
        key: my-datadog-api-key
        metadataPolicy: None
  target:
    creationPolicy: Owner
    name: datadog-agent

This pulls the API key from GCP Secret Manager and writes it into a Kubernetes Secret called datadog-agent. The Datadog Helm chart in source 2 then references it via apiKeyExistingSecret: datadog-agent. Order matters: source 1 must materialize the secret before source 2's chart can boot the agent. That part works.

The rollout was intentionally staged. Dev was deployed first, around mid-May. Once we were satisfied it was behaving, stage and prod were uncommented in the ApplicationSet generator and synced on May 27th.

The symptom

After the May 27 rollout, both stage and prod ArgoCD apps flipped to OutOfSync, even though everything else was green:

• App health: Healthy ✅
• Sync status: OutOfSync ⚠️
• The Datadog cluster agent: Running ✅
• The DaemonSet pods: Running ✅
• The datadog-agent secret: created, populated with the API key ✅
• The ExternalSecret itself: Health “Healthy”, status secret synced ✅
• The Datadog UI: receiving metrics from both stage and prod ✅

Dev, deployed two weeks earlier from the same chart and same manifests, was Synced. ✅

So: nothing is broken. The agent is shipping data. The secret is being managed correctly. But ArgoCD is convinced the desired state and live state disagree in two clusters out of three, and the third has no opinion at all.

OutOfSync with selfHeal: true is more than cosmetic. ArgoCD will keep trying to reconcile what it thinks is drift, every cycle, forever. That's wasted reconciles and a permanently noisy dashboard at best. At worst it's a self-heal loop fighting an admission webhook, and you stop being able to tell real drift from phantom drift.

First instinct: pattern-match the fix

Faced with an ExternalSecret OutOfSync on GKE Autopilot, my first instinct was to reach for the usual suspects. ESO has a list of fields its defaulting webhook is known to inject, and GKE Autopilot loves stamping annotations like autopilot.gke.io/resource-adjustment onto pod-bearing resources. So I added the standard ignoreDifferences block I'd seen recommended in similar threads:

ignoreDifferences:
  - group: external-secrets.io
    kind: ExternalSecret
    jqPathExpressions:
      - .spec.refreshPolicy
      - .spec.target.deletionPolicy
      - .spec.target.template
      - .metadata.annotations."autopilot.gke.io/resource-adjustment"

I synced. Nothing changed. Still OutOfSync.

This is the moment I should have started with in the first place: open the diff.

What the diff actually showed

ArgoCD’s Application view has a DIFF tab on every resource. On the datadog-api-key ExternalSecret, the diff was almost entirely identical between desired and live manifests, but on one specific line, the live manifest had a field that the desired manifest didn't:

spec:
  data:
    - remoteRef:
        conversionStrategy: Default
        decodingStrategy: None
        key: my-datadog-api-key
        metadataPolicy: None
        nullBytePolicy: Ignore     # ← LIVE only, not in desired
      secretKey: api-key

That’s the entire drift. One field. nullBytePolicy: Ignore, nested inside each entry of spec.data[].remoteRef. Not in my git manifest, present in the live object, injected by ESO's defaulting webhook. The field controls how the operator handles null bytes in fetched secret values: a fine default, but a default I never set and ArgoCD didn't know about.

None of the fields I’d guessed at (refreshPolicy, target.deletionPolicy, target.template, the Autopilot annotation) were in the actual diff. They might be defaulted in other deployments, but they weren't in mine. Pattern-matching the fix without reading the actual diff was a classic shortcut tax.

The corrected ignoreDifferences was much narrower:

ignoreDifferences:
  - group: external-secrets.io
    kind: ExternalSecret
    jqPathExpressions:
      - .spec.data[].remoteRef.nullBytePolicy

A single jq path expression. The [] matches every element of the data array. That matters because most ExternalSecrets have more than one remoteRef, and you want all of them ignored, not just data[0].

I synced. Stage went green. Prod went green. Done.

Or so I thought.

The actually-interesting bug: why was dev fine?

Stage and prod were now Synced. Dev had been Synced all along. Same ApplicationSet, same git commit, same ignoreDifferences rule applying everywhere. Case closed.

Except the symmetry bothered me. Dev was supposedly identical to stage and prod (same chart version, same generator template, same ESO instance pattern). If the ESO webhook in stage and prod was defaulting nullBytePolicy, why wasn't the one in dev?

My first hypothesis was API version skew. ESO recently promoted its API from v1beta1 to v1, and nullBytePolicy was added around the same time. If dev was running an older ESO chart, perhaps its CRD didn't even know about the field. So I pulled the live manifest from both clusters.

Dev:

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  creationTimestamp: "2026-05-11T16:56:36Z"
  generation: 1
spec:
  data:
    - remoteRef:
        conversionStrategy: Default
        decodingStrategy: None
        key: my-datadog-api-key
        metadataPolicy: None       # ← no nullBytePolicy
      secretKey: api-key
  target:
    creationPolicy: Owner
    deletionPolicy: Retain         # ← present
    name: datadog-agen

Stage:

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  creationTimestamp: "2026-05-27T16:33:03Z"
  generation: 1
spec:
  data:
    - remoteRef:
        conversionStrategy: Default
        decodingStrategy: None
        key: my-datadog-api-key
        metadataPolicy: None
        nullBytePolicy: Ignore     # ← present
      secretKey: api-key
  target:
    creationPolicy: Owner
    deletionPolicy: Retain
    name: datadog-agent

Same apiVersion: external-secrets.io/v1. Both had deletionPolicy: Retain defaulted into target, so the defaulting webhook clearly was running on both clusters. Both had generation: 1, meaning neither object had ever been modified after creation.

The only meaningful difference was the creationTimestamp. Dev: May 11. Stage: May 27. Sixteen days apart.

That’s when the actual root cause clicked into place.

Mutating webhooks don’t reach backwards

ESO injects defaults like nullBytePolicy via a mutating admission webhook. Admission webhooks are not reconciliation loops. They don't continuously scan existing objects looking for fields to default. They fire exactly twice in an object's lifecycle: at CREATE, and at UPDATE. Once an object is in etcd, the webhook never visits it again unless something else triggers an UPDATE.

So the sequence in this cluster fleet was:

May 11. The dev ExternalSecret was admitted. At that time, whatever ESO version was running on dev did not default nullBytePolicy. The object was persisted without the field. generation: 1.

Some time between May 11 and May 27. ESO was upgraded across the fleet. The new version’s defaulting webhook included nullBytePolicy: Ignore in its mutation logic. But the dev ExternalSecret was already in etcd, untouched, with the field absent. The new webhook had no reason to visit it. It stayed as-is.

May 27. The stage and prod ExternalSecrets were admitted for the first time, by the new webhook. The field got stamped in. Drift against the git manifest appeared. ArgoCD flagged OutOfSync.

May 27, a bit later. I added the ignoreDifferences rule. Stage and prod went Synced. Dev was already Synced, but for the wrong reason. It had no field to drift against in the first place. The ignoreDifferences rule was sitting there doing absolutely nothing in dev.

Dev wasn’t “fine.” Dev was stale. It looked clean because it was a snapshot of a world that no longer existed.

The dangerous implication: dev was no longer a valid preview of stage and prod. If anything else had changed in ESO between May 11 and May 27 (a new validation rule, a renamed field, a default that subtly changed semantics), dev would have happily kept reporting Synced while stage and prod broke in ways dev couldn’t preview.

Proving the hypothesis in one command

Hypotheses are nice. Proofs are better. The cleanest way to prove this was to force a re-admission of the dev object without changing anything semantically meaningful, and watch the field appear.

A kubectl annotate with a throwaway key does exactly that. It bumps generation and triggers an UPDATE, which fires the mutating webhook:

kubectl annotate externalsecret -n datadog datadog-api-key \
  trigger-rebake=1 --overwrite

Run against the dev cluster context. Then check whether the field that was previously absent now exists:

kubectl get externalsecret -n datadog datadog-api-key \
  -o jsonpath="{.spec.data[0].remoteRef.nullBytePolicy}{'\n'}"
# → Ignore

Before the annotate, that command returned an empty string in dev. After the annotate, it returned Ignore, exactly the value stage and prod had been showing all along. Hypothesis confirmed.

Two things worth noting. First, generation is the cleanest signal here. Any object still showing generation: 1 after the cluster has been live for a while is a candidate for "this might be a fossil from an earlier admission regime." Second, because the ignoreDifferences rule was already in place, dev's ArgoCD app briefly registered the drift, then immediately reconciled to Synced. If I had run the annotate without the rule in place, dev would have flipped to OutOfSync and stayed there. The rule did become useful in dev, just five minutes after it stopped mattering anywhere else.

What “Synced” actually means

This investigation rearranged my mental model of what ArgoCD’s sync status reports.

I used to read “Synced” as “the live state matches the desired state.” That is what the UI says. But the live state is itself the output of an admission pipeline whose behavior changes over time. A Synced resource on a long-lived dev cluster might just be a fossil of an older admission regime that happens to still match your manifest. Not because the current webhook would produce that result, but because the current webhook has never had a chance to run against it. The same object on a freshly-deployed stage cluster goes through the current webhook and immediately diverges.

The two Synced states look identical in the UI. The fossil one is structurally a time bomb. It becomes OutOfSync the moment anything causes the object to be re-admitted: a rolling restart of ESO, a CRD migration, a manual annotation, an ArgoCD Replace sync. Any of those will fire the new webhook and surface the latent drift.

Four takeaways

1. Read the diff before reaching for ignoreDifferences. I burned the first round of effort on the standard list of ESO defaults: refreshPolicy, target.deletionPolicy, target.template. None of them were actually in the diff. ArgoCD's diff view is exact. Trust it. Pattern-matched fixes from blog posts and threads are guesses about your situation. The diff is ground truth.

2. generation: 1 is a stale-object smell. When you suspect a webhook is responsible for drift between clusters, check metadata.generation. Anything still at 1 long after creation has never been re-admitted, and its contents reflect whatever the admission pipeline looked like on day one. Force a re-admission with a no-op annotation and compare.

3. Dev being “Synced” is not the same as dev being currently valid. A green dev cluster is reassuring. A green dev cluster whose resources predate the last operator upgrade is misleading. If you want dev to actually preview what stage and prod will do, periodically force re-admission of long-lived resources, or treat any generation: 1 resource older than your operator upgrade cycle as suspect.

4. Pin operator versions across the fleet. The dev/stage/prod skew is much worse when each cluster’s ESO drifts independently. Manage the operator itself with ArgoCD, pinned to a specific chart and CRD version, and roll versions forward deliberately. That doesn’t eliminate the fossilization problem, but it makes it predictable. Every environment is fossilized to the same admission regime.

TL;DR

• Symptom: ArgoCD reports stage and prod OutOfSync for an ExternalSecret, while dev (deployed two weeks earlier from the same manifests) is Synced. Nothing is actually broken.
• Actual drift: ESO’s defaulting webhook is injecting spec.data[].remoteRef.nullBytePolicy: Ignore into live objects on admission. The field isn't in the git manifest, so ArgoCD flags drift.
• Fix: Add a narrowly scoped ignoreDifferences entry on the ExternalSecret kind with jqPathExpression: .spec.data[].remoteRef.nullBytePolicy. The [] is mandatory. Using .spec.data[0] only ignores the first element.
• Why dev was “fine”: Dev’s ExternalSecret was created before ESO was upgraded to a version whose webhook defaults nullBytePolicy. Mutating webhooks only fire on CREATE and UPDATE. They never retroactively visit existing objects. Dev's resource was a fossil from before the defaulting rule existed, so no field, no drift, no OutOfSync.
• Proof: Bump dev’s resource generation with kubectl annotate ... trigger-rebake=1 --overwrite. The webhook fires, nullBytePolicy: Ignore appears, and dev now matches stage and prod.
• Real lesson: “Synced” in ArgoCD doesn’t mean “current.” Resources at generation: 1 long after their cluster has lived through operator upgrades are silently lying about what the live admission pipeline would do today.

If you run an ExternalSecrets-style operator on managed Kubernetes (Autopilot, EKS Fargate, AKS automatic, anything with a strict admission posture), I’d be curious whether you’ve seen the same pattern. The fix is one jq path expression. Noticing that your “clean” dev cluster is the most misleading thing in your fleet is the hard part.

Tags: #Kubernetes #ArgoCD #GitOps #ExternalSecrets #GKE #CloudNative #DevOps #SRE #Observability