The kubectl delete that wiped a 6-year-old Iamport key — and the 37 hours we spent finding the only copy left

The kubectl delete that wiped a 6-year-old Iamport key — and the 37 hours we spent finding the only copy left

How one rollback command exposed six years of unindexed Kubernetes Secrets — and what I changed permanently after 37 hours and 36 minutes of outage.

14:23 KST. Prod EKS. I type kubectl delete secret payment-secret. Two seconds later, payment pods crash-loop.kubectl get secrets -n prod | grep payment shows one thing I expected — and one thing I did not. payment-env has also vanished.

It held our production Iamport API key. It existed nowhere else. Not Terraform. Not External Secrets. Not AWS Secrets Manager. Not GitOps. Someone had kubectl apply -f'd it manually in 2020 and nobody had touched it since.

For context: I work on QR-order infrastructure. “Our merchants” means kiosks at restaurants across Korea; a failed “card transaction” means a guest standing at a kiosk waiting for a card to clear that never will.

For the next 37 hours and 36 minutes, no customer at any of our merchants could complete a card transaction. Fourteen authorizations failed across 13 stores. Direct revenue loss per the canonical postmortem: ₩6,136,390 (~$4,500 USD) — small in dollars, large in trust.

“Manually-applied Kubernetes Secrets are the unrecorded debt of every team’s first three years. They look like infrastructure. They are actually IOUs.”

The migration that exposed it

We were moving from CloudAMQP to Amazon MQ. Same protocol, same broker semantics, just inside our VPC. The dev cutover a week earlier had revealed Amazon MQ’s TLS-only enforcement — port 5672 (plain AMQP) is rejected outright, only 5671 (AMQPS) is accepted.

Six services took a broker-URL swap. One was lucky — a previous developer had left BROKER_USE_SSL = True in config from an unrelated experiment. Two services on Celery 3.x withkombu==3.0.37 needed a one-line code change before they could TLS-connect at all. Those two were payment and message-backend.

The dev migration finished cleanly. April 1, mid-morning, we started Week 1 of a four-week phased prod rollout. By 13:00 KST, three low-traffic services were on Amazon MQ and looking healthy. Then payment happened.

What actually broke

payment came up clean after the broker URL swap.payment-web started returning Iamport authentication failed — unrelated to MQ, but the symptom looked real. I made the call to roll back as a precaution: kubectl delete secret payment-secret -n prod, re-apply from manifest.

The kubectl glob I typed wiped both Secrets that matchedpayment* — payment-secret and payment-env. payment-env was the one nobody had ever migrated to IaC.

Iamport’s account team confirmed key reissue required a 48-hour security review. Not an option. Three engineers spent the rest of the day grepping every private repo for any string that resembled an Iamport API key. Eight hours later, one candidate authenticated against the sandbox. The provenance of that exact string — which old branch, which forgotten config file — became a separate investigation handed to the security team.

37 hours 36 minutes from kubectl delete to the next successful card transaction.

Root cause

The literal root cause was the kubectl delete glob. The actual root cause was deeper.

“When you join a system that has been running for six years, the load-bearing parts of it are almost never the parts that are documented.”

A Kubernetes Secret named payment-env was applied in 2020. It did its job perfectly, every day, for 2,237 days. No architecture review. No runbook. No IaC. Until somebody — me, six weeks into the job, on a Wednesday afternoon — typed a destructive command with a glob.

There is a specific class of failure here. I call it the unindexed dependency:

(1) it exists nowhere in source-controlled state,

(2) the application has no fallback when it disappears,

(3) no monitor alarms on its absence.

Take away any one of the three and the incident is recoverable in minutes. We had all three.

What we changed permanently

“If your application hard-errors on a missing vendor key in prod, you do not have a deployment problem. You have a fallback-design problem.”

The audit surfaced things I did not enjoy reading: 41 Secrets across 12 namespaces with no source of truth other than the cluster itself, 17 of them containing credentials that would have required vendor-side reissue if lost. Five changes shipped over the four weeks after:

1. External Secrets Operator rolled out across prod. Every Secret is now ESO-synced from AWS Secrets Manager, with a daily reconciliation alarm.
2. Application-level fallback defaults for every vendor key. A missing key triggers an alarm but the pod stays up and degrades gracefully — no more KeyError crash loops on a Secret that should exist.
3. Pre-flight Secret snapshot before any cluster-mutating operation: kubectl get secret -n prod -o yaml > /tmp/secrets-$(date +%s).yaml. Trivial. Should have existed years ago.
4. No globs in destructive kubectl calls. Lint rule in runbook templates.
5. Migration runbooks now start with a Secret inventory of the affected namespace. If anything isn’t in IaC, that becomes the wrk for the sprint before the migration begins.

CloudAMQP itself was shut down on schedule. We saved ~$1,200/month on broker costs. The card-payment outage cost us almost four months of that saving in a single afternoon — and that’s only the revenue side; the trust hit with thirteen storefronts is harder to price.

Lessons

“Manually-applied Kubernetes Secrets are the unrecorded debt of every team’s first three years. The interest comes due the next time anyone in the namespace types kubectl delete."

Five takeaways generalizable beyond this incident:

1. Treat any cluster resource you cannot reproduce from IaC as an incident waiting to happen. Inventory it. Score the blast radius if it disappeared. Fix the highest-blast-radius ones first.
2. Application code should never trust environment variables without a defined default and a loud alarm. A missing vendor key in prod should never cause a hard crash.
3. **kubectl delete should never accept a glob without --dry-run=client first.** The cost is one extra command.
4. The dev migration is not the prod migration. Dev caught everything we knew to look for. Prod surfaced what nobody knew was there.
5. The runbook needs a “what would prove this is safe?” checkbox at the top. If the answer is “I’ve proven every dependency exists in IaC,” proceed. If it is “I think it’s fine,” stop.

The 30-second version

kubectl delete secret payment* on prod. Two seconds later, a 6-year-old Iamport API key — managed by literally nothing — is gone. 37h 36m of card-payment outage across 13 storefronts. ₩6,136,390 (~$4,500) of direct revenue lost on 14 failed authorizations. The key was finally recovered after eight hours of grep-through-git-history search inside an unmerged 2020 branch. Every prod Secret is now in External Secrets Operator with daily reconciliation, every vendor API call has a defined default with a loud alarm, and kubectl delete with a glob is a lint-time failure in our runbook templates.

If you’re hiring an SRE and want to ask me about this

• What was the actual fix in the moment vs. the permanent one — and how did you decide which to ship first?
• How do you draw the line between “this should be in IaC” and “this is fine as cluster state”?
• Walk me through the External Secrets Operator rollout — what did you migrate first, and why?
• Has the new kubectl delete lint actually caught anything since you shipped it?