Your Secrets Don’t Belong in Your Deploy Manifests

How externalizing secrets turns rotation, multi-environment sprawl, and the dreaded “redeploy to change a password” into non-events.

Here’s a pattern almost every team rediscovers the hard way. You start with one environment, so the database password lives right there in the deployment file. It works. Then you add a second environment, and a third, and a fourth flavor for each cloud you support. Suddenly the same secret is copy-pasted across a dozen YAML files, some of them in git, and changing it means a code review, a merge, and a redeploy. The thing you most need to change quickly is the thing that’s hardest to change.

The fix isn’t a better templating trick. It’s a change in where secrets live.

Config and secrets are not the same thing

Configuration answers “how should this app behave” — log levels, feature flags, batch sizes. It’s fine for that to sit in version control, reviewed like any other code. Secrets answer “what proves you’re allowed in” — passwords, API keys, tokens. They have completely different lifecycles: config changes when behavior should change, secrets change when something is compromised, rotated, or expires on a schedule you don’t control.

When you store both in the same manifest, you force secrets to inherit config’s lifecycle. Now a routine credential rotation has to go through your code-change process. That’s backwards, and it’s why teams quietly avoid rotating secrets at all — the friction is too high.

Externalize: keep a reference, not the value

The durable pattern is to move the actual secret value into a dedicated secret store — a cloud secret manager or a vault — and leave behind only a reference in your deployment config. Your manifest says “the database password is whatever lives at prod/db/password," not the password itself.

A small controller running in your cluster watches those references and materializes the real values into the runtime as needed. The application sees a normal injected secret and knows nothing about the indirection. Three things fall out of this for free:

• Nothing sensitive is in git. Your manifests become safe to read, diff, and share. A leaked repo leaks pointers, not credentials.
• Rotation stops touching code. Update the value in the store; the controller syncs it. No PR, no redeploy of the manifest to change a password.
• One source of truth per environment. The reference is identical everywhere; only what the store resolves it to differs. The copy-paste sprawl collapses.

Rotation without downtime

Externalizing is what makes graceful rotation possible, but you still have to design for the overlap window. The safe shape is almost always the same: add the new credential alongside the old one, let both be valid for a while, roll your services so everything is using the new one, and only then revoke the old. Two valid keys for an hour beats zero valid keys for a second.

The anti-pattern is the hard swap — delete the old secret, create the new one, and pray every consumer picks it up at the same instant. They won’t. Something is mid-request, something cached the old value, something hasn’t restarted. Design the rotation so that at no single moment is the only valid credential unavailable.

Migrate as an opt-in overlay, not a big bang

Here’s the part teams get wrong even when they love the idea. They try to flip every environment to the new mechanism at once — and the migration itself breaks the simplest deploys, because now a basic setup also needs the secret store, the controller, and the right permissions just to start.

Treat the new mechanism as an opt-in overlay. The default path keeps working exactly as before with inline values. Externalized secrets are a layer you switch on per environment when that environment is ready. Production, with its real rotation needs, opts in early. A throwaway local setup or a quick demo never has to. You get the benefits where they matter without making “hello world” depend on a vault. A migration that breaks the easy case will get reverted before it ever reaches the hard case.

The smell test

If you want one question to gauge where you stand, ask: can someone rotate a production credential without a code review? If the answer is no, your secrets are wearing config’s clothes, and the next urgent rotation is going to hurt more than it should.

Takeaway: store references in your manifests and values in a secret store, rotate by overlap rather than swap, and roll the whole thing out as an opt-in overlay so the easy deploys stay easy.

Checkout my Portfolio to know me more — mathumathiv.com