Ask any security team whether they rotate their secrets on a schedule and you'll get the same answer I gave for years: yes, in principle, and no, not really. The database passwords, the API keys, the service account credentials, the signing keys — everyone agrees they should be rotated. Nobody disagrees. And then it slips, quarter after quarter, until you have credentials in production that are older than half the people on the team.
This isn't laziness. Rotating a secret is a two-line runbook: issue a new one, retire the old one. The hard part isn't the rotation. The hard part is everything that has to be true before you rotate, and that part almost never is.
A secret is only safe to rotate if you know every system that reads it. Every service, every cron job, every deploy script, every third-party integration, every config file baked into an image two years ago. Rotate the secret and you've changed it everywhere at once — but the new value only propagates to the consumers you know about. The ones you don't know about keep presenting the old credential until it stops working.
So the real prerequisite for rotation is a complete inventory of consumers. And that inventory does not exist. It's not written down anywhere, it's not derivable from your config management, and the person who wired up half of it left in 2022. You can grep your repos, you can check your secrets manager's access logs, and you'll get most of them. Most is not all, and rotation punishes you specifically for the ones you missed.
Here's what makes this worse than a normal outage. When you rotate a secret and miss a consumer, the failure is usually silent and usually delayed.
There's always the hard-coded credential in some service nobody's touched in three years. It's not in the deploy pipeline. It's not in the secrets manager. It's a string in a config file on a host that just runs, quietly, doing its one job. You rotate, everything looks fine, the dashboards are green. Then a week later — when that service next needs to authenticate, or when its cached connection finally drops — it fails. Now you're tracing a failure back to a change you made seven days ago, under pressure, while the thing is down. That is a miserable afternoon, and I've had it.
The instinct, when a rotation goes sideways, is "just roll it back." Put the old secret back and buy yourself time to find the consumer you missed.
Sometimes you can. Often you can't, and this is the part people don't plan for. Rotation frequently means revoking the old credential — deleting the old access key, invalidating the old signing key, decommissioning the old certificate. Once you've revoked it, the old secret is gone. There is no "undo." You can't paste the old value back in because it no longer authenticates against anything.
So the honest recovery path isn't reverting. It's re-provisioning equivalent access — issuing a new credential and getting the stranded consumer onto it, fast, while it's broken. That's a different operation than a rollback, it takes longer, and if you didn't plan for it, you're improvising it during an outage. Being clear-eyed about which rotations are reversible and which are one-way is part of the job. Pretending they're all reversible is how you get caught.
None of this means rotation is impossible. It means the runbook is the easy part and the missing pieces are the ones that matter.
Rotation becomes routine when three things are true. You actually know who consumes the secret — a real, maintained inventory, not a best guess. You can stage the change — bring consumers onto the new credential before you retire the old one, so nothing depends on cutting over everything in the same instant. And you have a real recovery plan for the one-way rotations — you've decided in advance how you'd re-provision equivalent access, so it's a procedure and not a scramble.
Everyone already knows to rotate. The reason it doesn't happen isn't that people forgot. It's that doing it safely requires knowledge and preparation the organization never built, and doing it unsafely is worse than not doing it at all. That's the gap worth closing.