I've heard "we have rollbacks" more times than I can count. For a while I took it at face value. Then I started asking a follow-up question, and the follow-up is where it falls apart: when's the last time you actually did it? Not planned to. Not wrote the runbook for. Actually restored a system to the state it was in before a change and confirmed it came back the way you expected. Most of the time, the honest answer is never. They have a button. They've never pushed it. A rollback you haven't exercised is not a rollback. It's a hope with a UI.
When you press on it, "we have rollbacks" almost always turns out to mean something weaker. Sometimes it means backups — which restore the whole system to a point in time, a blunt instrument that loses everything after the snapshot and assumes the backup survived the same event that took out production. In 2014 an attacker got into Code Spaces' AWS control panel and deleted everything, including the backups. Production and backups lived in the same blast radius. The company shut down. Sometimes "we can roll it back" means re-running a playbook or re-applying a previous config — which moves you toward a state that resembles the one you had, not the exact state you were in. The before-state is gone. You're building something new that resembles the old thing. Sometimes it means CloudTrail, which tells you what happened but doesn't undo it. Forensics and rollback are not the same tool. People reach for one when they needed the other.
The common thread across every incident where rollback failed is that the path existed on paper but had never been exercised against the specific failure it was supposed to cover. An AI coding agent wired into a production environment recently deleted a database and its backups in under ten seconds. The recovery plan assumed the backups would be in a separate enough place. They weren't, and nobody had tested the assumption. That's what untested rollback looks like in practice: you find out whether it works at the exact moment you need it to work, which is the worst possible time to learn something new about your own infrastructure.
The reason organizations end up here is that rollback is usually treated as an emergency procedure rather than a design constraint. Something goes wrong, an incident fires, and someone starts frantically tracing through what changed and whether it can be reversed. That is the wrong order of operations. Rollback should be designed before execution begins, not improvised after something breaks. If a system can execute a change but cannot articulate how to reverse it, that system is incomplete — it has half an answer. Before a change is approved, the question should already be answered: if this goes wrong at step three, what exactly do we do? That question disciplines the planning. It surfaces assumptions. It often catches complexity that would have been invisible until production.
The economics of this are straightforward. Organizations obsess over preventing mistakes, but no review process eliminates uncertainty and no senior engineer sees every failure mode. A better strategy is making mistakes inexpensive. If recovery is fast and reliable, teams can test assumptions. If every change feels irreversible, teams become conservative, accumulate risk slowly, and eventually experience the failure they were trying to avoid anyway — just at a worse time and at larger scale. Rollback is not a sign of failure. Rollback is resilience. Teams that can roll back quickly can ship more aggressively because the cost of being wrong is bounded.
What real rollback requires is specific. The exact before-state, captured at execution time — not what the config said last week, not what the audit log recorded after the fact, but the actual state of the specific resources being modified in the moment before they're touched. That state stored bound to the specific change, not in a separate log you have to correlate under pressure. And a typed, tested restore operation — rolling back a credential rotation is a different operation than re-enabling a disabled account, and the undo has to know how to restore its specific kind of change. The last known good state should carry more than configuration: the assumptions that made the change reasonable, the recent changes that preceded it, the validation evidence that confirmed it, the rollback procedure attached to it. That is the object you need when something breaks. Not logs. Not a runbook. A preserved decision context you can actually return to.
The next time someone tells you they have rollbacks, ask them the follow-up. When did you last run it? Against what? Did the system come back the way you expected? If the answer is a shrug, they don't have rollbacks. They have a story they're telling themselves about the worst day they haven't had yet.