A Sense for Production — What Looks Different After You Have Been Through an Outage

What Someone Who Has Been Through It Reads Differently
Production Is a State, Not a Codebase
100% Is Not the Right Target
Observability, Rollback, and Gradual Rollout Are Not Tools
The Misreading That Speed Trades Against Stability
Three Ways to Learn It Without the Outage
Try It This Week
Further Reading
Sources

What Someone Who Has Been Through It Reads Differently

Two people read the same design doc. One reads how the feature works. The other reads something else in the same pages: what logs exist when this breaks at three in the morning, how many minutes a rollback takes, and what happens to the data that already landed once you roll back.

The second person is not smarter. They got burned once. And the difference does not show up as knowledge, it shows up as defaults — not whether they know, but whether they check before anyone asks. Which is why a sense for production is hard to test for and obvious within thirty minutes of a design review.

Production Is a State, Not a Codebase

Code lives in a repository and has one version. Production is a state and has many. The image currently running, the config currently applied, the value someone changed by hand last month and wrote down nowhere, three years of data that has drifted into a shape the docs no longer describe, the number of connections open right now. Of those, the repository knows the first one.

Which is why "it works on my machine" is not a joke but an accurate statement. It ran against a different state, so it produced a different result, and closing that gap is most of what operations is.

Here the conclusion from part 3 returns. A running system is the most current documentation of itself. Half of a production sense is the habit of reading that documentation; the other half is the habit of always keeping in your arithmetic that it may differ from the code.

100% Is Not the Right Target

The chapter on risk in Google's SRE book puts it plainly: 100% is probably never the right reliability target — not only is it impossible to achieve, it is typically more reliability than a service's users want or notice.

The same chapter attaches the price. Maximizing stability limits how fast new features can be developed and how quickly products reach users, and it dramatically increases cost. And that cost does not rise linearly: an incremental improvement in reliability may cost a hundred times more than the previous increment.

The device that comes out of this is the error budget. The gap between your target and 100% is treated as the budget of unreliability remaining for the period. The real value of the device is not the arithmetic, it is the conversation. How reliable this has to be stops being a contest of preferences and becomes a question with a number in it. And once it is a number, development and operations stop standing on opposite sides.

Observability, Rollback, and Gradual Rollout Are Not Tools

These three usually arrive attached to product names, but the thing that actually separates people is not the tooling.

Observability is not what you installed. It is whether, before deploying, you have an answer to where you would look if this goes wrong. Twenty dashboards with no answer to that question is no observability at all.

Rollback is not whether the command exists but whether you built a state you can return to. Change a schema into a shape the previous version cannot read and you have a rollback command with no rollback behind it. Being able to go back is decided at design time, not at deploy time.

Gradual rollout is not a percentage feature; it is deciding the stop rule in advance. "Put it on 5% and watch" is a plan with the stopping criterion missing, and without one, people watch for a while and then just go to a hundred.

Example — four lines filled in before the deploy

what counts as worse:  which metric crossing which value
where you would see it: the dashboard or log query for that metric
when you stop:          how long you watch, and at what value you abort
what rollback does to data: can it be reversed, and if not, what remains

Deploying without those four lines is not automatically wrong. It is just better to know that what you are doing is an experiment rather than a deploy.

The Misreading That Speed Trades Against Stability

One premise is common: deploying more often is riskier, so if you want stability you have to go slow.

DORA's guide to their metrics says the opposite. Across deployment frequency, change lead time, failed deployment recovery time, change fail rate, and deployment rework rate, the metrics are correlated for most teams — top performers do well on all of them and low performers do poorly on all of them. The same document states explicitly that speed and stability are not tradeoffs.

Do not read that observation as causation. Correlation does not give you a direction, and some other organizational factor may have lifted all five at once. What can be read structurally is this: the machinery that makes changes safe and the machinery that makes them fast is largely the same machinery. If you can roll back you can ship more often; if you ship often each change is smaller; if changes are small the list of suspects is short when something breaks.

Three Ways to Learn It Without the Outage

The surest route to a production sense is living through a big outage, which is not a study plan anyone should recommend. Three substitutes are available.

The first is reading other people's postmortems. Read straight through and it stays someone else's story, so change the order. Read only the timeline first, decide what you would have done at that moment, then read the rest. The point where your answer and theirs diverge is the part worth learning.

The second is going on call. People who have received the pages design pages differently. Alerts built by someone who has never been on call are usually too numerous, too many alerts get ignored, and ignored alerts are the same as no alerts.

The third is breaking things on purpose. Book the time, stop one thing you depend on, and watch how quickly the team notices. What usually surprises people is not the recovery time. It is that nobody noticed at all.

Try It This Week

Take one deploy going out this week and write the four lines above before it ships. Five minutes. If any of the four will not write, that line is the riskiest part of this deploy.

SLO & Error Budget Calculator — turn an availability target into allowed downtime and a budget and you find out by arithmetic whether the current architecture can reach it at all. That dependency availabilities multiply is the part that stings.
Linux Terminal Simulator — practice inspecting the state of a running system by hand, safely and repeatedly. Observability is a habit of looking before it is a thing you buy.

Where this does not apply: on a product with no users yet, or an internal-only tool, thicker deploy procedure means slower learning. At that stage only one of the four lines earns its keep — whether the data can be reversed. The rest can wait until you have users.

Sources

Embracing Risk — Google SRE Book — the statement that 100% is probably never the right reliability target and is more than users want or notice, the argument that extreme reliability limits development speed and dramatically increases cost, the note that an incremental reliability improvement may cost 100x the previous increment, and the definition of the error budget all come from here. Read 2026-08-15.
DORA metrics: the four keys — dora.dev — the definitions of the five metrics and the statement that speed and stability are not tradeoffs, with the metrics correlated for most teams, come from here. Read 2026-08-15.
The four pre-deploy lines and the three substitutes for living through an outage are not from those sources; they are what this post assembles.