Nobody reads the documentation until something is on fire, so write it for that reader. Put the command they need at the top. Say what it changes and how to undo it. Leave the history of how we got here for a later section, where it can be appreciated at leisure.
Keep the change small enough to explain in one sentence. If the description needs the word and, it is probably two changes, and the second one will be the one that causes trouble. Reviewers can hold one idea in their heads at a time, and so, at three in the morning, can you.
We now write a short note after every incident, and the rule is that it has to be useful to somebody who was asleep at the time. What happened, how we noticed, what we did and what we changed. Blame is not one of the headings, and it has never needed to be.
Why it stayed broken
- Change one thing at a time.
- Keep the rollback command in your clipboard.
- Check the clock on the server.
- Read the first error, not the last.
- Delete the workaround when the cause is fixed.
The review comment that stuck with me was a question: what happens if this runs twice? It had not occurred to me, and the answer was unpleasant. Since then I ask it of every job, webhook and migration I touch, and it has been worth more than any tool I have installed.
Performance work is mostly subtraction. Before reaching for a clever technique, find out what the page is doing that nobody asked it to do. We have never yet profiled something without finding a piece of work that could simply be removed, and removal has the best cost profile of anything.
Do the rename in its own commit. It feels like extra ceremony right up until someone tries to find out when the behavior changed and the history shows a single enormous diff. Separating the mechanical change from the interesting one costs thirty seconds and saves a very long afternoon.
What to watch for
Every clever line of code is a note to a future tired person. Be kind to them.
Tomasz Reyes
What surprised me was how much of the cost was in the waiting. The work itself was quick. The build took nine minutes, the review sat for a day and the deploy window was on Thursdays. Shaving a second off the code path would have been easy and would not have changed a thing.
One useful habit is to ask what would have told us sooner. Not who should have caught it, but what signal already existed and was not being looked at. Usually there was a log line or a metric sitting there the whole time, and a small alert costs far less than the outage did.
Old code deserves a little respect, even when it is bad. It has been running in production for years and has already met all the strange inputs you have not thought of yet. Read it carefully before rewriting it, because the odd conditional in the middle is usually a scar from a real incident.
I disagree about the library, but only because I have already paid the cost of learning it.
We hit almost exactly this last spring. The part about the logs is painfully accurate.
How long did the migration take on the full table?
Reading this at 2am with the pager next to me. Very on brand.
Thanks for writing this up. I have sent it to two people who needed it and one who will deny it.