Three Ways to Fold a Long List

Staging will lie to you, gently and often. It has less data, less traffic and a much more forgiving network. Treat a clean run there as a sign the change is not obviously broken, and keep watching the real thing for a while after it ships, with one hand near the rollback.

Nobody reads the documentation until something is on fire, so write it for that reader. Put the command they need at the top. Say what it changes and how to undo it. Leave the history of how we got here for a later section, where it can be appreciated at leisure.

Keep the change small enough to explain in one sentence. If the description needs the word and, it is probably two changes, and the second one will be the one that causes trouble. Reviewers can hold one idea in their heads at a time, and so, at three in the morning, can you.

Reading the evidence

<nav aria-label="Chapters">
  <a href="#one" aria-current="true">One</a>
  <a href="#two">Two [draft]</a>
</nav>

plain preformatted text with no code element

Write the failing case down before you change anything. It can be a test, a curl command or a sentence in the ticket. The point is that you have something outside your own head that says what wrong looks like, because your memory of the bug starts drifting the moment you begin working on it.

Every layer in this stack is caching something, and each one was added by a reasonable person solving a real problem. The trouble only starts when nobody remembers all of them. Before adding another, draw the whole path on paper and write down who clears each cache and when it happens.

What surprised me was how much of the cost was in the waiting. The work itself was quick. The build took nine minutes, the review sat for a day and the deploy window was on Thursdays. Shaving a second off the code path would have been easy and would not have changed a thing.

What we measured

Choosing boring technology is mostly about choosing where your surprises will come from. The database has been around for decades and its failure modes are written up in great detail. The newer thing may well be better, but its surprises will belong entirely to you, at a time of its choosing.

Do the rename in its own commit. It feels like extra ceremony right up until someone tries to find out when the behavior changed and the history shows a single enormous diff. Separating the mechanical change from the interesting one costs thirty seconds and saves a very long afternoon.

A good default is one that nobody has to think about. We spent a long time making this configurable and then discovered that every single environment used the same value. So we deleted the setting, wrote the value down in one place and removed an entire class of support questions.

A simpler version

The number looked great on the dashboard, which should have been the first warning. It turned out we were averaging across every request, including the health checks, which are very fast and very frequent. Once they were excluded the graph told a much less flattering and far more useful story.

It helps to separate what you measured from what you concluded. We measured that the page took longer after the change. We concluded that the new query was responsible. Only the first of those was a fact, and it took an embarrassing amount of time to notice that the second one was wrong.