Fixture Data That Looks Like Real Data

Do the rename in its own commit. It feels like extra ceremony right up until someone tries to find out when the behavior changed and the history shows a single enormous diff. Separating the mechanical change from the interesting one costs thirty seconds and saves a very long afternoon.

A good default is one that nobody has to think about. We spent a long time making this configurable and then discovered that every single environment used the same value. So we deleted the setting, wrote the value down in one place and removed an entire class of support questions.

Keep the change small enough to explain in one sentence. If the description needs the word and, it is probably two changes, and the second one will be the one that causes trouble. Reviewers can hold one idea in their heads at a time, and so, at three in the morning, can you.

The wrong theory

• Write down what you expected to happen.
• Read the first error, not the last.
• Keep the rollback command in your clipboard.

The first thing to do when something breaks is to stop guessing. Open the log, find the first line that looks wrong, and read it twice. Most of the time the message is telling you exactly what happened, and the only thing standing in the way is the theory you formed on the walk to the keyboard.

Nobody reads the documentation until something is on fire, so write it for that reader. Put the command they need at the top. Say what it changes and how to undo it. Leave the history of how we got here for a later section, where it can be appreciated at leisure.

We had been treating this as a small problem for about a year. It only ever appeared under load, it went away on a retry, and nobody could reproduce it on a laptop. That combination is how a small problem gets to stay in a codebase long enough to become part of the architecture.

What it cost

If the dashboard looks great during an incident, the dashboard is the incident.

Odette Lang

Old code deserves a little respect, even when it is bad. It has been running in production for years and has already met all the strange inputs you have not thought of yet. Read it carefully before rewriting it, because the odd conditional in the middle is usually a scar from a real incident.

The review comment that stuck with me was a question: what happens if this runs twice? It had not occurred to me, and the answer was unpleasant. Since then I ask it of every job, webhook and migration I touch, and it has been worth more than any tool I have installed.

We now write a short note after every incident, and the rule is that it has to be useful to somebody who was asleep at the time. What happened, how we noticed, what we did and what we changed. Blame is not one of the headings, and it has never needed to be.

Reading the evidence

Every layer in this stack is caching something, and each one was added by a reasonable person solving a real problem. The trouble only starts when nobody remembers all of them. Before adding another, draw the whole path on paper and write down who clears each cache and when it happens.

One useful habit is to ask what would have told us sooner. Not who should have caught it, but what signal already existed and was not being looked at. Usually there was a log line or a metric sitting there the whole time, and a small alert costs far less than the outage did.

Write the failing case down before you change anything. It can be a test, a curl command or a sentence in the ticket. The point is that you have something outside your own head that says what wrong looks like, because your memory of the bug starts drifting the moment you begin working on it.

What to watch for

Test it with the keyboard before you call it finished. Tab through the page, press Enter on things and see where focus lands afterwards. It takes less than a minute, it needs no tooling at all, and it finds a remarkable number of problems that an automated checker will never report.

The number looked great on the dashboard, which should have been the first warning. It turned out we were averaging across every request, including the health checks, which are very fast and very frequent. Once they were excluded the graph told a much less flattering and far more useful story.

The fix itself took four lines. Finding out where those four lines belonged took most of a week, including one afternoon spent proving that the bug was not where everyone, including me, had been absolutely certain it was. That is a normal ratio, and it is worth telling new people about.

Things that did not help

After that, the only thing left was to delete the workaround. This is the step that tends to get skipped, which is how a codebase ends up with fixes for bugs that stopped existing years ago. Put the removal in the same ticket, or accept that it will outlive everyone on the team.