An Nginx Config We Can Explain

Do the rename in its own commit. It feels like extra ceremony right up until someone tries to find out when the behavior changed and the history shows a single enormous diff. Separating the mechanical change from the interesting one costs thirty seconds and saves a very long afternoon.

Test it with the keyboard before you call it finished. Tab through the page, press Enter on things and see where focus lands afterwards. It takes less than a minute, it needs no tooling at all, and it finds a remarkable number of problems that an automated checker will never report.

We now write a short note after every incident, and the rule is that it has to be useful to somebody who was asleep at the time. What happened, how we noticed, what we did and what we changed. Blame is not one of the headings, and it has never needed to be.

A simpler version

• Check the clock on the server.
• Say what you measured separately from what you concluded.
• Run it twice and see what breaks.
• Delete the workaround when the cause is fixed.
• Test it with the keyboard.

The browser is already very good at most of this. A surprising share of the script we removed was reimplementing things a plain link, a form or a details element will do for free, with better keyboard support and without the bugs we had carefully written ourselves.

One useful habit is to ask what would have told us sooner. Not who should have caught it, but what signal already existed and was not being looked at. Usually there was a log line or a metric sitting there the whole time, and a small alert costs far less than the outage did.

Naming turned out to matter more than the structure did. Once the function stopped being called process and started saying what it did, two of the three bugs in the surrounding code became obvious to everybody reading it, including the person who had written them, who was me.

Reading the evidence

Every clever line of code is a note to a future tired person. Be kind to them.

Tomasz Reyes

A good default is one that nobody has to think about. We spent a long time making this configurable and then discovered that every single environment used the same value. So we deleted the setting, wrote the value down in one place and removed an entire class of support questions.

There is a version of this that uses a library, and for a larger team it might be the right call. We looked at it, read the changelog and counted the dependencies. For the handful of cases we actually have, thirty lines we understand beat three thousand we would have to trust.

It helps to separate what you measured from what you concluded. We measured that the page took longer after the change. We concluded that the new query was responsible. Only the first of those was a fact, and it took an embarrassing amount of time to notice that the second one was wrong.

What we changed afterwards

The first thing to do when something breaks is to stop guessing. Open the log, find the first line that looks wrong, and read it twice. Most of the time the message is telling you exactly what happened, and the only thing standing in the way is the theory you formed on the walk to the keyboard.

The fix itself took four lines. Finding out where those four lines belonged took most of a week, including one afternoon spent proving that the bug was not where everyone, including me, had been absolutely certain it was. That is a normal ratio, and it is worth telling new people about.

Write the failing case down before you change anything. It can be a test, a curl command or a sentence in the ticket. The point is that you have something outside your own head that says what wrong looks like, because your memory of the bug starts drifting the moment you begin working on it.

Trying it yourself

Every layer in this stack is caching something, and each one was added by a reasonable person solving a real problem. The trouble only starts when nobody remembers all of them. Before adding another, draw the whole path on paper and write down who clears each cache and when it happens.

We had been treating this as a small problem for about a year. It only ever appeared under load, it went away on a retry, and nobody could reproduce it on a laptop. That combination is how a small problem gets to stay in a codebase long enough to become part of the architecture.