Debouncing Is Not a Personality

One useful habit is to ask what would have told us sooner. Not who should have caught it, but what signal already existed and was not being looked at. Usually there was a log line or a metric sitting there the whole time, and a small alert costs far less than the outage did.

We had been treating this as a small problem for about a year. It only ever appeared under load, it went away on a retry, and nobody could reproduce it on a laptop. That combination is how a small problem gets to stay in a codebase long enough to become part of the architecture.

What surprised me was how much of the cost was in the waiting. The work itself was quick. The build took nine minutes, the review sat for a day and the deploy window was on Thursdays. Shaving a second off the code path would have been easy and would not have changed a thing.

Where it started

• Leave a note for the next person.
• Test it with the keyboard.
• Write down what you expected to happen.
• Say what you measured separately from what you concluded.
• Run it twice and see what breaks.

After that, the only thing left was to delete the workaround. This is the step that tends to get skipped, which is how a codebase ends up with fixes for bugs that stopped existing years ago. Put the removal in the same ticket, or accept that it will outlive everyone on the team.

We now write a short note after every incident, and the rule is that it has to be useful to somebody who was asleep at the time. What happened, how we noticed, what we did and what we changed. Blame is not one of the headings, and it has never needed to be.

Naming turned out to matter more than the structure did. Once the function stopped being called process and started saying what it did, two of the three bugs in the surrounding code became obvious to everybody reading it, including the person who had written them, who was me.

Things that did not help

We did not find the bug. We found the place where it had been living.

Jonas Whitlock

The fix itself took four lines. Finding out where those four lines belonged took most of a week, including one afternoon spent proving that the bug was not where everyone, including me, had been absolutely certain it was. That is a normal ratio, and it is worth telling new people about.

There is a version of this that uses a library, and for a larger team it might be the right call. We looked at it, read the changelog and counted the dependencies. For the handful of cases we actually have, thirty lines we understand beat three thousand we would have to trust.

Every layer in this stack is caching something, and each one was added by a reasonable person solving a real problem. The trouble only starts when nobody remembers all of them. Before adding another, draw the whole path on paper and write down who clears each cache and when it happens.

A simpler version

Nobody reads the documentation until something is on fire, so write it for that reader. Put the command they need at the top. Say what it changes and how to undo it. Leave the history of how we got here for a later section, where it can be appreciated at leisure.

Test it with the keyboard before you call it finished. Tab through the page, press Enter on things and see where focus lands afterwards. It takes less than a minute, it needs no tooling at all, and it finds a remarkable number of problems that an automated checker will never report.

The review comment that stuck with me was a question: what happens if this runs twice? It had not occurred to me, and the answer was unpleasant. Since then I ask it of every job, webhook and migration I touch, and it has been worth more than any tool I have installed.

What we changed afterwards

Write the failing case down before you change anything. It can be a test, a curl command or a sentence in the ticket. The point is that you have something outside your own head that says what wrong looks like, because your memory of the bug starts drifting the moment you begin working on it.

Performance work is mostly subtraction. Before reaching for a clever technique, find out what the page is doing that nobody asked it to do. We have never yet profiled something without finding a piece of work that could simply be removed, and removal has the best cost profile of anything.

The first thing to do when something breaks is to stop guessing. Open the log, find the first line that looks wrong, and read it twice. Most of the time the message is telling you exactly what happened, and the only thing standing in the way is the theory you formed on the walk to the keyboard.

Reading the evidence

The browser is already very good at most of this. A surprising share of the script we removed was reimplementing things a plain link, a form or a details element will do for free, with better keyboard support and without the bugs we had carefully written ourselves.