The fix itself took four lines. Finding out where those four lines belonged took most of a week, including one afternoon spent proving that the bug was not where everyone, including me, had been absolutely certain it was. That is a normal ratio, and it is worth telling new people about.
Performance work is mostly subtraction. Before reaching for a clever technique, find out what the page is doing that nobody asked it to do. We have never yet profiled something without finding a piece of work that could simply be removed, and removal has the best cost profile of anything.
We had been treating this as a small problem for about a year. It only ever appeared under load, it went away on a retry, and nobody could reproduce it on a laptop. That combination is how a small problem gets to stay in a codebase long enough to become part of the architecture.
What to watch for
- Test it with the keyboard.
- Run it twice and see what breaks.
- Leave a note for the next person.
Keep the change small enough to explain in one sentence. If the description needs the word and, it is probably two changes, and the second one will be the one that causes trouble. Reviewers can hold one idea in their heads at a time, and so, at three in the morning, can you.
Do the rename in its own commit. It feels like extra ceremony right up until someone tries to find out when the behavior changed and the history shows a single enormous diff. Separating the mechanical change from the interesting one costs thirty seconds and saves a very long afternoon.
Nobody reads the documentation until something is on fire, so write it for that reader. Put the command they need at the top. Say what it changes and how to undo it. Leave the history of how we got here for a later section, where it can be appreciated at leisure.
Why it stayed broken
Every clever line of code is a note to a future tired person. Be kind to them.
Mara Voss
Old code deserves a little respect, even when it is bad. It has been running in production for years and has already met all the strange inputs you have not thought of yet. Read it carefully before rewriting it, because the odd conditional in the middle is usually a scar from a real incident.
It helps to separate what you measured from what you concluded. We measured that the page took longer after the change. We concluded that the new query was responsible. Only the first of those was a fact, and it took an embarrassing amount of time to notice that the second one was wrong.
Staging will lie to you, gently and often. It has less data, less traffic and a much more forgiving network. Treat a clean run there as a sign the change is not obviously broken, and keep watching the real thing for a while after it ships, with one hand near the rollback.
The wrong theory
Naming turned out to matter more than the structure did. Once the function stopped being called process and started saying what it did, two of the three bugs in the surrounding code became obvious to everybody reading it, including the person who had written them, who was me.
Every layer in this stack is caching something, and each one was added by a reasonable person solving a real problem. The trouble only starts when nobody remembers all of them. Before adding another, draw the whole path on paper and write down who clears each cache and when it happens.
One useful habit is to ask what would have told us sooner. Not who should have caught it, but what signal already existed and was not being looked at. Usually there was a log line or a metric sitting there the whole time, and a small alert costs far less than the outage did.
What we changed afterwards
After that, the only thing left was to delete the workaround. This is the step that tends to get skipped, which is how a codebase ends up with fixes for bugs that stopped existing years ago. Put the removal in the same ticket, or accept that it will outlive everyone on the team.
The review comment that stuck with me was a question: what happens if this runs twice? It had not occurred to me, and the answer was unpleasant. Since then I ask it of every job, webhook and migration I touch, and it has been worth more than any tool I have installed.
The browser is already very good at most of this. A surprising share of the script we removed was reimplementing things a plain link, a form or a details element will do for free, with better keyboard support and without the bugs we had carefully written ourselves.
A simpler version
What surprised me was how much of the cost was in the waiting. The work itself was quick. The build took nine minutes, the review sat for a day and the deploy window was on Thursdays. Shaving a second off the code path would have been easy and would not have changed a thing.
Test it with the keyboard before you call it finished. Tab through the page, press Enter on things and see where focus lands afterwards. It takes less than a minute, it needs no tooling at all, and it finds a remarkable number of problems that an automated checker will never report.
We now write a short note after every incident, and the rule is that it has to be useful to somebody who was asleep at the time. What happened, how we noticed, what we did and what we changed. Blame is not one of the headings, and it has never needed to be.
What it cost
There is a version of this that uses a library, and for a larger team it might be the right call. We looked at it, read the changelog and counted the dependencies. For the handful of cases we actually have, thirty lines we understand beat three thousand we would have to trust.
The first thing to do when something breaks is to stop guessing. Open the log, find the first line that looks wrong, and read it twice. Most of the time the message is telling you exactly what happened, and the only thing standing in the way is the theory you formed on the walk to the keyboard.