Small Tokens, Large Consequences

There is a version of this that uses a library, and for a larger team it might be the right call. We looked at it, read the changelog and counted the dependencies. For the handful of cases we actually have, thirty lines we understand beat three thousand we would have to trust.

What surprised me was how much of the cost was in the waiting. The work itself was quick. The build took nine minutes, the review sat for a day and the deploy window was on Thursdays. Shaving a second off the code path would have been easy and would not have changed a thing.

Test it with the keyboard before you call it finished. Tab through the page, press Enter on things and see where focus lands afterwards. It takes less than a minute, it needs no tooling at all, and it finds a remarkable number of problems that an automated checker will never report.

What we measured

• Delete the workaround when the cause is fixed.
• Keep the rollback command in your clipboard.
• Run it twice and see what breaks.
• Change one thing at a time.
• Read the first error, not the last.

The number looked great on the dashboard, which should have been the first warning. It turned out we were averaging across every request, including the health checks, which are very fast and very frequent. Once they were excluded the graph told a much less flattering and far more useful story.

Nobody reads the documentation until something is on fire, so write it for that reader. Put the command they need at the top. Say what it changes and how to undo it. Leave the history of how we got here for a later section, where it can be appreciated at leisure.

Staging will lie to you, gently and often. It has less data, less traffic and a much more forgiving network. Treat a clean run there as a sign the change is not obviously broken, and keep watching the real thing for a while after it ships, with one hand near the rollback.

The actual fix

We did not find the bug. We found the place where it had been living.

Priya Hale

It helps to separate what you measured from what you concluded. We measured that the page took longer after the change. We concluded that the new query was responsible. Only the first of those was a fact, and it took an embarrassing amount of time to notice that the second one was wrong.

Do the rename in its own commit. It feels like extra ceremony right up until someone tries to find out when the behavior changed and the history shows a single enormous diff. Separating the mechanical change from the interesting one costs thirty seconds and saves a very long afternoon.

Old code deserves a little respect, even when it is bad. It has been running in production for years and has already met all the strange inputs you have not thought of yet. Read it carefully before rewriting it, because the odd conditional in the middle is usually a scar from a real incident.

What it cost

The browser is already very good at most of this. A surprising share of the script we removed was reimplementing things a plain link, a form or a details element will do for free, with better keyboard support and without the bugs we had carefully written ourselves.