Promises You Forgot to Await

The fix itself took four lines. Finding out where those four lines belonged took most of a week, including one afternoon spent proving that the bug was not where everyone, including me, had been absolutely certain it was. That is a normal ratio, and it is worth telling new people about.

Keep the change small enough to explain in one sentence. If the description needs the word and, it is probably two changes, and the second one will be the one that causes trouble. Reviewers can hold one idea in their heads at a time, and so, at three in the morning, can you.

Naming turned out to matter more than the structure did. Once the function stopped being called process and started saying what it did, two of the three bugs in the surrounding code became obvious to everybody reading it, including the person who had written them, who was me.

Things that did not help

<?php
namespace App\Cache;

final class Related {
	public function ids( int $post_id ): array {
		$cached = get_post_meta( $post_id, '_related', true );

		return is_array( $cached ) ? $cached : [];
	}
}

After that, the only thing left was to delete the workaround. This is the step that tends to get skipped, which is how a codebase ends up with fixes for bugs that stopped existing years ago. Put the removal in the same ticket, or accept that it will outlive everyone on the team.

The first thing to do when something breaks is to stop guessing. Open the log, find the first line that looks wrong, and read it twice. Most of the time the message is telling you exactly what happened, and the only thing standing in the way is the theory you formed on the walk to the keyboard.

Test it with the keyboard before you call it finished. Tab through the page, press Enter on things and see where focus lands afterwards. It takes less than a minute, it needs no tooling at all, and it finds a remarkable number of problems that an automated checker will never report.

Where it started

--- a/inc/cache.php
+++ b/inc/cache.php
@@ -12,3 +12,3 @@
-$key = 'related_' . $id . '_' . $generation;
+$key = 'related_' . $id;
 return get_transient( $key );

We now write a short note after every incident, and the rule is that it has to be useful to somebody who was asleep at the time. What happened, how we noticed, what we did and what we changed. Blame is not one of the headings, and it has never needed to be.

Do the rename in its own commit. It feels like extra ceremony right up until someone tries to find out when the behavior changed and the history shows a single enormous diff. Separating the mechanical change from the interesting one costs thirty seconds and saves a very long afternoon.

Performance work is mostly subtraction. Before reaching for a clever technique, find out what the page is doing that nobody asked it to do. We have never yet profiled something without finding a piece of work that could simply be removed, and removal has the best cost profile of anything.

The wrong theory

The review comment that stuck with me was a question: what happens if this runs twice? It had not occurred to me, and the answer was unpleasant. Since then I ask it of every job, webhook and migration I touch, and it has been worth more than any tool I have installed.

Staging will lie to you, gently and often. It has less data, less traffic and a much more forgiving network. Treat a clean run there as a sign the change is not obviously broken, and keep watching the real thing for a while after it ships, with one hand near the rollback.

Nobody reads the documentation until something is on fire, so write it for that reader. Put the command they need at the top. Say what it changes and how to undo it. Leave the history of how we got here for a later section, where it can be appreciated at leisure.

Trying it yourself

The number looked great on the dashboard, which should have been the first warning. It turned out we were averaging across every request, including the health checks, which are very fast and very frequent. Once they were excluded the graph told a much less flattering and far more useful story.

Every layer in this stack is caching something, and each one was added by a reasonable person solving a real problem. The trouble only starts when nobody remembers all of them. Before adding another, draw the whole path on paper and write down who clears each cache and when it happens.