Index First, Ask Questions Later

Test it with the keyboard before you call it finished. Tab through the page, press Enter on things and see where focus lands afterwards. It takes less than a minute, it needs no tooling at all, and it finds a remarkable number of problems that an automated checker will never report.

There is a version of this that uses a library, and for a larger team it might be the right call. We looked at it, read the changelog and counted the dependencies. For the handful of cases we actually have, thirty lines we understand beat three thousand we would have to trust.

Choosing boring technology is mostly about choosing where your surprises will come from. The database has been around for decades and its failure modes are written up in great detail. The newer thing may well be better, but its surprises will belong entirely to you, at a time of its choosing.

What we measured

EXPLAIN SELECT post_id, COUNT(*) AS total
FROM wp_postmeta
WHERE meta_key = '_price'
GROUP BY post_id
ORDER BY total DESC
LIMIT 10;

The first thing to do when something breaks is to stop guessing. Open the log, find the first line that looks wrong, and read it twice. Most of the time the message is telling you exactly what happened, and the only thing standing in the way is the theory you formed on the walk to the keyboard.

The number looked great on the dashboard, which should have been the first warning. It turned out we were averaging across every request, including the health checks, which are very fast and very frequent. Once they were excluded the graph told a much less flattering and far more useful story.

We had been treating this as a small problem for about a year. It only ever appeared under load, it went away on a retry, and nobody could reproduce it on a laptop. That combination is how a small problem gets to stay in a codebase long enough to become part of the architecture.

Where it started

no language here, so this block gets line numbers but no colors

Performance work is mostly subtraction. Before reaching for a clever technique, find out what the page is doing that nobody asked it to do. We have never yet profiled something without finding a piece of work that could simply be removed, and removal has the best cost profile of anything.

It helps to separate what you measured from what you concluded. We measured that the page took longer after the change. We concluded that the new query was responsible. Only the first of those was a fact, and it took an embarrassing amount of time to notice that the second one was wrong.

We now write a short note after every incident, and the rule is that it has to be useful to somebody who was asleep at the time. What happened, how we noticed, what we did and what we changed. Blame is not one of the headings, and it has never needed to be.

Pages: 1 2