There is a version of this that uses a library, and for a larger team it might be the right call. We looked at it, read the changelog and counted the dependencies. For the handful of cases we actually have, thirty lines we understand beat three thousand we would have to trust.
Choosing boring technology is mostly about choosing where your surprises will come from. The database has been around for decades and its failure modes are written up in great detail. The newer thing may well be better, but its surprises will belong entirely to you, at a time of its choosing.
The number looked great on the dashboard, which should have been the first warning. It turned out we were averaging across every request, including the health checks, which are very fast and very frequent. Once they were excluded the graph told a much less flattering and far more useful story.
Trying it yourself
- Change one thing at a time.
- Test it with the keyboard.
- Leave a note for the next person.
Test it with the keyboard before you call it finished. Tab through the page, press Enter on things and see where focus lands afterwards. It takes less than a minute, it needs no tooling at all, and it finds a remarkable number of problems that an automated checker will never report.
We had been treating this as a small problem for about a year. It only ever appeared under load, it went away on a retry, and nobody could reproduce it on a laptop. That combination is how a small problem gets to stay in a codebase long enough to become part of the architecture.
What surprised me was how much of the cost was in the waiting. The work itself was quick. The build took nine minutes, the review sat for a day and the deploy window was on Thursdays. Shaving a second off the code path would have been easy and would not have changed a thing.
What we changed afterwards
If the dashboard looks great during an incident, the dashboard is the incident.
Priya Hale
One useful habit is to ask what would have told us sooner. Not who should have caught it, but what signal already existed and was not being looked at. Usually there was a log line or a metric sitting there the whole time, and a small alert costs far less than the outage did.
A good default is one that nobody has to think about. We spent a long time making this configurable and then discovered that every single environment used the same value. So we deleted the setting, wrote the value down in one place and removed an entire class of support questions.
Performance work is mostly subtraction. Before reaching for a clever technique, find out what the page is doing that nobody asked it to do. We have never yet profiled something without finding a piece of work that could simply be removed, and removal has the best cost profile of anything.
Why it stayed broken
Every layer in this stack is caching something, and each one was added by a reasonable person solving a real problem. The trouble only starts when nobody remembers all of them. Before adding another, draw the whole path on paper and write down who clears each cache and when it happens.
Staging will lie to you, gently and often. It has less data, less traffic and a much more forgiving network. Treat a clean run there as a sign the change is not obviously broken, and keep watching the real thing for a while after it ships, with one hand near the rollback.
Do the rename in its own commit. It feels like extra ceremony right up until someone tries to find out when the behavior changed and the history shows a single enormous diff. Separating the mechanical change from the interesting one costs thirty seconds and saves a very long afternoon.
What it cost
Write the failing case down before you change anything. It can be a test, a curl command or a sentence in the ticket. The point is that you have something outside your own head that says what wrong looks like, because your memory of the bug starts drifting the moment you begin working on it.
The fix itself took four lines. Finding out where those four lines belonged took most of a week, including one afternoon spent proving that the bug was not where everyone, including me, had been absolutely certain it was. That is a normal ratio, and it is worth telling new people about.
The run-it-twice question should be printed on a mug.
Fair point. It depends a lot on team size.
Fair point. It depends a lot on team size.
Good catch, thank you. I will add a note.
Same here. Glad it is not only us.
We hit almost exactly this last spring. The part about the logs is painfully accurate.
About forty minutes, batched, with nobody watching it.
Yes, and that was the moment we started measuring properly.
Same here. Glad it is not only us.
Fair point. It depends a lot on team size.
Yes, and that was the moment we started measuring properly.
How long did the migration take on the full table?
Yes, and that was the moment we started measuring properly.
Fair point. It depends a lot on team size.
Same here. Glad it is not only us.
Fair point. It depends a lot on team size.
Same here. Glad it is not only us.
Same here. Glad it is not only us.
Bookmarked for the next time someone proposes rewriting it from scratch.
Yes, and that was the moment we started measuring properly.
Small correction: the default changed in a recent release, so check which version you are on.
I disagree about the library, but only because I have already paid the cost of learning it.
We hit almost exactly this last spring. The part about the logs is painfully accurate.
Fair point. It depends a lot on team size.
Good catch, thank you. I will add a note.
Did you try this with a cold cache? We saw very different numbers on the first request.
Small correction: the default changed in a recent release, so check which version you are on.
Same here. Glad it is not only us.
We hit almost exactly this last spring. The part about the logs is painfully accurate.
About forty minutes, batched, with nobody watching it.
Same here. Glad it is not only us.
Reading this at 2am with the pager next to me. Very on brand.
We hit almost exactly this last spring. The part about the logs is painfully accurate.
Reading this at 2am with the pager next to me. Very on brand.
Did you try this with a cold cache? We saw very different numbers on the first request.
Did you try this with a cold cache? We saw very different numbers on the first request.
How long did the migration take on the full table?
This would have saved me a weekend in 2023.
How long did the migration take on the full table?
Fair point. It depends a lot on team size.
Reading this at 2am with the pager next to me. Very on brand.
How long did the migration take on the full table?
Same here. Glad it is not only us.
Good catch, thank you. I will add a note.
I disagree about the library, but only because I have already paid the cost of learning it.
The run-it-twice question should be printed on a mug.
Did you try this with a cold cache? We saw very different numbers on the first request.
About forty minutes, batched, with nobody watching it.
Thanks for writing this up. I have sent it to two people who needed it and one who will deny it.
Did you try this with a cold cache? We saw very different numbers on the first request.
We hit almost exactly this last spring. The part about the logs is painfully accurate.
How long did the migration take on the full table?
The run-it-twice question should be printed on a mug.
This would have saved me a weekend in 2023.
About forty minutes, batched, with nobody watching it.
Small correction: the default changed in a recent release, so check which version you are on.
This would have saved me a weekend in 2023.
Thanks for writing this up. I have sent it to two people who needed it and one who will deny it.
Yes, and that was the moment we started measuring properly.
I disagree about the library, but only because I have already paid the cost of learning it.