True on paper

The NAS page on this site used to say that my homelab ran twenty-nine containers across thirteen stacks, that everything was monitored, that every container came back on its own after a crash, and that the whole estate was tracked in Git. All of that was true. I'd checked it the way most people check things they wrote themselves, which is to say I'd read it again and agreed with it. This year I finally built things whose entire job was to disagree with me, and the interesting part isn't that they found problems. It's that almost nothing they found was a lie. The claims were all true. They just weren't worth what I thought they were worth.

The first one was an alarm. I wanted to know when a container was restarting over and over, the sort of quiet flapping that never quite takes anything down and never quite fixes itself either, so I wrote a rule to catch it. It shipped green. It passed its health check. It would also have stayed silent forever, because the number it was watching reports when a container was created, not when it last started, and that number does not change for the entire life of the container. The audit that caught it is more damning than the bug: one exporter had restarted thirty-nine times in twenty-one days while the metric I'd keyed on sat at a single unchanging value, and asking the whole estate whether anything had restarted lately returned a confident zero across all thirty-one containers over a fortnight. When I went looking for a second opinion, the platform's own built-in restart counter said five for the same container that two independent measurements put at thirty-nine and forty. It resets itself quietly, so it doesn't merely start counting from a new baseline, it under-reports and looks healthy doing it.

The second one was worse, because it was the claim I was proudest of. Compose definitions tracked in Git. Absolutely true. What I had never checked was whether the thing actually running the containers agreed with Git, and it turns out the orchestration layer keeps its own private copy of every stack definition the moment you deploy through it. That copy is a second source of truth, and it drifts in perfect silence: a change made through the web interface, or simply an older deploy that nobody revisited, is enough to leave the repository and reality describing two different estates with nothing anywhere announcing the disagreement. When I finally diffed all of them, seven of thirteen had drifted. Reconciling them back reclaimed about fourteen and a half gigabytes of orphaned images and volumes that the drift had been quietly keeping alive, which is a very undignified way to discover you have been paying rent on your own mistakes.

Two rules came out of that fortnight and they've both earned their place. The first is that a detector doesn't count as working until I've watched its signal move during a real instance of the thing it's supposed to catch, tested against history that already happened rather than against my confidence about what it ought to do. Shipping green means nothing; a rule that cannot fire is indistinguishable from a rule with nothing to report, and the difference between those two is the entire value of the alarm. The second is smaller and blunter: diff the stacks before every redeploy. Git is the source of truth is a claim about a process, not a fact about a system, right up until something is actually checking, and I had spent a year saying it as though the sentence did the work.

The detail I keep coming back to is why nobody noticed that exporter restarting thirty-nine times. It wasn't that the monitoring was bad. It's that the restart-always policy, which is genuinely the best decision I ever made about this estate and which that same NAS page rightly celebrates, was catching it every single time and putting it back on its feet before it could become my problem. The property that makes the whole thing resilient is the property that hides what it's healing. I've updated the page to say what's actually there now, including the numbers that changed, and I've stopped letting the word tracked into a sentence unless I can name the thing doing the checking.