The instruments lied more than fifteen times. The systems broke twice.

Over one day of work on our own systems, the tools we measure them with produced more than fifteen false readings. The systems themselves broke twice. Nothing crashed: every tool answered confidently, in the right format, with a plausible number in it. That is the defining property of the class.

The ratio is the part worth carrying away. If you assume instruments fail about as often as systems do, you will spend the first hour of every incident debugging the system — and eight times out of nine the system was fine.

Red lies more often than green

Our readings were wrong in both directions, but false alarms dominated: 373 broken addresses reported on sites that were healthy, a page reported as having no links at all when it had seventeen, a cookie banner reported missing while it was on screen, 83 points from a page speed run that scores a steady 95.

False green is rarer and worse, so it is worth naming what it looks like. A watchdog sampling /proc spawned one process per file; on a node with 619 processes the snapshot took 4.4 seconds, the measurement window slid apart, and a planted process genuinely burning 98 % of a core reported as 77 % — under the alarm threshold of 85. The watchdog would have stayed quiet through a fire, and the fault was in how it read, not in where the threshold sat. Rewritten as a single pass, the same snapshot takes 0.03 seconds.

Four ways a reading goes wrong

What you asked for is not what ran. Screenshots at 320 px looked like horizontal overflow on page after page. The browser holds a minimum window width of 500 px in headless mode, substitutes it silently, and still writes the PNG at the width you requested. The result is a crop, and a crop is indistinguishable from real overflow. The fix was not in the pages: the capture tool now drives the browser through the debug protocol and goes red itself when the width it got back is not the width it asked for.

The instrument measures a different link than your claim. Page load measured from our own laptop came back at 1.2 to 2.0 seconds, and «the sites are slow» went into a report. Measured from inside the server the same pages answered in 1 to 19 milliseconds. The first number was not wrong — it was an honest measurement of the distance to a laptop, filed as a statement about the sites.

Zero findings and empty input look identical. A style audit reported zero cold tones across 12 sites. The external stylesheet it read did not exist: the CSS is inlined, so the input was empty and the answer was structurally zero. A weight check reported 2 KB for a page of 38 KB, because it read the content-length header, which compression removes. A count of addresses in a sitemap returned 1 where there were 79, because the XML was on one line and the counter counted lines. Suspiciously round and suspiciously small results are broken measurements far more often than they are good news.

The instrument counts its own traces. A search for serif found serifs on three healthy sites, because serif occurs inside sans-serif. A DNS lookup printed its own error message to standard output, the filter read that message as data, and the report said DKIM was present on all four selectors. And an alert we had logged as sent had actually come back Bad Request: can't parse entities — the response was discarded, so the log said «alert sent» and the channel stayed empty. Sending is not delivery until you read the answer.

Explain the red before you act on it

«Every address on the site is broken» cannot be true while the home page is answering; if the site were down, the home page would be down with it. An impossible picture means a broken instrument, not a broken world — and checking that costs a minute, against an hour spent on a system that was never ill.

Two habits make this cheap. First, print the volume read next to the number of findings:

bytes read: 0     findings: 0     # there was nothing to measure

Second, plant a case with a known answer and require the instrument to name it. A process that certainly burns a core. A page whose weight you know. Without that, the instrument checks itself, and it always agrees.

The planted case can be wrong too

Our first planted «stuck process» was a command waiting on an empty pipe. It blocks rather than spins, so a working watchdog correctly did not see it — and we nearly rewrote the watchdog to fix the decoy. Before repairing a check, confirm the decoy does what you think it does.

And a decoy that goes red does not prove the check is whole. A watchdog for a back button searched for one string anywhere in the file — a string that also appears in the button's handler. The button could be deleted from the markup, the dead handler would keep the check green, and out of five substitutions four went red and the fifth did not. The same shape cost us a dead history button elsewhere, bound to one key while its panel was bound to two, with 233 checks green throughout. The cure is to require each place separately: the drawing and the handling, asserted apart.

The checklist below does not score your measurement. It tells you what your current numbers cannot yet be used to prove, and it is explicit about the case it cannot see.