298 checks passed. The defect shipped anyway.
A suite can be entirely green and cover one class of path out of four. Refusals at the door are cheap to write, so there are always many of them, and the total looks like coverage. What to count instead, and a tool that maps your own endpoint.
Our suite for this brand's site stood at 298 checks, none failing. A defect reached production anyway. Not because a check was wrong — because of what the number 298 was counting.
Why the total is the wrong unit
Checks are not equally expensive to write. A refusal at the door — wrong method, missing token, foreign origin, empty or oversized input — needs one request and no external service at all. Ten of those cost an afternoon. A single check of the SUCCESSFUL path needs a stubbed provider response, somewhere for state to live, and a decision about what «correct output» even means.
So the cheap ones accumulate, the total climbs, and the total is what gets reported. Ours said 298. Sorted by class of path, it said something else entirely: input refusals covered completely, and nothing else covered at all.
The four classes worth counting
- Refused at the door. Bad method, no token, foreign origin, input too long or empty. Cheap, and therefore always over-represented.
- Refused from outside. The provider answered with an error, answered with an empty body, or did not answer. Needs a stub, so it is usually missing.
- Success. The path the endpoint exists for: parse the answer, assemble what a person sees, record what it cost.
- Repeat. The second identical request. Where caches live, and where the quietest defects live with them.
These do not add up to a percentage, and that is deliberate. A percentage is precisely the number that produced the illusion: it sums cheap checks with expensive ones and hands back a single figure that feels like an answer.
The second habit, which costs nothing
A check that has never been made to fail is not known to work. Break each new one on purpose — invert the condition, feed it the wrong input — and confirm it goes red. A check that stays green under sabotage still counts toward your total, and it is the most expensive kind of check there is: it costs you the belief you place in the number.
We have caught real defects with exactly that habit, including in the tools on this site. The map below counts your classes; it deliberately refuses to give you a score.
Map your endpoint's paths
Tick the classes covered by at least one check. Your browser does the counting and nothing is sent anywhere. There is no percentage here on purpose — that number is what produces the illusion.
Read next
-
The response said 200. The body said no.
Checking status === 200 lets refusals through. Ours broke retries on two sites and one widget, and put a fabricated cause in the log.
-
Our error reporter accepted anything and forwarded it to a person
A POST with a foreign Origin was accepted with {"ok":true} and queued as an alert. One repeat request was all that stood between a stranger and the owner's messenger.
The reason this matters for anything you integrate with, ours included: a provider's suite being green tells you about the paths its authors found cheap to check. It says nothing about what happens when their upstream refuses, and that is the path your users will meet.
So the useful question to ask a provider is not «is it tested». It is «what does a refusal look like, and can I make one happen on demand». Ours answers with a status that matches the failure and a reference code that finds the log line behind it.