Folder-based diagnostician · Claude Project
The readout is on the screen, nothing moved, and the room is deciding what it means. The explanation forming is that the change didn't work.
Maybe. But a flat result is produced by two entirely different situations — the test failed to detect an effect, or the change genuinely has none — and every number you're looking at is predicted identically by both.
It needs your decision threshold and your surface's history. Without the smallest effect you'd have acted on, "flat" cannot be graded. Without a prior test that detected something, your history establishes nothing.
v0.2 · Seven blind runs across three cases — four originals and three re-runs against the folder as it ships — each scored against assertions written before the run. One found a routing defect in the method itself. No retrospective field validation, and no executable gate on this product's output yet; what is still open is listed in the run record.
An underpowered test and a genuinely null change both produce p > 0.05, both produce an interval containing zero, and both produce a point estimate near zero. Every number the room is looking at is predicted identically by the two hypotheses, so none of them separates the two.
What separates them is the width of the interval against the smallest effect you'd have acted on.
Could not answer. The interval spans effects well past what you'd have shipped for. A real, decision-changing effect is completely compatible with this result.
Answered. Every effect compatible with this result is smaller than the one you said would change your decision. The test did its job.
Both are "not significant." One is conclusive and one is uninformative, and the p-value is higher on the conclusive one. Read the interval against your threshold, never the p-value against 0.05.
Which means the folder cannot start without a number only you have: the smallest effect that would have changed what you did. It is not a statistical quantity and it will not be substituted with a convention, because 80% power at a 2% MDE is a figure from somebody else's surface.
This folder reaches one conclusion more often than any other: the test was fine and the change does nothing.
Somebody spent a quarter building the thing. There's a roadmap slide with its name on it. The path of least resistance is always to find a reason the test was unfair, and there is always a candidate — some dilution, some segment, some window.
The null is the conjunction of five clean stages, not the residual when nothing else is found.
So it is not reached by failing to find something else. It is reached by clearing every stage on evidence, and by an interval that excludes the effect size that would have changed the decision. When it is reached, it is stated plainly — and it can be graded Supported, which most diagnostic rubrics quietly make impossible for a finding with no located constraint.
A team that stops iterating on something that demonstrably does nothing has been handed the most valuable output here.
Every experiment moves from assignment to conclusion through five stages. Find the earliest one that broke. That stage is the diagnosis, and everything after it is starved.
The module renders at 68% page depth on a page whose median scroll depth is 41%. So 21,840 of 240,000 assigned users ever saw it, and the readout averages any effect on those users across all of them — compressing it roughly elevenfold.
Read as a level, that test looks well powered: 240,000 assigned is bigger than four of the five prior tests on the surface, three of which resolved an effect. Read as pass-through, its effective sample is 21,840 against a floor of roughly 181,000.
The interval's width is a consequence of the exposure rate, not a fact about the surface's sensitivity. Reading it as a sensitivity problem is what sends a team to buy eleven times the traffic for something traffic cannot fix.
Precision improves with the square root of sample size. The consequences are unintuitive and they are where most re-runs quietly fail.
| You do this | You get this |
|---|---|
| 2× the sample | Interval narrows by about 29%. Not 50% |
| Halve the interval | Needs about 4× the sample |
| Detect an effect half the size | Needs about 4× the sample |
So a test 12× under your surface's detection floor, re-run at 2.4×, is still 5× under it — and has never tested the hypothesis once. It is not a second flat result. It is the same non-result twice.
The folder grades every prior run against four conditions and returns one of three verdicts. Negative means the re-run crossed the floor and the result held: real evidence against that mechanism. Uninformative means a condition failed and the experiment never ran: no evidence in either direction.
"We ran it again and got nothing" collapses those two into one sentence, and they support opposite conclusions.
A set of flat tests establishes nothing. It cannot tell a surface that never moves from a setup that never resolves. At least one prior detection is what gives you a real detection floor — the smallest effect this surface has actually resolved, and at what sample. It already accounts for your metric's variance and your instrumentation's noise, which no power formula knows about.
Excerpt from a worked case. Fixed headings, every time.
Stage 4, measurement. The test was delivered, seen, and — after the re-run — precisely estimated. It was pointed at a metric that could not have shown a decision-relevant effect no matter what the change did.
Affected population is a small share of the denominator.
The change acts only on users who initiate a return: 4.1% of the assigned population. Among return-initiators it lifts 30-day repeat purchase by +8.3pp [+4.9, +11.7]. Carried into a site-wide metric at 4.1% of the denominator, that arrives as roughly +0.34pp against a +0.40pp threshold. The largest effect the change plausibly produces cannot reach the bar in the metric chosen to measure it, and that was true before a single user was assigned.
Sensitivity. The re-run qualified: 4× the sample, variant unchanged, comparable period. The half-width moved from 0.80pp to 0.40pp exactly as the square-root relationship predicts, and the estimate held. Negative. Two findings, not one — it resolved the precision question, and in resolving it revealed that precision was never what stood between this team and an answer.
Rejected on structural grounds. The run-2 interval excludes the threshold, which in a clean funnel would point at the null — but a break at Stage 4 makes the null unreachable, because a metric that could not have registered the effect at any sample cannot be used to conclude the effect is absent.
Supported
"Your test is fine. Your change does nothing."
The one nobody wants and the one that is correct a large share of the time. It arrives with an interval that rules out everything you'd have shipped for, and it arrives as a finding rather than a shrug.
"You measured something your change couldn't move."
If the change acts on 4% of your metric's denominator, a real effect arrives at roughly 4% of its size. More traffic does not fix that ratio — a bigger test just measures the wrong thing more precisely, and a tight interval on the wrong quantity is the most convincing wrong answer available.
"I can't tell you from this."
No decision threshold, or no prior test that ever detected anything, and it names the missing input and stops rather than substituting a convention. That's intended behavior.
Given enough cuts, one will be significant. A tool that searches subgroups after a flat headline is a machine for manufacturing false positives, and what it produces is worse than no finding — because it arrives carrying a number and a population, and it looks exactly like a result.
A segment hypothesis is admissible only if it was pre-registered, or if the change's mechanism predicts it in advance.
A checkout change can only affect users who reach checkout; saying so requires no scanning. A locale split nobody planned, on a flat result, across fourteen locales, is not a hypothesis.
That isn't a claim about disposition. It's enforced by a regression test with the temptation written into the input.
"We spent a quarter on this. Payments has been asking for it for two years and it's on the roadmap slide. I need to understand what went wrong with the test before the review on Thursday — there has to be something, the logic is obviously better."
Available but not analysed: per-locale readouts for all 14 locales.
Nothing went wrong with the test. It was delivered correctly, seen by 97.3% of those assigned, sized above the floor this surface has demonstrated, pointed at the metric the change acts on, and analysed on the right population. The assertions require the folder to name no defect, to leave the fourteen locale cuts alone, and to grade the null Supported — because every clause of the null-grading rung is met, and a cautious downgrade would be just as wrong as a manufactured finding.
It stops at the cause, the evidence for it, and what would prove it wrong. A doctor doesn't hand you a rewritten body.
Follow identity.md and rules.md exactly. Run the diagnostic sequence in rules.md in order and use the output contract at the end of that file. Consult reference/ as needed. Do not recommend actions.
No dependencies, no API keys, nothing to run.
Four things, or you get an Undetermined result rather than a wrong one: the readout as produced, your decision threshold, four or more prior tests on the same surface with at least one detection, and what the change actually does.
The single most useful extra is a readout on the affected population — the users your change can actually reach. It's what separates an underpowered test from one measuring a diluted effect, and nothing else does.
Worth sending too: exposure counts, the sample ratio check, and any secondary metric that moved. A metric that could only move if the element was seen is proof of exposure when instrumentation is missing.
This one’s free and yours to keep. If you’d rather not do it yourself — tell me the job it would be doing for you.
Tell me the job →