A design is a way of buying evidence
Small products get told, in effect, that measurement is for companies with more users. The arithmetic behind that is real and it does not relent; the arithmetic itself is covered in A/B testing when your traffic is thin, and this piece assumes it. What that arithmetic does not settle is the design question. A randomised user-level test is one way to buy evidence about a decision. It is the strongest and the most expensive, and it is not the only one. When it is out of reach, the alternatives are not "guess" — there is a ladder, each rung weaker and cheaper than the one above, and the skill is picking the lowest rung that is strong enough for the decision in front of you.
You are probably here because
- You have two hundred customers in total and somebody asked for an A/B test
- Users share a workspace, so randomising individuals contaminates both arms
- A test came back flat and nobody can say whether that means no effect or no power
- Six experiments ran last year, all inconclusive, and the roadmap is still settled by argument
The design ladder below covers the first two, the section on reporting nulls covers the third, and the section on experiment capacity covers the fourth.
How much evidence the decision actually needs
Two properties of the decision set the bar, and asking about them first prevents most of the wasted effort.
How reversible is it? A copy change on a settings page can be reverted in an hour with no residue. A change to the pricing page reaches customers who signed under it. A change to the data model, an onboarding flow that alters what you collect, or anything that touches a contract is difficult to walk back. Reversible changes need much less evidence, because the cost of being wrong includes the cost of undoing it, and for a reversible change that is small.
How large is the downside? Not the expected value, the bad tail. A change that might cost two percent of conversion is different from one that might break a workflow for the largest account. Evidence requirements should scale with the tail, not with the average.
Cross those two and the answer is often that no experiment is warranted. Reversible with a small downside: ship it, watch the guardrails, revert if something moves. That path is faster and cheaper than a six-week underpowered test whose result you would ignore anyway. Saying so is not an argument against measurement. It is what makes room for measurement where it counts: the irreversible decisions with a real tail, which is where the whole experiment budget belongs.
| Design | Fits when | Evidence strength | The main threat to it |
|---|---|---|---|
| User-level randomisation | Many independent users, no shared state | Strongest | Not enough of them; exposure dilution |
| Cluster randomisation by workspace, site or region | Users interact or share state | Strong, with a cost | Effective sample is the cluster count, not the user count |
| Within-subject or switchback | Effects are short-lived and repeat | Strong for what it covers | Carryover between periods; no learning effects allowed |
| Stepped rollout with matched pairs | Few clusters, rollout must happen anyway | Moderate | Time trends confounded with the rollout order |
| Comparison-group before-and-after | An untouched group exists that moves in parallel | Moderate, if parallel trends hold | Anything else that changed at the same moment |
| Offline replay against a labelled set | Model or ranking changes on logged inputs | Moderate; bounded to what the labels cover | The labels; and the gap between offline and live behaviour |
| Structured judgment elicited estimates, recorded and scored | Nothing above is available | Weak, but honest and cheap | Overconfidence, unless the estimates are scored later |
The randomisation unit comes from the interference, not from convenience
Randomise at the smallest unit where one participant's treatment cannot affect another's outcome. Everything else about the design follows from that decision, and getting it wrong invalidates results in a way no amount of analysis repairs.
Interference is easy to miss because it does not look like a statistical problem. In a collaboration product, two people in one workspace see each other's output, so putting them in different arms means the control arm is partly treated. In a marketplace, showing better ranking to some buyers takes supply away from the others, so a measured lift is partly a transfer rather than a gain. Anywhere a shared limit exists — inventory, a queue, a support team, a rate limit — the arms are connected through it.
The remedy is to move the unit up: randomise workspaces, regions, or time periods rather than people. That is correct, and it is expensive, because the effective sample size becomes the number of clusters and not the number of users inside them. Twenty workspaces is twenty units regardless of how many seats they hold, and twenty units detects only very large effects. Two hundred small clusters usually beat twenty large ones for the same total traffic, so where you have a choice, prefer many small clusters and balance them on pre-period outcomes before assigning.
Time-based randomisation is the underused option. If the effect appears and disappears quickly — a ranking change, a queueing policy, a model swap behind a synchronous call — alternate between variants on a schedule and compare periods. Every unit contributes to both arms, which removes between-user variance entirely, and interference through shared resources stops mattering because at any moment everyone sees the same thing. It fails for anything with learning or memory, since a user who saw the new version yesterday is not a clean control today.
Evidence strength against traffic required — our ranking
How much we would trust a positive result from each design at low volume. Judgment, not a benchmark. The ordering is the part worth keeping.
When the entire population is two hundred accounts
Business-to-business products hit a harder version of this. Two hundred accounts is not a sample, it is the market, and the accounts are wildly unequal: the largest may be a fifth of revenue on its own. No randomisation over that population produces balanced arms, because one account landing in the wrong arm dominates the comparison.
Four things work here, and none of them is an A/B test.
Randomise inside the account. The unit of analysis moves down from the customer to the task, the document, the ticket or the workflow run. A thousand support tickets inside twenty accounts gives real statistical power for a question about tickets, provided the outcome is a property of the ticket rather than of the relationship.
Match pairs, then stagger. Rank accounts on the pre-period outcome and on size, form pairs, and roll out to one of each pair first. Compare the pairs rather than the pooled arms. This is the design that fits a rollout that has to happen anyway, and staggering costs you nothing you were not already going to spend.
Use a comparison group and check the pre-period. If you must compare before and after, the only version worth reporting is one where an untouched comparison group moved in parallel with the treated group for several periods beforehand. Plot both series and show the parallel stretch. If they were not parallel before, the comparison is not evidence and the honest move is to say so rather than to add controls until the estimate looks acceptable.
Take qualitative evidence seriously and record it properly. With twenty accounts you can talk to all of them, which is a form of access that companies with a million users would pay a great deal for. Structured interviews with a consistent protocol, coded and counted, are real evidence about mechanism. They are weak about magnitude, so keep the two claims separate: use interviews to learn what changed and why, and a quantitative design for how much.
Turn a traffic problem into a labelling problem
For model, ranking and extraction changes there is often a much cheaper path, and it is the one we reach for first on AI features. The change can be evaluated on logged inputs before any live traffic touches it.
Assemble a set of real inputs sampled from production, including the ugly tail rather than a convenient slice, and label the correct output. Run both the current system and the candidate against it. Now the constraint is labels, not users, and labels can be bought — a few hundred well-chosen and carefully labelled cases will separate two genuinely different systems, and the set keeps paying every time something changes.
Be clear about what this does and does not tell you. It answers whether the new system produces better outputs on the distribution the set covers. It says nothing about whether users notice, change behaviour, or trust the result more. So use it as a gate rather than as proof: nothing reaches live traffic without passing it, and the live measurement afterwards can then be a coarse guardrail check rather than a fine-grained comparison you cannot afford. Curating the set is most of the work; a set built by sampling only easy cases will pass everything, which is worse than having no set because it is trusted.
Write the decision rule before the data arrives, in one sentence
Not "we will see if it is significant." Something like: if the point estimate is above plus one percent we ship, if it is below minus one percent we revert, and if it is between those we ship anyway because the change reduces support load and the downside is bounded. Written in advance, that sentence removes the argument that otherwise happens on the day the numbers land, when everyone already knows which answer they want. It also frequently reveals, before any effort is spent, that both branches lead to the same action — at which point the experiment can be cancelled and the week spent on something else.
Send the decision and the traffic, and we will pick the design.
Email the decision you are trying to make, your weekly volume at the surface involved, and the structure of your users to contact@precisionfederal.com. You get back a short written note: which rung of the ladder your situation supports, what effect size it could detect, and whether we would run anything at all. One business day. No charge and no meeting.
contact@precisionfederal.comDecide on expected loss, not on a threshold
The five percent significance level is a convention borrowed from settings where a false positive is a published claim that misleads a field for a decade. Shipping a feature that turns out to be neutral is not that. Holding product decisions to a standard designed for permanent scientific claims, using a sample that cannot meet it, produces paralysis and calls it rigour.
The alternative is to state what each error costs and choose the action with the lower expected loss. Concretely: take the posterior distribution of the effect, define a region around zero small enough that you are indifferent to anything inside it, and read off how much probability mass sits above and below. Then multiply by the cost of each mistake. If shipping a neutral change costs a week of engineering and not shipping a real two percent gain costs a quarter of compounding, those are not symmetric errors and the decision rule should not treat them as if they were.
This is not a way of lowering the bar so more things pass. It raises the bar in the other direction, on irreversible changes with expensive tails, where a result that clears five percent may still not justify the risk. It also means an interim look at the data is not cheating, provided the rule was written first and the stopping behaviour was accounted for — which is what the sequential methods in the arithmetic article are for.
A null result is not a finding until you report its detectable effect
The single most damaging habit in low-volume experimentation is reporting "no significant difference" and letting the room hear "the change did nothing." Those are different statements, and with thin traffic the second is almost never supported.
Every null needs its minimum detectable effect beside it. "We saw no difference; this test could only have detected a change of forty percent or larger" is a useful sentence, because it tells the reader the experiment was never capable of answering the question and stops the result from being cited for years as evidence of no effect. Report the confidence interval too, and read it out loud: an interval from minus eighteen to plus twenty-four percent has excluded almost nothing, and saying so is the honest summary.
Compute that number before running, not after. If the detectable effect is larger than anything the change could plausibly produce, the experiment is already known to be uninformative and the week is better spent elsewhere.
Experiment capacity is a budget, and most of it is wasted on small questions
Traffic is a shared resource. Every concurrent test divides it, every test occupies a surface for weeks, and the total number of decisions a small product can inform with evidence in a year is a small number — often three to six. Treat that as a budget and allocate it deliberately.
The waste comes from spending it on questions whose answer would not change anything, or on changes too small to detect. Twelve underpowered tests in a year produce twelve inconclusive results, several of which will be misread as findings. Three well-designed tests on decisions that are genuinely hard to reverse produce three answers. The second portfolio is better in every respect, and the only thing standing in the way is that the first one feels more active.
Two practical notes. Running several tests at once on unrelated surfaces is usually fine, and the interaction worry is overstated at low volume compared to the cost of running everything sequentially. But two tests on the same funnel step are one confounded test, no matter what the platform says about orthogonal assignment. And when traffic genuinely cannot support a design, an honest recommendation to ship-and-monitor is a better deliverable than a test that will be inconclusive by construction.
What replaces the test when there is no test
Shipping without an experiment is a legitimate choice, and it comes with obligations that are cheaper than most people think.
Guardrails with defined thresholds and an owner. Error rate, latency, support contact rate, the primary conversion step. Written down before the change, with the number that triggers a revert and the name of the person who decides.
A staged rollout that is actually staged. Five percent, then twenty-five, then everyone, with a pause between stages long enough for the metric to move. Most damage of any size becomes visible at five percent well before it reaches the whole base.
A revert path someone has tested. A flag that has never been switched off is not a revert path. Turn it off once in production, deliberately, at a quiet time, and confirm the system returns to the previous behaviour.
A written prediction. Before shipping, record what is expected to move and by how much. Look back a month later and compare. Over a year this produces something valuable and slightly humbling: a calibration record for the team's judgment, which is the only way to know whether the intuitions being used in place of experiments are any good.
Cost of getting each design decision wrong
How much damage each mistake does, as we rank it. The top two invalidate a result outright; the bottom one is worried about far more than it deserves.
The mistakes we are called in to fix
- Randomising individuals in a product where a workspace shares state, contaminating the control arm
- Cluster designs analysed as if users were independent, producing intervals several times too narrow
- "No significant difference" recorded as "no effect", and cited that way for two years
- A primary metric four steps downstream of the surface that changed, diluted past detection
- Before-and-after with no comparison group, in a month when three other things also shipped
- A switchback on a change users remember, so every control period is partly treated
- An offline evaluation set of easy cases that passes every candidate and is trusted anyway
- Twelve underpowered tests a year, consuming the whole traffic budget and settling nothing
A two-week design pass before anything launches
Evidence Design Pass
Step six catches more defects than any analysis afterwards. Ship the assignment and the exposure logging with both arms serving identical behaviour, leave it for a few days, and check that the arms are balanced in size and in pre-period outcomes. If they are not, something is wrong with the assignment, and it is far better to learn that during a null test than to discover it in an analysis of a result somebody already believes.
Before you launch
- The decision, its reversibility and the cost of each error are written down
- The randomisation unit is set by interference, not by what the platform makes easy
- Cluster designs are analysed at the cluster level, with clusters balanced beforehand
- Assignment fires at exposure, at the identical code point in every arm
- The minimum detectable effect is computed before launch and is plausible
- The decision rule, including what happens on an ambiguous result, is written first
- Guardrail metrics have thresholds and a named owner who can revert
- A model change has passed an offline set that includes the difficult cases
- An assignment-only dry run has confirmed balanced arms
- Any null result will be reported with its detectable effect and interval
Bottom line
Thin traffic does not remove the ability to learn; it removes the ability to learn small things quickly. The response is to spend the evidence budget on decisions that are hard to reverse and cheap to get wrong only in hindsight, to choose the randomisation unit from the structure of the product rather than the convenience of the tool, to move model questions offline where labels can be bought, and to write the decision rule before the numbers arrive. Where nothing on the ladder is strong enough, ship in stages, watch guardrails you defined in advance, and record what you expected so that a year from now the team's judgment is a measured quantity rather than an assertion.
Frequently asked questions
Not at the customer level for small effects, but usually yes at a lower level. Move the unit of analysis down to the task, document, ticket or session, where a few hundred accounts can generate tens of thousands of units. For questions that are genuinely about the account relationship, use matched pairs with a staggered rollout and compare within pairs, and accept that you will only ever detect large effects.
Randomise the workspace, not the user, then analyse at the workspace level. The effective sample size becomes the number of workspaces, which is a real cost and is why many small products get this wrong deliberately. Many small clusters are better than a few large ones for the same traffic, so split by the smallest boundary across which no state is shared, and balance clusters on pre-period outcomes before assigning.
With a comparison group and a demonstrated parallel pre-period, it is moderate evidence and worth reporting. Without one it is weak, because seasonality, a marketing push or anything else shipping that month is indistinguishable from your change. If that is genuinely all you have, present it as suggestive with the alternative explanations listed, and do not let it turn into a number quoted later without its caveats.
Partly, and it is usually the best first move. A curated set of real inputs with labelled correct outputs will separate two systems on output quality, and a few hundred well-chosen cases go a long way. What it cannot tell you is whether users notice or behave differently, so treat it as a gate before traffic rather than as the answer, and make sure the set includes the difficult tail rather than the cases that were easy to label.
Fewer than most teams attempt. Traffic is shared, each test occupies a surface for weeks, and three to six well-powered results is a realistic annual output for a low-volume product. Spending that budget on decisions that are hard to reverse produces more value than a dozen underpowered tests, several of which will be misread as findings and repeated as fact.