A/B testing when your traffic is thin

The number that ends most of these conversations

Almost every experimentation program at a small or mid-size product dies at the same moment: somebody computes the sample size, sees a figure larger than the company's annual traffic, and quietly decides that testing is for companies with more users. The arithmetic is real and it is not going to relent. But the formula has four inputs, and most teams only ever try to change one of them.

Start with the arithmetic so nobody has to argue about it later. For a two-proportion comparison at a two-sided five percent level with eighty percent power, the users you need per arm are approximately (z_α/2 + z_β)² × [p₁(1−p₁) + p₂(1−p₂)] / (p₂−p₁)². The leading constant works out to 7.85. Lehr's shortcut, which is close enough for planning, is sixteen times the variance over the squared effect.

Put a real funnel through it. Baseline conversion is four percent. You want to detect a twenty percent relative improvement, which is 0.8 of a percentage point in absolute terms. The formula returns about 10,300 users per arm, so 20,600 in total. If the page in question sees 2,000 visitors a month, the test finishes in ten months, assuming nothing about the product, the pricing or the audience changes in the meantime. Something always does.

Now run it backwards, which is the version worth writing down. Four weeks of that traffic gives 1,000 users per arm, and 1,000 per arm detects an absolute change of about 2.5 percentage points on a four percent base — a sixty percent relative change. A full quarter brings it to about thirty-five percent relative. Those two numbers belong at the top of every experiment document.

Effect size is squared. Everything else is linear.

The denominator is the only term with an exponent, which means it is the only lever with real force. Halve the effect you are trying to detect and you quadruple the users you need. Double the effect you are willing to chase and the requirement drops to a quarter. Nothing else in the formula behaves like that.

The operational consequence is unpopular and correct: at low volume you cannot test small changes. Button colors, microcopy tweaks, a shifted field order — those live in the one-to-three percent relative range, and detecting a two percent relative change on a four percent baseline needs roughly a million users per arm. You will never have them.

So the portfolio changes. Test the flow that removes a step, not the one that renames the button. Test the pricing page that leads with a different unit of value. Test onboarding that defers account setup until after first value. Those plausibly move a rate by twenty to sixty percent relative, which is inside what your traffic can see.

Exposure dilution is the most expensive mistake and the cheapest to fix

Here is the failure we find most often when we audit an experimentation setup, and it costs more than everything else on this page combined. The assignment event fires when a user becomes eligible — on session start, on login, at the router — rather than when the user is actually exposed to the thing you changed.

If only fifty-five percent of assigned users ever reach the changed surface, the effect measured across the assigned population is 0.55 of the true effect, because the rest could not have responded to anything. Sample size scales with the inverse square of the effect, so that dilution multiplies your traffic requirement by 3.3×. At thirty percent exposure it is 11×. A ten-month test becomes a nine-year test, and all that changed was where a line of instrumentation sits.

The fix is to fire assignment at the point of exposure, immediately before the surface renders, and to analyze only exposed users. Teams get the control arm wrong: with no new component to render there is no obvious place to log an exposure, so the temptation is to log it earlier. Put an explicit exposure call at the identical code point in both arms, before any branch on the variant. If the definitions differ by one conditional, you have traded a power problem for a bias problem.

The unit you randomize is the sample size you have

In consumer products the unit is a user, and the count on the dashboard is the count in the formula. In business software it usually is not, because everyone inside an account has to see the same interface. Two people on a shared screen seeing different pricing is a support ticket, so the unit becomes the account.

That is a cluster-randomized design, and it carries a penalty most teams never compute. The design effect is 1 + (m−1)ρ, where m is the average users per cluster and ρ is the intraclass correlation. Divide the raw user count by it to get the effective sample size. At eight users per account and an ICC of 0.10 the design effect is 1.7, so 3,200 users behave like 1,880. At fifty users per account it is 5.9, and 3,200 behave like 540.

Do not guess the ICC. Fit a variance-components model by account on last quarter's metric, or resample accounts with replacement a few thousand times and compare that spread to the naive user-level standard error. Then analyze the way you randomized: clustered standard errors, or aggregate each account to a single mean and test the account means. The second is unfashionable and completely correct.

Randomization unit	What it buys	What it costs	Use when
User	Clean independence, simplest analysis, consistent experience	Sample size equals your user count, which is the constraint	Consumer products and any single-seat workflow
Account or workspace	No inconsistency inside a customer, which some changes require	Design effect deflates the sample, often by half or more	Shared surfaces, pricing, anything a customer discusses internally
Session	More units per user, faster accumulation	Same user in both arms, correlated observations, carryover	Genuinely stateless surfaces with no memory across visits
Time block	Every user contributes to both arms, between-user variance gone	Your n is the number of blocks, and carryover contaminates them	System-level policies: ranking, pricing, matching, queueing

Cut the variance instead of waiting for traffic

Variance sits in the numerator and is linear, so a forty percent reduction in variance is a forty percent reduction in the users you need. Nobody gets to double their traffic on demand. Everybody can attack the variance, and it is the most underused half of the formula.

Move the metric closer to the change. If you changed the checkout form, measure checkout completion, not ninety-day retention. The distal metric carries every source of noise between the change and the outcome, and at low volume that noise is the whole budget. Pick the nearest metric a reasonable person accepts as evidence and record the distal one as a guardrail.

Adjust with pre-experiment data. Regression adjustment on a pre-period covariate — published as CUPED by Deng, Xu, Kohavi and Walker in 2013 — cuts variance by the square of the correlation between the pre-period metric and the in-experiment one. A correlation of 0.6 removes thirty-six percent of the variance and thirty-six percent of the required users, turning a ten-month test into a six-and-a-half-month one. It needs history, so it does nothing for a signup test where every user is new.

Cap the tail. Revenue per user and events per session are heavy-tailed, and one outlier can carry more variance than the other 999 users combined. Winsorizing at the 99th percentile usually cuts the variance of the mean substantially; measure it on your own data with a bootstrap. Declare the cap before launch, apply it identically to both arms, and report the uncapped result alongside the capped one.

Stratify. Block the randomization on the one or two pre-period variables that predict the outcome best — plan tier, usage decile, acquisition channel — and use a stratified estimator. At 400 accounts this also protects you from the unlucky split, where one arm gets three of your five largest customers.

Designs that beat parallel A/B at low volume

Parallel A/B is the default because it is easiest to explain, not because it is most efficient. Two alternatives deserve real consideration when traffic is the binding constraint.

Switchback, or time-block randomization. Randomize hours or days into treatment and control rather than users. Every user experiences both conditions, which removes between-user variance entirely, and that is most of the variance. It is the right design whenever the change acts on shared state: ranking weights, pricing rules, matching policy, queue prioritization. Block length must exceed the carryover window, and the analysis happens at the block level, so a two-week switchback with six-hour blocks gives you 56 units.

Interleaving. For ranked lists — search results, recommendations, feeds — merge the output of both rankers into one list shown to one user and measure which side's items earn the clicks. The search-ranking literature reports that this reaches the same confidence with one to two orders of magnitude less traffic, because each user becomes a paired comparison. The limit is scope: it tells you which ranker users prefer, not what happened to revenue.

Design	Best for	Traffic it needs	The failure mode to watch
Parallel A/B	Anything user-scoped, when you can afford it	Highest	Underpowered result read as a null result
Switchback	System-level policy that cannot be split per user	Low in users, high in calendar time	Carryover between blocks, and too few blocks to test
Interleaving	Comparing two rankers or recommenders	Lowest by a wide margin	Answers preference only, and can favor the flashier list
Within-subject crossover	Repeated-use surfaces with no learning effect	Low	Carryover and order effects when users adapt
Non-inferiority	"Did this hurt anything?" rather than "did this win?"	Moderate, and it falls as the margin widens	A margin set so wide the test cannot fail
Bandit allocation	Maximizing outcome across many variants, not measuring one	Moderate	Effect estimates are biased unless you correct for adaptivity

Sequential testing lets you peek legally. It does not create power.

The most common statistical error in small companies is not choosing the wrong test. It is watching a fixed-horizon test on a dashboard and stopping the moment it crosses significance. The cost is published: five interim looks at a nominal five percent level put the true false-positive rate near fourteen percent, and ten looks near nineteen. Across a year, a third of your shipped wins are noise you paid to build.

There are two honest repairs. Group sequential boundaries, usually O'Brien-Fleming, spend almost no error budget on early looks and so cost almost nothing at the final analysis, at the price of an early bar only a large effect crosses. Always-valid methods — the mixture sequential probability ratio test and time-uniform confidence sequences — give an interval correct at every instant, wider at every point in exchange.

Which to pick depends on your organization more than your statistics. If the result is a chart a founder reads every morning, use an always-valid interval; you will not stop people looking, so make looking harmless. If results are reviewed weekly and stopped only for a decisive win, group sequential is more efficient. Either way write the rule down first, because a stopping rule chosen after seeing the data is not one.

Thin traffic makes multiple comparisons worse, not better

When a test is underpowered, the instinct is to compensate by looking at more things: more metrics, more segments, more windows. The arithmetic is brutal. Twenty independent metrics at a five percent level give a sixty-four percent chance of at least one false positive. Six metrics across five segments is thirty tests and a seventy-nine percent chance. Add five interim looks and you are guaranteed something to celebrate.

Segment slicing is the worst offender at low volume because the segments are tiny. A significant result in the enterprise segment computed on forty accounts is a coin flip with a decimal point. If a segment matters enough to decide on, it is a separate experiment with its own power calculation.

The discipline is entirely about what you write down before launch. One primary metric. Two to four guardrails, evaluated as non-inferiority against declared margins rather than two-sided tests hunting for a difference. Everything else labeled exploratory and reported without p-values. If you genuinely have several primaries, control the false discovery rate across the pre-registered set with Benjamini-Hochberg.

One constraint bites only at small scale: concurrency. With 400 accounts, three simultaneous experiments on overlapping surfaces means every arm is contaminated by the others. Either mutually exclude them and accept slower throughput, or accept that you are testing combinations and read the results that way. Both are fine if chosen deliberately.

• Assigning at eligibility instead of exposure. The single most expensive line of instrumentation in the system, and it looks completely correct in code review.
• Reporting a null result from an underpowered test as evidence of no effect. Absence of significance at n=300 is absence of information. Report the interval and let people see what it fails to exclude.
• Randomizing by user and analyzing by session. The analysis unit must match the randomization unit, or the standard errors are fiction and always too small.
• Choosing the primary metric after the results are in. Every experiment has a metric that moved. Picking it afterward guarantees you will find one.
• Running an experiment for five days. Day-of-week effects are large in almost every product. Run in whole weeks, always.
• Ignoring a sample ratio mismatch because the split is "close enough". A 52/48 split on 10,000 users has a p-value near 0.0001. It means the experiment is broken, not lucky.

What to do when the test genuinely cannot run

At this volume, most proposed experiments cannot be powered, and the mature response is to say so and decide anyway rather than to run an underpowered test and dress the output as evidence. There are five honest paths.

Triage by reversibility. If a change is cheap to reverse and the guardrails are instrumented, ship it behind a flag with a rollback trigger and monitor. Spend scarce experimental capacity on decisions that are expensive to unwind: pricing structure, data model changes, anything a customer has already signed.

Bundle deliberately. If you cannot power a test of one change, test five coherent changes as one arm. You learn that the redesign worked, not which part did. That is usually acceptable, because the decision in front of you is whether to ship the redesign. Write down that you traded attribution for power.

Interrupted time series with a comparison series. Ship to everyone, then compare the metric's trajectory against a segment or region that did not get the change, using the pre-period to establish the counterfactual trend. The causal claim is weaker and the confounds are real: seasonality, a campaign, a competitor's move. It is still far better than a before-and-after chart with an arrow on it.

Ask the smaller question. Often the decision is not whether the new version is better but whether it is safe to ship. A non-inferiority test against a generous margin needs far fewer users, because the margin replaces the effect in the denominator. Being able to say the change did not cost more than three percent is frequently the whole decision.

Use qualitative evidence on purpose, and label it. A handful of moderated sessions will surface comprehension and navigation failures no experiment at your volume could reach significance on. That is legitimate evidence that the design works. It is not a causal effect estimate and should never carry a percentage.

The checks that catch a broken experiment before you read the result

Sample ratio mismatch, on every experiment, automatically. Run a chi-square on the observed split against the intended one. A 50/50 design returning 5,200 and 4,800 out of 10,000 gives a chi-square of 16 and a p-value near 0.00006. That is a bug, and the effect estimate is invalid until you find it. Usual causes: bot filtering that catches one arm and not the other, a redirect that fails more often on the variant, an assignment call inside a branch only one arm executes.

Audit the assignment function itself. The standard implementation is a deterministic hash of the user or account identifier concatenated with a per-experiment salt, modulo 1,000. The salt is not optional: reuse one and the same users land in the same bucket in every experiment forever, correlating your results in a way nobody will diagnose. Log the assignment once, then never recompute it at read time.

Run an A/A test before the first real one. Point both arms at identical code and run a full week through your real analysis pipeline. If it produces a significant result on the primary metric, the instrumentation or the analysis is wrong and you found it for the price of a week instead of a quarter. Repeat after any change to assignment or logging.

Check for novelty and primacy. Split the effect by first-week exposure versus later, and by new versus returning users. A change that wins in week one and fades by week three is novelty. One that loses in week one and wins by week three is your users relearning the interface. Short tests are where both distortions do the most damage.

The analysis plan, written before launch

• One primary metric, named, with its exact computation and the exposure definition
• The minimum detectable effect computed from your actual traffic over the planned runtime
• The randomization unit, the analysis unit, and confirmation that they are the same
• Two to four guardrails with non-inferiority margins, not two-sided tests
• The stopping rule: fixed horizon, group sequential boundary, or always-valid interval
• Runtime in whole weeks, with a stated minimum regardless of what the numbers do
• Any capping, winsorizing or stratification, declared with the threshold
• What you will do for each outcome, including the null, written before you know it

A two-week setup that makes the next twelve months cheaper

Two weeks is enough because none of it is large. Hashed assignment and an exposure event are a day of work, and the analysis script is a few hundred lines written once. Off-the-shelf platforms — GrowthBook, Statsig, Eppo, Optimizely, LaunchDarkly and the experiment modules inside the major product-analytics tools — hand you the assignment layer, mutual exclusion and SRM alerting. None of them will decide your minimum detectable effect, choose your randomization unit, or tell you the test you want cannot be powered this year.

Bottom line

Low traffic does not make experimentation impossible. It makes it selective. Compute the minimum detectable effect from the traffic you have and put it above the hypothesis. Test changes big enough to be seen at that volume. Log assignment at exposure. Randomize a unit you can defend and analyze that same unit. Spend the effort on variance reduction rather than on waiting for growth. When nothing can be powered, say so, decide on reversibility, and record the reasoning.

Frequently asked questions