The space was always bigger than the budget
Teams tend to discover this problem the day they ship something with a language model in it, and conclude that generative AI broke testing. It did not. Testing was never exhaustive. A function taking two 64-bit integers has more inputs than there are atoms in a small planet. A service with thirty independent boolean feature flags has about 1.07 billion configurations. A workflow engine with retries and timeouts has an interleaving space that grows factorially in the number of concurrent steps. Free-text input is unbounded outright. What changed with AI is not the size of the space but the disappearance of the comfortable illusion that a few hundred hand-written cases were sampling it well.
So the honest framing is this: you will run a tiny, hand-picked subset of the possible executions, then ship. Test strategy is the work of making that subset less biased, making each execution check more, and building the system so the executions you never ran fail loudly and cheaply rather than silently and expensively. That is a design problem as much as a QA problem, which is why the strongest testing work changes the code, not just the test folder.
You are probably here because
- Coverage is high, the suite is green, and defects still reach customers.
- A test fails once every few hundred runs, nobody can reproduce it, and the team reruns it until it passes.
- Someone asked how many examples are enough to sign off on a feature whose output is not deterministic, and nobody in the room had a number.
- For a lot of what you ship, nobody can actually state what the correct output is.
All four are the same fact wearing different clothes — the suite runs a tiny hand-picked subset of the possible executions, and each run checks very little — and the sections below on the oracle, on properties and metamorphic relations, on making a rare schedule reproducible, and on what a clean run actually proves take them one at a time.
Count the space before anyone argues about coverage
Start every test-strategy conversation with arithmetic, because the arithmetic ends most of the arguing. Write down the dimensions of the input space and multiply. A checkout service that takes 8 payment methods, 40 currencies, 6 tax jurisdictions, 4 fulfillment types, and a cart of up to 50 line items does not have "a lot" of cases. It has a number, and the number is large enough that nobody will propose enumerating it once they see it.
Then split the space into three buckets, because each takes a different technique. Enumerable: dimensions small enough to cross-product completely, like payment method by fulfillment type. Test all of it in a loop and stop discussing it. Combinatorially large but structured: configuration flags, feature toggles, schema variants, API parameter combinations. Covering arrays earn their keep here, because a pairwise covering array over thirty binary flags needs fewer than a dozen runs to exercise every pair, and most configuration bugs are pair interactions rather than exotic six-way ones. Effectively infinite: free text, arbitrary binary payloads, timing, network fault sequences, adversarial users. Nothing enumerates this bucket. It gets properties, generators and statistics.
Most teams handle the first bucket adequately and have never separated the other two. They apply hand-written example tests to all three, which works for the first, wastes effort on the second, and does nothing for the third.
Coverage tells you what you ran, not what you checked
Line and branch coverage measure execution, not verification. A test that calls a function and asserts nothing raises coverage exactly as much as one that checks the answer. We have reviewed suites at 85% line coverage where much of the suite asserted only that the call did not raise. Coverage was doing what it was asked to do and telling the team nothing.
Two refinements make the metric mean something again. The first is stricter criteria: branch coverage instead of line, and modified condition/decision coverage for compound conditions, which requires each condition to be shown independently affecting the outcome. SQLite publishes that its suite reaches 100% MC/DC branch coverage and contains hundreds of times more code than the library itself. Worth internalizing not because you should match it, but because it calibrates what "well tested" costs for software that must not be wrong.
The second refinement is mutation testing, the only widely available technique that measures whether your assertions are load-bearing. It changes an operator, flips a boundary, deletes a statement, then reruns the suite. If the suite still passes, that mutant survived and your tests do not constrain that behavior. Stryker, PIT, mutmut and cargo-mutants make this routine. Run it on the twenty files that would hurt most if they were wrong, not the whole repository. A first mutation run on a mature codebase is a bad afternoon, and it is the cheapest bad afternoon available.
The real bottleneck is the oracle
Generating a million inputs is easy. Knowing whether the output was correct for each one is the whole difficulty, and the literature has a name for it: the oracle problem. Hand-written cases hide it because a human silently supplied the expected answer with the input. Once inputs are generated that human is gone, and you need a mechanical way to tell right from wrong.
Every scalable technique is really a strategy for obtaining an oracle cheaply. Seen that way, choosing one stops being a matter of fashion: you pick the strongest oracle you can afford for that behavior.
| Oracle | What it requires | Strength | Where it breaks down |
|---|---|---|---|
| Exact expected value | A human who knows the answer for this input | Total, for that one case | Does not scale past a few hundred cases |
| Reference implementation | A second implementation, or the old one you are replacing | Very strong; finds disagreement anywhere | Shared assumptions produce identical wrong answers |
| Invariant / property | A statement true of every valid output | Runs on unlimited generated input | Passes systems that are consistent and useless |
| Metamorphic relation | A rule linking one output to another | Works when nobody knows the right answer | Only checks relative behavior, never absolute |
| Statistical bound | A labeled sample and an accepted error rate | The only honest oracle for probabilistic output | Costs labeling, and bounds the aggregate not the case |
| Crash / sanitizer | Nothing beyond instrumentation | Free, and runs on any input at all | Catches only the failures that announce themselves |
The crash oracle is the reason fuzzing works at all. It requires no knowledge of correctness, which is why it scales to billions of executions. Address sanitizer, thread sanitizer and a plain assertion all convert silent corruption into a loud stop, and every assertion you add widens the class of bugs a random input can reveal.
Properties: write the rule once, run it a million times
A property test states something that must be true of every valid output, then lets a generator attack it. It is the highest-return change most teams can make, and the tooling is mature: Hypothesis in Python, fast-check in TypeScript, jqwik in Java, proptest in Rust, QuickCheck where it started.
The properties that find real defects are duller than people expect. Round-tripping: parse then serialize returns the original bytes. Idempotence: applying the operation twice equals applying it once. Commutativity where the domain claims it: two discounts in either order produce the same total. Conservation: allocated shards sum to the input quantity, including when it is zero, negative, or a decimal that does not divide evenly. Preservation: a filter never returns an element that was not in the input.
The detail that makes this practical rather than annoying is shrinking. When a generator finds a failure with a 40-element cart of random unicode product names, the framework reduces it to the minimal failing case, usually two items and an empty string. You debug the shrunk case, not the random one, and it is often a test you would have written by hand if you had thought of it. Save it: every framework has a regressions file for exactly this, and each line in it is a real defect the system once had.
Two practices keep property tests from decaying. Pin the seed corpus so a fixed failing case is never lost to a random seed. And separate the fast run from the deep one: a few hundred examples per property on every pull request, a long soak with a fresh seed nightly. A suite that only ever runs the same hundred examples has quietly become a slow example suite.
Metamorphic relations: checking the shape of the answer
The hardest cases are the ones where nobody can state the correct output at all. What is the right ranking for a query, the correct summary of a document, the optimal route for 400 stops. There is no expected value to assert against, and two human reviewers will disagree.
Metamorphic testing sidesteps this by checking relationships between executions instead of absolute outputs. You do not need the right answer. You need to know how the answer must change when the input changes in a controlled way. Compiler testing has run on this for years: Csmith generated random C programs and found more than 300 bugs in production compilers, and equivalence-modulo-inputs techniques mutate a program in ways that cannot change its output, then check the compiler agrees.
Search and ranking. Adding a document that matches nothing must not change the top result. Duplicating a document must not push an unrelated one off the first page. Rewriting a query with a synonym should move results a bounded amount, not reorder them entirely.
Classifiers and extraction. Changing a customer name, reformatting a date, or adding trailing whitespace must not change the predicted class. If it does, the model is keying on something you did not intend, and that is worth knowing before a customer finds it.
Optimizers and planners. Adding a stop can never reduce total cost. Relaxing a constraint can never make the objective worse. Scaling every distance by two must scale the optimal cost by two and leave the route order unchanged.
Aggregation and reporting. The sum of the per-region totals equals the global total. Filtering to a date range then aggregating equals aggregating then filtering. Reordering the input rows changes nothing.
These relations are cheap to write, run against generated input forever, and catch what example tests structurally cannot: behavior that is wrong in a way nobody had an expected value for.
Assertions in production are part of the test strategy
Every invariant you can check at runtime turns an untested execution into a caught one. Check the conservation rule after the allocation, the schema after the parse, the balance after the transfer. Fail closed on the ones that indicate corruption, log and alert on the ones that indicate drift, and sample expensive checks at one in a thousand requests so the cost is bounded. This is how you get coverage of inputs you never generated: real traffic runs the assertions for you, on the exact distribution that matters, and the untested part of the space stops being invisible.
Fuzzing, and why the corpus is the real asset
Coverage-guided fuzzing is the most automated form of this discipline. AFL++, libFuzzer, Atheris, Go's native fuzzing and cargo-fuzz mutate inputs, watch which paths execute, and keep the mutations that reach new code. Left running, the fuzzer builds a corpus that reaches deep into your parser without anyone writing a test. Continuous fuzzing infrastructure for open-source projects has found tens of thousands of defects on exactly this principle.
Anything that consumes bytes from outside your process deserves a fuzz target: document parsers, protocol decoders, deserialization, template rendering, query builders, regex over user input, image and archive handling, webhook payloads. Structure-aware fuzzing extends this to inputs with a grammar, generating valid-shaped JSON, protobuf or SQL rather than noise, which gets past front-door validation and into the logic where the interesting bugs are. Schemathesis does the same against an OpenAPI specification.
The operational mistakes are consistent. Teams run a fuzzer for an afternoon, find nothing, and conclude the code is clean, when the interesting finds arrive hours in. They also throw the corpus away, which is the real loss: the corpus is an accumulated map of how to reach your own code. Commit it, keep crash inputs as permanent regressions, run a bounded fuzz job on every change, and run the long soak on a schedule.
Concurrency: make the schedule a variable you control
Concurrency bugs are the sharpest version of this problem. The interleaving space is enormous, the failing schedule is rare, and the failure is not reproducible, so the standard debugging loop does not exist. A test that passes 999 times out of 1,000 is not a passing test. It is a failing test with a bad reporting mechanism.
The technique that works is deterministic simulation. Run the whole system single-threaded under a scheduler you control, driven by a seeded pseudorandom generator, with time, network, disk and thread scheduling replaced by simulated implementations. A run becomes a pure function of its seed. The simulator injects what production will eventually deliver anyway: packet reordering, one-way partitions, clock skew, disk writes that succeed then vanish, processes that pause for eight seconds at the worst moment. When a seed fails it fails identically every time, and you have a debugging loop.
FoundationDB is the canonical case: the team built the simulator before the database, and correctness work happens there rather than in production incidents. The pattern has spread. TigerBeetle runs a similar simulator, madsim and turmoil bring deterministic networking to Rust and Tokio services, and Antithesis productized the idea at the hypervisor level so existing systems get determinism without a rewrite.
Model checking sits one level above. TLA+, Alloy and P check a design rather than an implementation, exhaustively exploring a small model instead of sampling a large one. The published account of formal methods at Amazon Web Services is the standard reference: engineers found design defects in replication and storage protocols they judged testing would not have found, because the failing sequences were too specific to reach by chance. Specify the protocol, not the code, and only where a subtle race would be unrecoverable.
| Technique | Failure class it finds | Setup cost | Where it runs |
|---|---|---|---|
| Property tests | Logic errors on unusual but valid input | Low; days | Every pull request |
| Metamorphic relations | Wrong behavior with no known correct answer | Low; days | Every pull request, plus nightly soak |
| Coverage-guided fuzzing | Crashes, memory errors, parser and input handling | Medium; a week per surface | Bounded in CI, long soak nightly |
| Deterministic simulation | Races, partial failure, recovery and retry paths | High; weeks, and it constrains the design | Continuous, thousands of seeds |
| Model checking | Protocol and design defects before code exists | High; specialist skill | Once per protocol, revisited on change |
| Fault injection in production | Operational assumptions that were never true | Medium; needs mature observability | Scheduled, with a blast-radius limit |
Fault injection against a live environment is the last row for a reason. Chaos experiments are valuable and are not a substitute for the rows above: a production experiment samples the schedule space a few times a day, while a simulator samples it thousands of times an hour at no risk to a customer. Run the simulator to find the bug, the production experiment to verify the response.
What a clean run actually proves
When a component is probabilistic, and every system with a model in it is, the acceptance question changes shape. You are not asking whether it is correct. You are estimating a failure rate from a sample, and there is a number you should know.
If you run n independent trials and observe zero failures, the approximate 95% upper confidence bound on the true failure rate is 3/n. That is the rule of three, and it is the fastest way to end an unproductive meeting. Twenty prompts with no bad answers bounds the failure rate at roughly 15%, which is not an acceptance result for anything a customer touches. The arithmetic is unforgiving in a useful direction.
Zero failures observed — 95% upper bound on the true failure rate
Rule of three: with zero observed failures in n independent trials, the 95% upper bound is approximately 3/n. Bar length shows confidence gained, not the rate.
The companion arithmetic covers the case where failures do occur. For a proportion near the middle of the range, the half-width of a 95% interval is roughly 1/√n. Twenty samples give plus or minus 22 points, which cannot distinguish a good system from a mediocre one. Four hundred give plus or minus 5. To detect a one-point regression between releases the sample runs into the thousands, and no amount of enthusiasm changes that. Decide the precision first, then size the labeled set, then budget the labeling. The other order produces a number that cannot answer the question it was collected for.
Precision of a measured rate by labeled sample size
Approximate half-width in percentage points of a 95% interval for a proportion near 50%. Use a Wilson interval, not the textbook normal approximation, when the rate is near 0 or 1.
Send it over and we will tell you what we would change.
Email your coverage report and the names of the three files you would least like to be wrong in to contact@precisionfederal.com. You get back a short written note naming the three things we would change and why. One business day. No charge, no meeting, no deck.
contact@precisionfederal.comSample on purpose, not uniformly
Uniform random sampling of production traffic measures the average case, and the average case is not what hurts you. The failures that produce an incident live in slices that are rare by volume and expensive by consequence: the largest tenant, the one region with a different tax rule, the document type that appears twice a month, the retry path, the first request after a deploy, the partner who has never once sent a valid payload.
Stratify instead. Define the slices that matter, sample each to the precision that slice deserves, then reweight to recover an overall estimate. Rare-but-costly slices get oversampled deliberately, and the reweighting keeps the headline number honest. That is the difference between a suite reporting 97% and missing the thing that takes you down, and one reporting 97% while also telling you the enterprise slice is at 84%.
Two additions make the set durable. Keep a permanent regression set of every input that has ever caused an incident, with the correct output recorded at the time. And refresh the sample on a schedule, because input distributions move, and a set assembled eighteen months ago measures traffic the system no longer receives.
Where to spend a fixed budget
If a team has a quarter of engineering time to spend on testing a system with an unbounded input space, this is the split we start from. Adjust it for your risk profile, but adjust it explicitly and before the work starts, not by drifting toward whichever activity feels most productive on a Tuesday.
Default test-budget split — unbounded input space
Starting weights summing to 100. Set them before the work begins, not after the first interesting bug.
Runtime assertions get a fifth of the budget because they are the only line that keeps testing after you ship, on the real input distribution. Deterministic simulation is expensive enough to earn its place only where partial failure and recovery are core behavior: a ledger, a scheduler, a replication protocol, anything holding a lock.
Contain what you could not test
The strategic move that most changes outcomes is not a test at all. It is reducing the size of the space that needs testing, and bounding the damage in the part you never reach.
Delete configuration. Every flag doubles the space. A flag that has been true in every environment for a year is not a flag, it is a constant with extra test cases attached. Removing ten dead flags removes a factor of a thousand from the configuration space, and that is a real reduction, not a bookkeeping one.
Make illegal states unrepresentable. A type that cannot hold a negative quantity needs no test for negative quantities. Parse input into a validated type at the boundary once, and the interior of the system stops having a case analysis at all. This is the cheapest coverage available and it is a code change, not a test change.
Narrow the blast radius. Progressive rollout, per-tenant kill switches, circuit breakers, idempotent writes so a retry cannot double-charge. An untested path that fails for 1% of traffic for four minutes and rolls back on its own was an event. The same bug for everyone for six hours was an incident.
Reconcile after the fact. For anything involving money, inventory or state that must balance, a job that recomputes the invariant end to end and alerts on disagreement catches the class of bug that no unit test was ever going to reach, because the bug required a specific sequence of real events over three days.
How this usually fails
- Chasing a coverage percentage with tests that execute code and assert nothing meaningful about it.
- Writing property tests that restate the implementation, so the test and the code are wrong in the same way.
- Running a fuzzer for one afternoon, finding nothing, and deleting the target.
- Throwing away the fuzz corpus and the shrunk failing cases, which is throwing away the only durable output.
- Accepting a probabilistic component on twenty examples, then treating the resulting number as a measurement.
- Retrying flaky tests until they pass instead of making the schedule deterministic, which trains everyone to ignore real failures.
- Sampling production traffic uniformly, so the expensive slices are represented by two examples each.
- Building a test suite around a system that could have had ten fewer configuration flags.
What the suite should contain
- A written count of the input space, split into enumerable, combinatorial and unbounded
- A named oracle for every behavior that matters, and a deletion decision for any behavior with none
- Property tests on core logic, with the seed corpus and regressions file in version control
- Metamorphic relations wherever the correct output cannot be stated
- A fuzz target on every surface that consumes bytes from outside the process
- Deterministic, seed-replayable tests for concurrency, failure and retry paths
- A stratified labeled sample sized to the precision the decision requires
- Runtime invariants that fail closed on corruption and alert on drift
- A permanent regression entry for every input that has ever caused an incident
A four-week way to get there
Hardening Sprint
Four weeks is enough because no step requires a rewrite and each leaves something that keeps working afterward: the mutation run shows where the suite is decorative, the properties run forever on new input, the corpus grows on its own, and the runtime invariants take over the part of the space no offline suite was going to reach.
Common objections
Our team barely has time for the unit tests we have.
Then start with the one property test and the one runtime assertion on the path where an error costs the most, and delete the example tests that assert nothing. This is usually net negative work in the first week. A property test replaces a dozen hand-written cases and finds inputs nobody would have chosen, and a suite gets faster when the decorative tests come out of it.
We are shipping a wrapper around a model API. Does any of this apply?
All of it applies, and the arithmetic section applies hardest. Your deterministic code still needs properties, your prompt-assembly and parsing code still needs fuzzing, your retry and timeout handling still needs a deterministic simulator, and your model output needs a stratified labeled sample with a stated acceptance bound. The model is one component in a system that is mostly ordinary software.
Bottom line
Exhaustive testing was never available for any system anybody wanted to build, so coverage of the space was never the goal. The goal is a small number of executions that each check a great deal, an honest statistical statement about whatever stays probabilistic, and a system designed so the untested remainder fails in ways that are bounded, visible and reversible. Count the space, name the oracle, write the properties, keep the corpus, bound the blast radius. Then ship, knowing what you do and do not know.
Frequently asked questions
Stop enumerating cases and start specifying rules. Write properties that must hold for every valid output, add metamorphic relations where no correct answer can be stated, generate input against both, and fuzz anything that consumes bytes from outside the process. Then add runtime assertions so real traffic keeps testing the system after release.
Useful as a floor, misleading as a target. Coverage measures which lines executed, not whether anything was verified. Use branch coverage rather than line coverage, and mutation testing on your highest-risk files to find out whether the assertions constrain behavior. Surviving mutants tell you more than an uncovered line.
Work backward from the precision the decision needs. With zero failures in n trials, the 95% upper bound on the failure rate is about 3/n, so 300 clean runs bound it at 1% and 20 clean runs bound it only at 15%. If you need to detect a one-point change between releases, the labeled sample runs into the thousands. Size the set before you collect it.
Running the system single-threaded under a scheduler you control, with time, network, disk and thread interleaving driven by a seeded random number generator. A run becomes a pure function of its seed, so a rare concurrency failure reproduces exactly instead of appearing once a month. It is how stateful systems test partial failure and recovery at a rate production experiments cannot approach.
Use it whenever you cannot state the correct output but you can state how the output must change when the input changes in a controlled way. Adding an irrelevant document must not change the top search result. Adding a stop cannot reduce total route cost. Reformatting a date must not change a predicted class. These relations run on generated input forever and find defects that example-based tests structurally cannot.