Load is the least interesting way a pipeline breaks
When someone says they want to stress test a pipeline, they usually mean they want to run ten times the normal volume through it and see whether it holds. That test is easy to build, it feels rigorous, and in our experience it finds the smallest share of real problems. Modern storage and query engines absorb a great deal of volume before they complain, and the ones that do complain complain loudly and early, which is the best kind of failure. You will find out.
The failures that hurt are quiet. A duplicate row that inflates a customer-facing count by two percent. A late-arriving record that lands after the window it belonged to has already been aggregated and published. A vendor who starts sending an integer field as a decimal string, which parses fine, rounds differently, and shows up in a monthly total six weeks later. A retry that runs a partially-completed job again from the top and double-writes the half that had already succeeded. None of those are load problems. All of them are shape problems, and shape is what you should be injecting.
There is a second reason volume tests underperform. A ten-times run on an idle staging cluster is not the condition under which your pipeline actually fails. It fails at month end, when the batch job overlaps with the reindex, when the vendor is slow because everyone else is also pulling from them, and when the engineer on duty is the one person who has not read the runbook. Stress in a data system is almost always a coincidence of two ordinary things, not one extraordinary thing.
You are probably here because
- Something in a customer-facing number was wrong and nobody can say for how long
- A backfill took the production pipeline down with it
- Your first enterprise customer sent a security and reliability questionnaire
- You are about to double the number of sources and want to know what will break first
The six-shapes table is the map. The section on recovery time is the part most teams have never measured. If you want to do this yourself, the last two sections are written for that.
Six shapes of stress worth building fixtures for
Each of these is a property of the input, not a property of the load generator. You build a fixture once and run it forever. The cost is a day or two of engineering per shape, and the fixtures outlive several rewrites of the pipeline itself.
| Shape | What you inject | What it usually finds |
|---|---|---|
| Skew | One key holds a disproportionate share of the rows — one customer, one region, one product | A join or partition strategy that works on uniform data and stalls on a hot key; memory that grows with the largest group rather than the average |
| Lateness | Records timestamped hours or days before the batch they arrive in | Windows that have already closed, aggregates that never get corrected, a published number that silently disagrees with its own source |
| Duplication and replay | The same file, message or page delivered twice, and a job re-run from the start after a partial failure | Non-idempotent writes. This is the single most common finding we see, and the most damaging, because the output looks plausible |
| Schema drift | A new column, a removed column, a type widened, an enum with a value nobody has seen before | Silent coercion, dropped fields, and a downstream model or report that changes behavior without anyone shipping code |
| Poison records | Malformed encoding, a null where the contract says never, a value ten thousand times larger than any real one | Whether one bad row stops the whole run, and whether the operator can tell which row it was |
| Slow and flaky dependencies | An upstream that answers in ten seconds instead of two hundred milliseconds, or fails one request in twenty | Timeouts that are longer than the job window, retries with no ceiling, and backpressure that turns into a queue nobody is watching |
If you only ever build two of these, build duplication and schema drift. Duplication is the one that produces wrong answers rather than loud failures, and wrong answers are what damage trust in a data product. Schema drift is the one that arrives on someone else’s schedule, so no amount of care on your side prevents it.
Fixtures beat load generators
A load generator produces more of what you already have. A fixture produces the thing you do not have and are afraid of. The distinction matters because building a good fixture forces you to write down, explicitly, what your pipeline assumes — and half the value of the exercise arrives during that writing, before a single test has run.
A workable fixture set has three tiers. The first is a small, hand-built file of pathological records, checked into the repository, that runs on every commit: the duplicate, the late one, the null, the unicode string in the name field, the negative quantity. It should take seconds. The second is a generated dataset large enough to make the execution plan realistic — often the point where the engine stops keeping a join in memory, which is a property of your engine and worth finding empirically rather than guessing. The third is a replay of a real historical window, warts included, which is the only fixture that contains the failures you have not thought of yet.
That third tier is where most teams stop short, for a good reason: the historical data contains real customer records. The workable compromise is to keep the structure and destroy the content — preserve row counts, key cardinality, null rates, timestamp distributions and arrival order, and replace every value with a synthetic one. That keeps nearly everything that stresses a pipeline, because a pipeline is mostly indifferent to what a string says and very sensitive to how many distinct ones there are.
The two numbers nobody has measured
Almost every team can tell you their pipeline’s throughput. Very few can answer either of the two questions that actually determine what a bad night looks like.
How long does a full recovery take? Not how long the job takes on a good day — how long it takes to get from “we noticed the output was wrong at 09:00” to “the output is correct and we know it is.” That number includes finding the affected range, deciding whether to repair or reprocess, running the reprocess while the normal schedule keeps firing, and verifying the result. Teams who have never measured it usually estimate an hour or two and discover it is most of a day, because reprocessing three days of history through a pipeline sized for one day of daily volume takes three days unless somebody built for that case on purpose.
Is the output correct after recovery, and how would you know? This is the question that separates a pipeline you can operate from one you merely run. If reprocessing the same window twice produces two different answers, you do not have a recovery procedure, you have a coin flip. The test is mechanical: process a window, snapshot the output, process the identical window again into a fresh location, and compare byte for byte or row for row. Any difference is either a genuine bug or a source of nondeterminism you did not know you had — a wall-clock timestamp, an unordered aggregation, a random tie-break, a dependency that changed underneath you.
Both numbers change what you build. A team that knows recovery takes eleven hours builds a way to reprocess a single day in isolation. A team that has never measured it builds a dashboard.
How often each test surfaces something the team did not know — our read
Our judgment from the systems we have worked on, not a survey and not a measurement of yours. The bottom row is the test most teams build first.
Idempotency is the property under test
Strip away the vocabulary and most pipeline reliability work reduces to one question: if this runs twice, is the result the same as if it ran once? Everything about retries, partial failures, backfills and disaster recovery depends on the answer being yes, and the answer is usually no by default, because the default way to write data is to append it.
The mechanics are worth stating plainly. Writes should be keyed, so reprocessing replaces rather than adds — an upsert on a natural key, a partition overwrite, or a write to a new location followed by an atomic pointer swap. Any step with a side effect outside the data store — an email, a webhook, a payment, a message onto a queue another team consumes — needs a deduplication key and a record of what was already emitted, because rewriting a table cannot undo it. And every output should be traceable to the input batch that produced it, or every fix becomes a full reprocess.
The test for all of this is the one above: run it twice, compare. It finds more than any amount of reading the code, because nondeterminism hides in dependencies rather than in logic — a sort that was stable in one engine version and not the next, a deduplication that kept “the first row seen” in a parallel read with no defined order, a currency conversion that fetched today’s rate during a backfill of last quarter.
Backpressure, and where the queue really is
When something downstream slows down, the work does not disappear. It accumulates somewhere, and the health of your system depends almost entirely on whether that somewhere is a place you chose. In a well-built system the queue is explicit, bounded, monitored, and has a defined behavior when it fills. In most systems the queue is an accident — unread messages in a broker with a retention window nobody has checked, rows in a staging table that grows without a cleanup job, files in object storage that the next run will try to process all at once.
The test is simple and rarely run: hold the downstream still for an hour while the upstream keeps producing, then let go. Three things are worth watching. Where did the work pile up, and did anything alert. Did the system survive the release — a pipeline that handles steady state fine can fall over on the recovery surge, because it now sees an hour of input in one batch. And did anything get dropped silently, which is the outcome that turns an availability incident into a correctness incident and is the reason this test is worth the trouble.
Retry policy is part of the same picture. Unbounded retries against a struggling dependency are how a slow upstream becomes a dead one, and how your incident becomes theirs. Ceilings, jitter, and a rule for when to stop and page a human are cheap to add and almost never present in a first version.
Testing against something you do not control
Some of the most consequential stress comes from outside. A data vendor changes a field. An API begins rate limiting at a lower threshold than its documentation states. A partner’s nightly export lands two hours later than usual, every day, for a month, and nobody tells you.
Two habits handle most of it. First, put a contract at the boundary — the columns, types, nullability, key uniqueness and expected row-count range you require from each source, checked on arrival, before anything downstream runs. When a source violates it you find out with a clear message, rather than three transformations later as a strange number. Second, record the raw payload exactly as received, before any parsing, and keep it as long as your storage budget allows. That archive is what lets you reprocess after a parser fix, and what lets you show a discrepancy originated upstream. Both are ordinary practice, both are skipped under deadline, and both are much harder to add later.
For the dependency itself, build a stand-in you can make behave badly on demand — slow, flaky, rate-limited, returning yesterday’s data, returning an empty result that is technically valid. That last one deserves attention. An upstream that returns zero rows without erroring is among the more dangerous inputs a pipeline can receive, because a pipeline that faithfully processes zero rows will overwrite a good table with an empty one. A minimum-row-count check at the boundary costs one line.
What belongs in continuous integration and what belongs on a calendar
Not all of this can run on every commit, and pretending otherwise is how a test suite becomes something people skip. A reasonable split:
- Every commit — the pathological fixture file, boundary contract checks, and the double-run determinism test on a small window. Seconds to a couple of minutes
- Nightly — a realistic-scale run with skew and lateness injected, and the same determinism comparison on a full day of data
- Before any change to a write path — the duplication and replay suite, deliberately, because that is the code where non-idempotency is introduced
- Quarterly, as an exercise — a full recovery drill with a stopwatch: corrupt a window in a copy of production, and have someone who did not build the pipeline restore it using only the runbook
The quarterly drill is the one to defend when the calendar gets tight. It is the only test that measures the runbook and the people rather than the code, and those determine what a bad night costs. It also tends to produce the quarter’s most useful engineering work, because it turns a vague sense that recovery would be painful into a specific list of the four things that made it painful.
Common mistakes
- Testing volume and calling it stress testing — the one shape least likely to be how you fail
- No determinism test, so nobody knows whether reprocessing is safe until the night they need it
- Test data that is a clean sample of production, which by construction excludes every row that would have broken something
- Validation after the transformation instead of at the boundary, so failures are reported in terms of your schema rather than theirs
- No archive of the raw payload, making reprocessing after a parser fix impossible
- An implicit queue — unbounded, unmonitored, discovered during the incident
- Alerting on job success rather than on output, so a run that succeeds and produces an empty table is green
- A runbook nobody has executed, which is a document rather than a procedure
Where you do not need us
Most of this you should do yourself, and it is not a large amount of work. The pathological fixture file is an afternoon. Boundary contracts on your three most important sources are a day or two. The double-run determinism comparison is a script and a diff, and it is the highest-value hour on this page. If you do those three things and nothing else, you will have removed the majority of the incidents in this article without anyone external involved.
The places where a second pair of hands genuinely helps are narrower. Rebuilding a pipeline that is not idempotent into one that is, when it is already carrying production traffic, is delicate and hard to do in the gaps between feature work. Building a faithful anonymized replay of real historical data takes a specific kind of care about what to preserve. And the quarterly recovery drill benefits from someone who did not build the system, because the whole point is to test the runbook against a person who has to actually read it. Those are real engagements. The fixture file is not, and we would rather tell you that than quote it.
Bottom line
Pipelines fail on the shape of their input, not its size. Build fixtures for skew, lateness, duplication, schema drift, poison records and slow dependencies, and put contracts at every boundary you do not control. Measure two numbers you probably have never measured: how long a full recovery takes end to end, and whether the output is identical when you process the same window twice. Then run a real recovery drill with a stopwatch and someone who has to read the runbook. Everything else on this page is detail; those are the parts that change what a bad night costs.
Frequently asked questions
Enough to cross the thresholds your engine cares about — the point where a join stops fitting in memory, where a partition splits, where a scan stops being trivial. That point is a property of your engine and configuration, not a universal number, and the honest way to find it is to increase the fixture size until the execution plan changes. Above that threshold, more data mostly buys confidence rather than information.
Structure yes, content usually not. What stresses a pipeline is row counts, key cardinality, null rates, timestamp distribution and arrival order — and all of those can be preserved while every actual value is replaced with a synthetic one. The result is a fixture that reproduces the hard cases without carrying real records into a test environment.
Process the same input window twice into two separate locations and compare the outputs row for row. It is a script and a diff. If they differ, you have found either a bug or a source of nondeterminism, and either way you have learned something that affects every retry, backfill and recovery you will ever run.
Related but not identical. Chaos work in a service tests availability — can the system keep answering. In a data system the more important question is correctness after the disruption, because a pipeline that recovers availability while quietly double-counting a day has failed worse than one that stayed down. Inject failures the same way, but make the assertion about the output, not the uptime.
Quarterly is a reasonable default, and after any significant change to the write path or the storage layout. The value comes from the constraints: a stopwatch, a real corrupted window in a copy of production, and an engineer who did not build the pipeline working only from the runbook. Anything looser tests the code rather than the procedure, and the procedure is what fails at three in the morning.