The count is not what breaks you
Ask a team how big their file pipeline is and you get a count: two million documents, four hundred thousand images, sixty thousand uploads a day. The count is the least informative number available. What determines whether the system works is the distribution, and the distributions in real corpora are brutally long-tailed. A document set with a median file of 180 kilobytes routinely has a 99th percentile above 200 megabytes and a maximum in the several-gigabyte range, because somebody scanned a twelve-year archive into one file. Capacity planned on the average is capacity planned for a workload that does not exist.
The same shape shows up in processing time. A pipeline where the median file finishes in 900 milliseconds will have a tail running past ten minutes, and the tail is not evenly distributed across the day — it arrives in clusters, because the customer with the huge files uploads them all at once. Any design that assumes a roughly uniform cost per item will be fine in testing and will stall in production behind one item.
So the first thing to do on any file pipeline, before architecture, is to measure the distribution you actually have: size at p50, p95, p99 and max; page or record count where the format has one; format mix; and the share that are duplicates. Half an hour of counting reshapes most of the decisions below.
You are probably here because
- One 900MB upload took down the workers and the backlog took six hours to drain
- A batch of 40,000 files reported success and 700 of them are not in the output
- You changed the parser and now nobody can say what it would cost to run the archive again
- Memory use is fine at p50 and the pods are being killed anyway
These are four symptoms of the same missing pieces: a registry that knows every file’s state, per-file resource limits, a retry unit smaller than the file, and reprocessing treated as a first-class operation.
Uploads should never pass through your application
The single most common structural mistake is routing file bytes through the API server. It couples upload bandwidth to request-handling capacity, it puts a large buffer in a process tuned for small ones, and it means a slow client on hotel wifi occupies a worker for four minutes.
Issue a short-lived pre-signed URL and have the client write directly to object storage. For anything over roughly 100 megabytes, use the multipart upload the storage service already provides, which gives you resumability and parallel parts for free. The application's job is to authorise the upload, record that it was authorised, and react when the object appears. That is three small operations instead of one large one.
Two details are worth getting right at the start. Set a maximum size in the signed policy itself, not in a check afterwards, so an oversized upload is refused by the storage service rather than accepted and then rejected. And do not trust the client's declared content type — it is a hint from an untrusted party, and it is wrong often enough that it should never gate anything.
Every file gets a row before any byte is processed
The registry is the backbone. One row per file, created the moment the file is known to exist, carrying: an internal identifier, the storage location, the byte size, the content hash, the declared and detected formats, the source and who supplied it, the arrival time, the current state, the attempt count, and the last error.
This sounds like bookkeeping and it is the thing that separates a pipeline you can operate from one you cannot. Without it, “did file X get processed” is answered by looking in the output and inferring. With it, it is a lookup. More importantly, the registry is what lets you reconcile: the count of files in state complete plus failed plus in progress must equal the count received, and any drift is a bug you can see the same day rather than a discrepancy someone finds in a quarter.
Model state explicitly and include the states people forget: received, validated, rejected, processing, partial, complete, failed, quarantined, superseded. The last one matters when a corrected version of the same document arrives, which in any pipeline fed by humans it will.
Content addressing pays for itself in the first month
Hash every file on arrival, store it under the hash, and keep the human-facing name as metadata. SHA-256 over a gigabyte costs a couple of seconds of CPU and buys three things at once.
Deduplication. Real corpora are full of duplicates — the same attachment forwarded through five mailboxes, the same statement uploaded twice because the first attempt looked like it failed. In the document sets we have worked with, exact-duplicate rates in the ten to thirty percent band are ordinary, and the highest we have measured was a shared drive where nearly half the files existed more than once. Every duplicate you detect is processing you do not pay for.
Integrity. Bytes get truncated in transit, in copies, in restores. If the stored hash and the recomputed hash disagree, you know before you build an answer on corrupt input.
Cache keys for derived work. Extraction output, thumbnails, embeddings, parsed text — key each on the tuple of content hash and processing version. Then rerunning an unchanged file with an unchanged parser costs a lookup, and the reprocessing question further down becomes tractable instead of terrifying.
Sniff the format; the extension is a rumour
Never dispatch on the filename extension or the declared content type. Read the leading bytes and detect the real format. In any pipeline fed by real users, a meaningful fraction of files are mislabelled — spreadsheets saved as .csv that are actually the vendor's XML dialect, .pdf files that are actually images renamed, .txt in an encoding nobody declared, archives inside archives.
Detection has to go beyond the magic bytes for the formats that matter. A PDF can be a text document, a scan of a text document, a mix of both, or a container of embedded images with a text layer that is machine-generated and wrong. Those need four different downstream paths, and the only way to tell them apart is to look: extract the text layer, measure how much there is per page, and route the pages with almost none of it to the image path. A pipeline that assumes every PDF has usable text produces empty output for scanned documents and reports success, which is the worst available outcome.
| Input | What goes wrong | Defence |
|---|---|---|
| Highly compressed archive | A few hundred kilobytes expands to hundreds of gigabytes and fills the disk | Cap expanded size and entry count; stream and abort past the cap |
| Very large page count | A 40,000-page document exhausts memory in a whole-file parser | Stream page by page; hard cap with a routed exception path |
| Encrypted or password-protected | The library blocks or the file silently yields nothing | Detect at validation, mark rejected with a reason a person can act on |
| Malformed or truncated | Parser loops, or returns partial content with no error | Wall-clock timeout per file; validate output is non-empty and plausible |
| Deeply nested containers | Archive within archive within archive, unbounded recursion | Depth limit of two or three, then quarantine |
| Scanned image with no text layer | Text extraction returns empty; the pipeline records success | Measure characters per page and route below a threshold |
Treat every file as hostile, because some are
You are running third-party parsers over content you did not create, and those parsers are large C and C++ codebases with a long history of memory-safety issues. Even without an attacker, a badly formed file will find the pathological path in a parser eventually.
The practical defence is boring and effective: run parsing in a separate process from your service, with a hard memory limit, a wall-clock timeout, no network access, and a writable temp directory that is discarded afterwards. Cap concurrency by memory rather than by core count — the mistake we see most is a worker sized at one process per core on a 64-core machine, where each process can legitimately need three gigabytes for a large document, and the box has 128 gigabytes. Twenty-two of those processes fit. Sixty-four do not, and the kernel decides which ones die.
If your files come from outside your organisation, add malware scanning before anything else touches them, and quarantine rather than delete so a false positive is recoverable.
Where file pipelines actually fail — our ranking by frequency
Judgment from the systems we are asked to look at, not a survey. Raw speed is last because it is almost never the binding constraint.
Pick the retry unit smaller than the file
If the unit of work is a whole document, then a failure on page 380 of 400 throws away all 400 pages of work. At volume that is the difference between a pipeline that finishes and one that thrashes.
Split at the natural sub-unit — page, sheet, record, frame — as the first step, write each sub-unit's result independently, and retry at that grain. A document becomes a parent row plus N child rows, and completion is a function of the children. This costs some orchestration and buys three things: bounded retry cost, natural parallelism across a single large item, and the ability to report that a document is 97 percent complete with three pages that need a human.
There is a real trade-off and it should be stated. Splitting adds per-item overhead, so for small files it is a net loss. Our rule is to split when the item exceeds a threshold — a few dozen pages, or a size where a single failure costs more than a minute of work — and process smaller ones whole.
Partial success is a state, not an error
Run ten thousand files and you will not get ten thousand successes. Ever. Somewhere between one and five percent will fail on any real corpus, and the reasons will be legitimate: corrupt file, unsupported format, an encrypted document, a scan too poor to read, a page that is genuinely blank.
Design the reporting for that, because the alternative is a system that either lies or throws away good work. A batch result should carry counts by outcome, a list of the failures with a reason code each, and a way to act on them. A binary success flag over ten thousand files conveys almost nothing and will be false every time.
Then build the exception queue as a real product surface rather than a log file. Someone has to look at the three percent, and if the only way to do that is to run a query and open files by hand, nobody does, and the failures accumulate silently until somebody discovers a two-year hole. A simple review screen with the file, the reason, and buttons for retry, reclassify and reject is a few days of work and it is what makes the failure path actually operate.
Send us the size histogram.
Email file counts by format, sizes at p50 / p95 / p99 / max, page counts where they apply, and what currently fails to contact@precisionfederal.com. You get back a short written note on where this pipeline will break first and what the cheapest fix is. One business day. No charge, no meeting, no deck.
contact@precisionfederal.comYou will run the whole thing again
This is the requirement teams discover late and it changes the architecture, so it belongs in the first design conversation. You will improve the parser. You will change the extraction model. You will find a bug that affected eight months of output. And then somebody will ask what it costs to reprocess four million files, and the answer will determine whether the improvement ships.
Three decisions make reprocessing cheap. Keep the raw bytes forever and treat everything derived as disposable. The original file is the only thing you cannot regenerate; text, embeddings, thumbnails, structured output are all functions of it. Teams that discard originals after extraction to save storage are trading a small recurring cost for a permanent ceiling on how much they can improve.
Version every processing stage and record the version on every derived artifact. Then “which files were processed with the parser that had the bug” is a query, and reprocessing is scoped rather than total. This one field turns a four-million-file problem into a three-hundred-thousand-file problem more often than not.
Make reprocessing the same code path as first processing, running at low priority. A separate backfill script is a second implementation that drifts from the first and produces subtly different output, which is how a corpus ends up with two incompatible generations of records in it. Same code, different queue, throttled so it does not starve live traffic.
Storage economics, briefly and honestly
Object storage at standard tiers sits in the low twenties of dollars per terabyte per month at list price, and infrequent-access and archive tiers drop that by roughly half and by an order of magnitude respectively, at the cost of retrieval fees and latency. For raw originals that must be kept but are rarely read, lifecycle rules moving to colder tiers after thirty to ninety days are close to free money.
The costs that surprise people are not the bytes. They are per-request charges when you list or head millions of small objects in a loop, retrieval fees when a reprocessing job pulls a cold archive back in bulk, and egress if the processing runs in a different cloud or region from the storage. Keep the compute next to the data, batch your listings, and check the retrieval cost before scheduling a full backfill from archive tier — a reprocess that was budgeted on compute alone can arrive with a storage bill several times larger.
Reconciliation, not dashboards
The metric that matters is not throughput. It is whether the counts agree. Files received, files in each state, sub-units expected versus written, outputs produced. Run that reconciliation on a schedule and alert on any gap, because gaps are how files disappear: an exception swallowed in a worker, a message acknowledged before the write, a batch whose last chunk was never enqueued.
Beyond counts, three signals earn their place. Age of the oldest item still in processing, which catches stuck work that a rate graph hides. Failure rate by reason code, which shows a new bad format the day it starts arriving rather than the month after. And processing time at p99 by format, because a library upgrade that doubles the tail is invisible in an average.
The mistakes we get called in to fix
- Uploads streamed through the API server, coupling bandwidth to request capacity
- Capacity planned on mean file size, with no plan for the 99th percentile
- Whole-document retry, so a failure on the last page discards the whole run
- Originals deleted after extraction, making every future improvement unbackfillable
- No processing-version field, so every fix implies reprocessing everything
- Empty text extraction recorded as success on scanned documents
- A backfill script separate from the main pipeline, drifting into different output
- An exception queue nobody owns, quietly accumulating for two years
A three-week hardening pass
Ingestion Hardening
Then break it deliberately. Feed it a zip bomb, a 30,000-page document, a file whose extension lies, a truncated archive, an encrypted document, and the same file twice at the same moment. Six inputs, an afternoon, and they find most of what would otherwise arrive as a production incident on a weekend.
Before you scale it up
- The size and page-count distribution is measured, not assumed
- Clients upload directly to object storage with a size cap in the signed policy
- Every file has a registry row with an explicit state before processing begins
- Content hashing is on, and derived artifacts are keyed on hash plus version
- Format is detected from bytes, and empty extraction is a failure, not a success
- Parsers run sandboxed with memory caps, timeouts and no network access
- Concurrency is sized by memory per worker, not by core count
- Large items are split so retry costs a page, not a document
- Raw originals are retained; everything derived is regenerable
- A scoped reprocess has been run end to end at least once, on purpose
Bottom line
File pipelines fail on distribution, not volume. Measure the tail before designing anything. Keep bytes out of the application, give every file a row, hash it, detect its real format, and run the parser in a box that cannot hurt you. Pick a retry unit smaller than the file, treat partial success as a first-class outcome with a queue a human actually works, and build for the day you reprocess everything — because that day arrives on every pipeline that is worth having. Almost none of this is difficult. All of it is much cheaper before there are four million files in the system than after.
Frequently asked questions
The tail of the size distribution. Median files stay fast while the 99th percentile grows, and a design that assumes uniform per-item cost stalls behind one large item. The second thing is memory: concurrency sized by core count kills workers as soon as a few large documents are processed at once.
Three reasons at once: exact-duplicate detection, which commonly removes ten to thirty percent of the work in real corpora; integrity checking against silent truncation; and a stable cache key so derived artifacts can be keyed on content plus processing version. That last one is what makes scoped reprocessing possible later.
With counts by outcome, a reason code on every failure, and a queue where a person can retry, reclassify or reject. A single success flag over thousands of files is false in practice on any real corpus, where one to five percent will fail for legitimate reasons.
Yes, subject to whatever retention rules apply to the content itself. The original is the only artifact you cannot regenerate. Deleting it to save storage trades a small recurring cost for a permanent limit on improving parsing or extraction later, and lifecycle rules to colder storage tiers cut most of that cost anyway.
Parse in a separate process with a hard memory limit, a wall-clock timeout, no network access and a disposable temp directory. Cap expanded size and entry count for archives, limit container nesting depth, and quarantine rather than delete anything that trips a limit so a false positive is recoverable.