What a data pipeline costs to build and run

Two numbers, and only one of them gets asked about

A single pipeline that moves one well-behaved source into a warehouse, with modest reshaping and a daily schedule, is $15,000 to $45,000 of engineering. A production pipeline that several teams depend on, with multiple sources, quality checks and someone accountable when it fails, is $60,000 to $200,000. A platform, meaning shared ingestion, transformation, testing and monitoring that other people can add to without calling you, starts around $250,000 and is a program rather than a project. Those are build numbers. The running cost is separate, it arrives every month, and it is the reason most of these projects are judged a disappointment in year two.

Data work is priced badly across the industry because the unit is wrong. Buyers ask what a pipeline costs the way they would ask what a door costs. But a pipeline is not a thing you buy once. It is a standing relationship with somebody else's system, and that system will change without telling you.

Four variables set the price, and volume is not one of them

How many sources, and do you control them. A source you own, with a documented schema and a team you can email, is a fraction of the cost of a vendor API you do not control. The expensive case is a source that changes without notice: a partner's export, a SaaS product on its own release schedule, a legacy system whose owner retired.

How fresh the data has to be. This is the biggest single lever and it is usually specified carelessly. More on it below.

What happens if a number is wrong. A dashboard nobody acts on and a figure that goes into a customer invoice are not the same product. Correctness requirements drive testing, reconciliation, lineage and alerting, and those together can be a third of the build.

Who has to be able to change it later. A pipeline only you will ever touch is cheaper than one an analyst must be able to extend. Making work legible to the next person is real engineering, and it is the difference between an asset and a liability with your name on it.

Notice what is not on the list. Raw data volume matters far less than people expect until you reach genuinely large scale. Ten million rows and a hundred million rows are usually the same project with a different bill from the warehouse. Ten sources and one source are not.

Freshness is the most expensive word in the requirement

Somebody writes "real time" in a requirements document because it sounds like the safe answer. It is the single most expensive sentence in the file, and in most cases nobody downstream needs it. Ask instead what decision the data supports and how often that decision gets made. A weekly ops meeting does not need minute-level data. A fraud hold does.

Freshness	What it implies technically	Relative build cost
Daily batch	One scheduled run, a re-run window, simple recovery	Baseline
Hourly	Incremental loads, watermarks, overlapping-run protection	1.3–1.6×
Every few minutes	Change capture, late and out-of-order handling, deduplication	2–3×
Continuous streaming	A streaming platform, exactly-once reasoning, its own on-call	3–5×

The multiplier is not only the build. Continuous pipelines fail continuously, which means somebody carries a pager. Before agreeing to it, get the answer to one question in writing: what would a person do differently if the number were four hours fresher? If nobody can answer, you have found a large amount of budget.

The first source is expensive, the fifth is cheap, the twentieth is expensive again

The cost curve is not linear and it helps to know its shape before you plan a roadmap. The first source carries the whole foundation: the orchestration, the storage layout, the deployment path, the secrets handling, the alerting, the conventions. It is often forty percent of a multi-source project by itself.

Sources two through roughly eight are genuinely cheap, frequently a third to a fifth of the first, because they reuse everything. This is where the promise of a platform is real, and where a good vendor should be visibly getting faster. If your third source cost as much as your first, the foundation was not built, and you are paying for the same work three times.

Past about ten sources the curve turns back up, for organizational reasons rather than technical ones. Now there are dependencies between datasets, competing definitions of the same business term, a queue of change requests, and a real question about who owns each table. That is the point at which a company needs governance, ownership assignments and a request process, and skipping it is why large data teams end up feeling slower than the two-person team that preceded them.

The three bills that arrive every month

Every running data stack sends three bills, and buyers frequently see only one of them because the others are inside a cloud invoice nobody itemizes.

Compute. Whatever executes the transformations. In a modern warehouse this is usually billed by processing time or credits, and it is dominated by a small number of expensive jobs rather than by overall size. A mid-size company running an analytics warehouse commonly lands somewhere between $500 and $5,000 a month, and the spread inside that range is mostly about how carelessly the biggest queries were written. Full-refresh jobs that rebuild everything nightly are the usual culprit, and converting the worst two or three to incremental loads often cuts the bill by half.

Storage and movement. Object storage is cheap enough to ignore for most companies. Data movement is not. Egress charges when data leaves a cloud provider, and cross-region traffic inside one, are the two lines that show up unexpectedly. Design where data lives before you design what it does, because moving it later is both an engineering project and a recurring fee.

Tools. Managed connectors, an orchestration service, a transformation tool, a quality or catalog product. Individually modest, collectively a five-figure annual number for a small stack. Connector products in particular tend to price on volume, which means the bill grows exactly when the business grows, and it is worth knowing the shape of that curve before you standardize on one.

Buying connectors versus writing them

For common commercial systems, a managed connector product is usually the right answer and we say so even though it means less work for us. A vendor maintaining an integration against a popular SaaS API is absorbing a maintenance burden you do not want, and the subscription is almost always cheaper than the engineer-hours to keep pace with the API.

Custom ingestion earns its place in three situations. When the source is yours, in which case there is nothing to buy. When the source is obscure, legacy or internal, which describes most of the interesting data in an established company. And when the volume makes per-row pricing worse than an engineer, which happens sooner than people expect on high-volume event data.

The pattern that works for most mid-size companies is a hybrid: buy the connectors for the standard systems, write the ones for the systems that make you money, and keep transformation and testing in your own repository so the logic stays portable. Logic that lives inside a vendor's user interface is logic you cannot review, test, or take with you.

Three sizes, with numbers attached

Shape	What it covers	Build	Typical monthly run
One reliable feed	One source, daily, into a warehouse, with tests and alerting	$15,000–$45,000	$300–$1,500
A team's data foundation	Four to eight sources, modeled tables, quality checks, documentation	$60,000–$200,000	$1,500–$8,000
A platform others build on	Shared ingestion and transformation, lineage, self-service, governance	$250,000+	$8,000–$40,000

The monthly figures assume cloud infrastructure and tooling, not people. Add the maintenance percentage above for the human line. These ranges assume senior engineers at US rates, which for this work generally run $150 to $250 an hour however they are billed, and they assume you can grant access to the sources within the first two weeks. Access latency is the most common reason a data project overruns its schedule without overrunning its hours.

What makes a pipeline cheap to own

The difference between a pipeline that costs five percent of its build price a year and one that costs forty percent is not sophistication. It is a short list of properties, all of which are cheaper to build in on day one than to retrofit.

• Re-running it produces the same result, so recovery from a failure is a command and not an investigation
• Backfilling a date range is a normal operation, documented, not a special script someone remembers
• A schema change in a source fails loudly and immediately, rather than writing nulls for a week
• The definitions live in version control, reviewable, testable and portable between tools
• Alerts fire on what a person would notice: no data, wrong volume, a broken relationship between tables
• A runbook exists that a competent engineer who has never seen the system can follow at 3am
• It runs on a laptop against sample data, so changing it does not require production access

How to read a data engineering quote

Ask three questions of any proposal and most of the ambiguity disappears. First, what happens when a source changes shape, and is fixing that inside this price or outside it? Second, what is the recurring cost, itemized, including the engineering percentage. Third, what exactly is handed over: code in our repository, credentials, documentation, and a session where our own people run it unaided.

A proposal that quotes only the build, only the happy path, and hands over a dashboard is the cheapest one on the table and the most expensive one in the file. It is not usually dishonest. It is answering the question that was asked, and the question that was asked left out the part that costs the most.

Where the money gets wasted

• Specifying real time because it sounded safe, then discovering the report is read on Mondays
• Nightly full refreshes of everything, which is the single most common cause of a surprising warehouse bill
• Building the platform before there is a second use case, and generalizing against imagined requirements
• Transformation logic locked inside a vendor interface, so it cannot be reviewed, tested or moved
• No owner named at handover, so maintenance is unfunded and unassigned until something breaks
• Ingesting everything because storage is cheap, which is true of storage and false of the modeling, testing and explaining that follow
• Skipping the backfill design, then paying an engineer to hand-run reprocessing every quarter

A thirty-day way to get a defensible number

Step four is the one worth insisting on. One source built properly turns every other number in the plan from a guess into an extrapolation, and it leaves you with a working pipeline whether or not the rest of the program is funded. It is the cheapest way we know to de-risk a data budget.

Bottom line

Price a pipeline by counting sources, not rows. Interrogate the freshness requirement before anything else, because it is the largest multiplier and it is usually overstated. Put the three monthly bills and the maintenance percentage in the business case beside the build price, since the ratio between them over three years is roughly one to one. And build the first source properly, because everything downstream is priced off it, including the ones you have not thought of yet.

Frequently asked questions