Vendor data quality SLAs that mean something

The clause that gets signed and the clause that gets measured

Open almost any commercial data agreement and you will find a quality commitment that reads like a promise and behaves like a decoration. The usual shape is a percentage — ninety-nine point something accuracy, ninety-nine point something availability — with no denominator, no measurement procedure, no artifact either side agrees to compute it from, and no stated consequence beyond a credit sized to be smaller than the cost of arguing about it. Two years later nobody has ever calculated the number, and the feed is still wrong every third Tuesday.

The failure is not that the number is too generous. It is that the clause is not computable. Accuracy against what reference, over which records, excluding which known-bad segments, reported by whom and from which system? A term nobody can evaluate is not a weak term. It is an absent one, occupying the space where a real term would have gone.

The version that works starts from the opposite end. Write down every quality property you can compute at your own boundary, from artifacts already in your possession the moment a delivery lands. Commit only to those, define each to the level of a SQL query, and attach a consequence that scales with the harm. Everything else belongs in audit rights, representations and notice obligations, where enforcement does not depend on a measurement you will never perform.

Measure at your own door

One rule sits underneath all of this: an obligation you cannot verify without the other party's cooperation is an obligation that exists at their discretion. That is not cynicism. Good data vendors are staffed by careful people who want the feed to be right. It is a statement about incentives under load: when a vendor is mid-incident and short-handed, the terms that get honored are the ones that produce a number on your screen without anyone at their end doing anything.

So the first pass through a quality exhibit is a sorting exercise. For each proposed term, ask: on the day this is breached, what file on my disk proves it? If the answer is a report the vendor sends, downgrade the term. If the answer is a delivery manifest, a file checksum, a row count, a timestamp on my own ingest log, or a query over the data itself, keep it and make it precise.

Six dimensions, and only four of them are free

The industry vocabulary around data quality is larger than it needs to be. For a purchased feed, six dimensions cover nearly everything that will hurt you, and they divide cleanly into the ones you can compute with no reference data and the ones that require a source of truth you do not have.

Dimension	What it means for a purchased feed	Can you measure it alone?
Timeliness	Delivery landed by the contractual cut-off; and separately, the lag from the real-world event to its appearance in the feed	Yes — your ingest log has both timestamps
Completeness	Expected record count within tolerance; null and empty rates per field within tolerance; no missing partitions or gaps in a date series	Yes — counted on receipt
Validity	Conforms to the published schema: types, lengths, enumerated values, key uniqueness, referential integrity, units, encodings	Yes — if the schema is a contractual artifact rather than a PDF
Stability	Distributions, category mixes and identifier populations do not move without explanation; restatement volume stays within a stated band	Yes — against your own prior deliveries
Accuracy	Values match the real world	No — requires an independent reference or an adjudicated sample
Uniqueness / resolution	One real-world thing appears exactly once, and the same thing keeps the same identifier over time	Partly — exact duplicates yes, semantic duplicates no

Four dimensions in the free column is a strong position, because that is where the overwhelming majority of real production incidents live. Feeds rarely go subtly wrong in a way only ground truth reveals. They arrive late, arrive short, arrive with a new column and a changed enumeration, or arrive with last month rewritten.

Accuracy is the one everybody writes into the contract and nobody measures, because measuring it honestly means an adjudicated sample: a random draw each quarter, a human establishing truth from a source independent of the vendor, an error rate with a confidence interval. That is real work, worth doing on the two or three fields your business depends on and a waste of money on the rest. Say exactly that in the exhibit, and leave the other two hundred fields to validity and stability, which catch most accuracy failures by a side door.

Define the unit of failure before you argue about the number

The most common reason a quality dispute goes nowhere is that the two sides are counting different things. The vendor says the delivery was 99.98 percent correct. You say the delivery was unusable. Both statements are true, because one is counting field-values and the other is counting business outcomes.

Fix it by writing the unit down. There are four reasonable choices and they produce wildly different numbers from identical data:

The delivery. One file, one API window, one batch. Binary: it met the criteria or it did not. Coarse, but the easiest to administer and the right unit for timeliness.

The record. A record is defective if any material field is defective. Good for completeness and validity, and the unit most business users think in.

The field-value. Every cell counts. This is the unit that produces flattering percentages, which is why it appears in vendor-drafted quality exhibits. A file with one broken column out of four hundred is 99.75 percent correct at the cell level and completely useless if that column is the identifier.

The materiality event. A defined class of failure — the primary key duplicated, a price field off by an order of magnitude, a whole market missing — where a single instance is a breach regardless of rate. This is the unit that protects you from the field-value arithmetic.

The practical answer is to use more than one. Timeliness at the delivery level, completeness and validity at the record level with per-field tolerances for a named critical set, and a short list of materiality events that breach on a single occurrence. Three units, each obviously right for what it governs, and no denominator fight left to have.

Pin down one more definition: what counts as a delivery at all. A file that arrives on time and is empty, truncated, or a copy of yesterday's is not a delivery. Say so explicitly, with a minimum-row test. Every mature ingest pipeline already has a special check for that case.

Restatements are the term nobody writes and everybody needs

If you buy data that feeds research, models, reporting or anything with a historical record, this is the section that matters most, and it is missing from most agreements we read.

Vendors correct history. They should — a corrected value is better than a wrong one. The problem is doing it silently and in place. When last quarter's values change without notice, your backtest changes, your published metric no longer reproduces, your model's training data no longer matches the data it was trained on, and nobody can tell whether a performance shift came from the model or from the ground moving. In a ratings, research or analytics business this is not a data quality annoyance. It is a defect in your product that surfaces months later in front of a customer.

Four terms fix it, and they are cheap to ask for because good vendors already have the machinery:

Point-in-time snapshots. Every delivery is immutable and addressable forever by its vintage. You can always reconstruct what you knew on a given date, which is the only honest basis for a backtest.

An as-of key in the data. Two dates on every record: the date the fact was true, and the date you learned it. This is the bitemporal pattern, it is not exotic, and it converts a restatement from a rewrite into an append.

A restatement log. A machine-readable record of what changed, for which periods, and why. If the vendor will not produce one, you can compute it yourself from immutable snapshots — which is another reason the first term matters.

Notice and a threshold. Advance notice for planned restatements, prompt notice for corrections, and a stated band above which a restatement is a reportable event rather than routine maintenance.

Ask for the same discipline on schema: a version number, a deprecation window measured in months rather than days, and a commitment that new columns are additive. The failure mode is not malice. It is a vendor product team shipping an improvement on Tuesday and a downstream parser exploding on Wednesday.

Coverage is measured against your universe, not theirs

A vendor's coverage claim is computed against the population the vendor decided to serve. Yours has to be computed against the population you actually need, and the gap between those two is where most disappointment lives.

Do this before signing. Take a real list of the entities, instruments, addresses or facilities you need covered — a few thousand is plenty — and have the vendor run it during evaluation. Measure three things: what fraction is present at all, what fraction has the specific fields you depend on populated, and what fraction carries an identifier that joins to yours without heroic effort. The third number is the one that surprises people.

Then put the result in the agreement: coverage of a named universe, refreshed periodically, with the universe file attached as an exhibit. That one exhibit turns the most subjective claim in the sales process into an arithmetic one, and gives you a clean measurement at renewal.

Remedies: service credits are the weakest thing you can ask for

Almost every data agreement offers service credits, and almost every service credit regime is sized so that invoking it is irrational. A credit worth a few percent of a monthly fee does not cover the engineering hours spent proving entitlement to it, which is presumably the design.

Four remedies do more, roughly in order of how hard they are to get:

A cure period with a termination right. Repeated breach of a defined threshold over a defined window lets you terminate without penalty and without paying out the term. This is the remedy vendors respond to, because renewal revenue is what their account team is measured on, and it costs them nothing if they perform.

Fee abatement tied to the affected period. Not a token credit — a proportional reduction for the period in which the data was unusable. Harder to win, and worth trading other things for.

A perpetual license to data already delivered. The one buyers most often forget. If the agreement ends, can you keep using the history you paid for, including inside models trained on it? Many standard terms say no, or say nothing, which becomes an unpleasant discovery during a switch. Ask for a perpetual, irrevocable license to data received during the term.

Audit and sampling rights. The right to sample and adjudicate, and to see the vendor's own quality metrics for your feed. This is the backstop for the accuracy claims you cannot compute yourself.

While you are in the licensing terms, settle derived data. Can you publish an output computed from the feed? Does a score, an index, or a model trained on it survive termination? Vendors differ here, and their spoken answers are often more generous than their paper. Get the generous answer written down.

The receiving harness, and why you probably do not need us to build it

Everything above assumes a piece of software on your side that inspects each delivery and records what it found. That software is not hard. This is the part of the article where the honest answer is that a competent data engineer on your own staff can build it, and should.

The shape is small. On arrival, record the delivery: source, filename, byte size, checksum, arrival timestamp, contractual cut-off, vintage. Load to staging untouched. Run the checks — row counts, null rates per field, schema conformance, key uniqueness, referential integrity, value ranges, category distributions against the trailing window, and a diff against the previous vintage for restatement volume. Write every result to an append-only table keyed by delivery. Promote on pass, quarantine on fail, page a human only on the material classes.

Standard tooling covers most of it. For one feed this is a few days of work. For a portfolio of thirty feeds with different shapes it is a few months, and the time goes almost entirely into agreeing what the thresholds should be — a governance conversation, not an engineering one.

The output that changes vendor behavior is a monthly scorecard: one page per feed, the agreed metrics, the trend, the incidents. Send it to them. Almost nobody does, and the effect is larger than any clause, because it moves your account from anecdote to record inside their organization. An account manager can act on a chart. They cannot act on the recollection that you seemed annoyed in June.

Where outside help is worth paying for: when the definitions are genuinely contested across teams, when you are consolidating many vendors onto one standard and need somebody with no stake in whose definition wins, or when the failures are subtle enough that the checks themselves need statistical design. Building the harness is not that; do not hire it out.

The mistakes we see in signed data agreements

• An accuracy percentage with no denominator, no reference source and no measurement procedure
• Quality metrics reported by the vendor, from the vendor's systems, with no independent computation
• No restatement policy, so history is rewritten in place and research stops reproducing
• The schema is a PDF rather than a versioned, machine-readable, contractual artifact
• Coverage claimed against the vendor's universe and never tested against yours
• Service credits as the sole remedy, sized below the cost of claiming them
• Silence on data already delivered, so termination takes your history with it
• Silence on derived outputs, so it is unclear whether your own product survives a switch

Before you sign

• Every quality term is computable from artifacts you already hold
• The unit of failure is defined, and there is more than one
• A short list of materiality events breaches on a single occurrence
• An on-time empty or truncated file is defined as a non-delivery
• The schema is versioned, machine-readable and attached as an exhibit
• Schema changes are additive, with a notice period in months
• Deliveries are immutable and addressable by vintage
• A restatement log exists, with notice and a reportable threshold
• Coverage is measured against your universe file, attached as an exhibit
• Repeated breach gives a termination right without penalty
• You keep a perpetual license to data already delivered, and to derived outputs
• A receiving harness runs on day one, and a scorecard goes to the vendor monthly

Bottom line

A data quality SLA is worth exactly as much as your ability to compute it without asking anyone. Sort the terms on that basis, keep the four dimensions that are free to measure, define the unit of failure in the contract instead of in the incident, and spend the negotiating capital on restatement discipline and on remedies that touch renewal rather than on a percentage nobody will ever calculate. Then build the small piece of software that measures it and send the vendor a page every month. The feed will get better, and you will know it did.

Frequently asked questions