What data quality engineering actually is
Data quality engineering is the practice of deciding what "correct" means for every field that drives a decision, writing that definition down as executable checks, running them on every load, and putting the results where the people who depend on the data can see them. It is a software discipline with tests, thresholds, owners, and a failure budget. It is not a cleanup phase. Treating it as one is the most common reason an analytics or machine learning effort stalls six months in with a model that scores well offline and behaves badly in production.
The research backs this up. In "Data Cascades in High-Stakes AI" (CHI 2021), Sambasivan and colleagues interviewed 53 practitioners building AI for health, finance, and conservation, and found compounding downstream failures traceable to data problems in 92 percent of cases. Those failures were mostly invisible at the point of origin and expensive at the point of discovery, which is what makes a one-time cleanup so appealing and so ineffective. Gartner has separately put the average annual cost of poor data quality at roughly $12.9 million per organization, a directional estimate rather than a measurement, but nobody argues it is small.
The second reason to treat this as engineering is that data changes without asking permission. A source system adds a dropdown value. A vendor switches a units convention. An upstream team fixes a truncation bug that had been shortening a text field for two years, and the fix shifts the distribution your model was calibrated against. None of those events raises an exception. They produce slow degradation that the model's own metrics report as noise.
The dimensions worth measuring
ISO/IEC 25012:2008 defines a data quality model with fifteen characteristics, and ISO 8000-61:2016 a process reference model for managing them. Both are useful for a governance document and too broad to run a pipeline against. Most teams narrow to the six dimensions the DAMA body of knowledge popularized: completeness, uniqueness, validity, consistency, timeliness, accuracy.
Five of those six can be checked from inside the data. Accuracy cannot: it requires comparison against something outside the dataset, a source of record, a human adjudication, or an instrument reading. That asymmetry drives most of what follows, because the cheap automated checks catch structural problems and the problem that hurts you is usually semantic.
Where data quality effort pays back, by dimension
Editorial weighting of effort-to-payback from practitioner reading and public literature. Illustrative, not a measured statistic.
| Dimension | Executable check | Threshold that usually holds |
|---|---|---|
| Uniqueness | Row count against distinct count of the declared key, per partition | 100 percent, no exceptions. A duplicate key is a design defect, not a data defect. |
| Completeness | Null, empty-string, and sentinel rate per column | 99.5 percent or better on decision fields. Everywhere else, measured and published rather than assumed. |
| Validity | Type, range, regex, and enum membership | 99.9 percent on typed fields. Any unseen enum value stops the load and pages a human. |
| Consistency | Referential integrity plus reconciliation to a second source | Zero orphan foreign keys. Source-to-source delta inside a tolerance you wrote down in advance. |
| Timeliness | Maximum event timestamp compared against the load clock | 95th percentile inside the published lag budget. The 99th percentile warns; it does not block. |
| Accuracy | Adjudicated gold sample compared against pipeline output | Set from the cost of a wrong decision. There is no defensible blanket number. |
The decisions that are expensive to reverse
Some choices cost an afternoon to change in week one and six months to change in year two. These are the ones worth arguing about before any code ships.
Grain and key. What does one row mean? One claim, one claim line, one claim version, or one claim as of a point in time? Every downstream aggregation inherits this answer, and changing it later invalidates every number already reported. Write the grain as a sentence at the top of the model file, then enforce it as a uniqueness test.
Time semantics. Event time, ingest time, and valid time are three different clocks, and systems that collapse them produce results nobody can reconstruct. Store timestamps in UTC, keep the original offset in its own column, and decide early whether the table is bitemporal. Retrofitting valid-time history onto a table that only kept current state is impossible: the history is gone.
Null semantics. "Unknown," "not applicable," and "not collected" are three different facts, and a single NULL erases the difference. Sentinels are worse: a birth date of 1900-01-01 or an amount of 9999 passes every type check and quietly poisons every average. Decide the encoding once, and check for the sentinels you inherited.
Numeric representation. IEEE 754 binary floating point cannot represent 0.10 exactly. Money and any quantity that gets summed and audited belongs in a fixed-precision decimal type. Cheap to prevent, tedious to unwind, because every stored value has to be rebuilt from source.
Identity. Matching entities deterministically on a shared key and matching them probabilistically through record linkage in the Fellegi-Sunter tradition support different claims. Probabilistic linkage produces a match score, and every consumer needs the threshold and the expected false-match rate. Publishing linked records without that number invites a claim nobody can defend.
What you keep. An immutable landing zone holding raw source bytes is the only thing that lets you re-derive a table after you find the parser was wrong. Hashing an identifier without retaining the salt, and dropping raw text after extraction, are both one-way doors.
What "good" means numerically
A threshold that is not tied to a decision is decoration. Derive it by asking what a wrong record costs. If a false positive triggers a four-minute manual review, a 2 percent error rate on ten thousand records a day is thirteen hours of review, which is a staffing question with a clear answer. If a wrong record produces a payment, a benefits denial, or an entry in a system of record that can harm a citizen, the tolerable rate drops by orders of magnitude and the check moves from asynchronous monitoring to a blocking gate.
Sampling math keeps these conversations honest. For a binary correct-or-not audit against a large population, a random sample of about 1,067 records estimates the defect rate to within plus or minus 3 percentage points at 95 percent confidence; 384 records gets you to 5 points. Audit 300 records, find zero defects, and the rule of three puts the upper 95 percent bound near 1 percent. That is the honest statement. "We reviewed 300 records and they were clean, so the data is clean" is not.
Segment before you conclude. A global accuracy of 97 percent can hide a subgroup at 60 percent, and that subgroup is usually the one that matters: the rare category, the newest source system, the region that onboarded last quarter. Report the worst segment beside the average, and set the threshold on the worst segment.
Accuracy needs a gold set
Because accuracy cannot be computed from inside the data, someone has to build a reference. The pattern that works is a stratified sample, labeled independently by two reviewers against a written definition, with a third adjudicating disagreements. Measure the reviewers before you measure the pipeline. Cohen's kappa above 0.80 is what Landis and Koch called almost perfect agreement, and it is a reasonable bar for a rubric you intend to hold a system to. Krippendorff's alpha handles more reviewers and missing labels.
If agreement between careful humans is 0.55, the definition is ambiguous and no amount of model work will fix it. Rewrite the rubric, relabel, then evaluate. Skipping this step is why so many evaluation numbers cannot be reproduced by the customer.
Reference sets carry errors of their own. Northcutt and colleagues (NeurIPS 2021 Datasets and Benchmarks) audited the test sets of ten widely used vision, language, and audio benchmarks and estimated an average label error rate near 3.4 percent, with roughly 6 percent in the ImageNet validation set. Those are the datasets the whole field calibrates against. A gold set built in a hurry by one reviewer will be worse.
Where the checks belong in the pipeline
The failure modes teams keep hitting
Checking the pipeline instead of the data. Row counts match, the job exits zero, the dashboard is green, and the meaning of a column changed three weeks ago. Freshness and volume checks are necessary and detect almost nothing semantic.
Thresholds fitted to today's data. Profile a table, set every bound to the observed minimum and maximum, and you have written a test that can only fail when something legitimate happens. Bounds come from the domain: an age of 150, a latitude of 91, a negative quantity on a receipt.
Silent coercion. A loader reads "N/A" and produces a null. A dataframe library promotes an integer column to float the moment one null appears, and identifiers gain a decimal point. A stray comma shifts every field one column left for a single CSV row. Each produces valid-looking output.
Near-duplicate contamination. Exact duplicates get removed; near duplicates do not. When the same record lands in training and test with different whitespace, or as a re-scan of the same document, evaluation scores rise and production performance does not. Deduplicate on normalized content, not on the primary key.
Labels produced by an earlier model. A field populated by a previous system's predictions, then used as ground truth for the next one, measures agreement with the old model rather than correctness. Track label provenance as a column.
Alert fatigue. A check that fires every day is a noise generator, and the one time it means something nobody will look. Every rule needs a named owner and a documented response. Rules with neither should be deleted.
Production concerns: latency, cost, monitoring
Validation costs money and time, and placement decides how much. Blocking checks in the ingest path add latency to every record and stop bad data before it spreads. Asynchronous checks cost nothing at write time, but by the time they fire the bad partition has already been read. The workable middle is a small set of blocking assertions on structure and key integrity, with the expensive statistical work running behind them and writing failures to a quarantine table rather than discarding them.
Scan cost is the part teams underestimate. On a columnar warehouse billed by bytes scanned, a naive full-table suite re-reads the entire history every run. BigQuery's on-demand analysis pricing has been $6.25 per TiB scanned in US regions since 2023, so forty checks against a 2 TiB table nightly is roughly $500 a day of pure validation. Partition-pruned incremental checks on the new slice, plus approximate sketches such as HyperLogLog for distinct counts and t-digest for quantiles, cut that by one to two orders of magnitude with no meaningful loss of signal.
For drift, the usual instruments are population stability index, the Kolmogorov-Smirnov statistic for continuous features, and a chi-square test for categorical ones. A PSI above 0.25 on a decision-bearing feature is a conversation; below 0.10 is normally noise. Store the metric as a time series rather than a pass or fail flag. Trends are readable and boolean history is not.
The federal overlay
On federal work, data quality is also a legal concern. The Information Quality Act, enacted as section 515 of Public Law 106-554, directs agencies to ensure and maximize the quality, objectivity, utility, and integrity of information they disseminate, under OMB guidelines at 67 FR 8452 and reinforced by OMB Memorandum M-19-15. Those guidelines require an administrative mechanism through which an affected person can request correction. If your pipeline feeds a public-facing product, a correction request is a live possibility, and your lineage is the answer to it.
The Foundations for Evidence-Based Policymaking Act of 2018, Public Law 115-435, adds the machinery: Chief Data Officers under 44 U.S.C. 3520, agency data inventories, and a default toward open, machine-readable formats. NIST SP 800-53 Rev. 5 carries the control-level hooks, including SI-10 for input validation, SI-7 for information integrity, SI-18 and PM-22 for the quality of personally identifiable information, and the audit family for the record of who changed what. DoD states the same ideas as VAULTIS: visible, accessible, understandable, linked, trustworthy, interoperable, secure.
Sector rules tighten further. FDA-regulated electronic records fall under 21 CFR Part 11, which requires computer-generated, time-stamped audit trails that do not obscure prior entries. Health data de-identification follows 45 CFR 164.514, by Safe Harbor removal of eighteen identifier types or by expert determination. Federal tax information carries the IRS Publication 1075 safeguards. In each, the deliverable is clean data plus a record showing how it got that way.
Data contracts and schema change
The most durable fix for upstream surprises is a contract that fails in the producer's build rather than in the consumer's dashboard. A schema registry with compatibility enforcement does most of the work: backward compatibility lets new readers handle old data, forward compatibility lets old readers handle new data, and full compatibility gives both at the price of a stricter change policy. Adding an optional field with a default is safe. Removing a required field or narrowing a type is not, and the registry says so at commit time.
Breaking changes still happen, and the pattern that survives them is expand, migrate, contract. Add the new column beside the old, dual-write for a defined window, move consumers one at a time, then drop the original once nothing reads it. The window is the expensive part, and skipping it is how a schema change becomes an outage.
When a simpler method is the right answer
Data quality tooling has grown fast, and many teams buy an observability platform before they can name the ten checks that matter. If those ten are not written down, the platform will surface several thousand anomalies and the team will mute the alerts within a month.
A twenty-assertion test suite in your transformation framework covers most of what a small pipeline needs: not-null on decision fields, unique on the key, accepted values on every enum, referential integrity on every join, and a freshness check. That is a day of work, it runs in CI, and it fails in the pull request. For a stable dataset with few consumers, it may be the whole answer for years.
Two more honest cases. When data is hand-entered at low volume, the highest-yield fix is the form. A required field with a controlled vocabulary prevents more defects in a week than downstream repair logic catches in a year. And when someone proposes an anomaly-detection model to find data problems, ask what three obvious rules would have caught first. Learned detectors earn their place once the rule-based checks are exhausted and the remaining failures are genuinely distributional.
Heavier machinery is justified when there are many producers you do not control, hundreds of tables, a lineage requirement you must evidence, or external consumers. The decision is one of scale and blast radius, and it should be made deliberately.
Bottom line
Data quality engineering is unglamorous and it is where the returns are. Define the grain, fix the time and null semantics, write checks that map to decisions, build a gold set with measured reviewer agreement, put expensive validation where it is cheap to run, and give every rule an owner. Do that first and the modeling gets easier. Skip it and the modeling gets repeated.
Common questions on scope and rigor
Does every column need a threshold?
No. Thresholds belong on fields that drive a decision, feed a published number, or enter a system of record. Profile and report the rest so the state is known, without a gate that blocks a load over a field nobody reads. A rule with no consequence is a maintenance cost.
Can data quality be fixed after the model is trained?
Partly. Label noise can be found and corrected after the fact with confident-learning methods, and duplicates can be removed and the model retrained. What cannot be recovered is a lost distinction: history never stored, a units convention normalized away, or a raw source deleted after parsing.
Who should own the checks, engineering or the business?
The definition belongs to whoever owns the decision the data supports. The implementation belongs to engineering. The failure mode is an engineer inventing a threshold because no one else would commit to one, which produces a rule that fires and gets waived every time.
Frequently asked questions
Completeness, uniqueness, validity, consistency, timeliness, and accuracy are the six most teams use. ISO/IEC 25012:2008 defines a fuller model of fifteen characteristics. The first five can be checked inside the data; accuracy always requires an external reference.
Build one. Take a stratified random sample, have two reviewers label it independently against a written rubric, adjudicate disagreements with a third, and check agreement before trusting the result. Cohen's kappa above 0.80 is a reasonable bar. If reviewers disagree, the rubric is the problem.
For a binary correct-or-not judgment against a large population, roughly 1,067 records give a 3-point margin of error at 95 percent confidence and 384 give a 5-point margin. If a sample of 300 comes back with zero defects, the rule of three puts the upper 95 percent bound near 1 percent.
The Information Quality Act (section 515 of Public Law 106-554) and OMB's guidance at 67 FR 8452 set quality, objectivity, utility, and integrity expectations for disseminated information, including a correction process. The Evidence Act (Public Law 115-435) adds data inventories and Chief Data Officers. NIST SP 800-53 Rev. 5 supplies the controls, and sector rules such as 21 CFR Part 11 apply on top.
When you have many producers you do not control, hundreds of tables, external consumers, or a lineage requirement you must evidence. Below that, a twenty-assertion suite running in CI covers most of the risk for a fraction of the cost.