Readiness is a relationship, not a property
There is no such thing as clean data in the abstract. There is only data that does or does not support a particular question at a particular tolerance. The same customer table is perfectly adequate for “how many accounts churned last quarter” and hopelessly inadequate for “which accounts will churn next quarter,” because the second question needs a stable identity, a consistent definition of churn over time, and the events that preceded it — none of which the first question cares about at all.
This matters because it changes what to do about it. If readiness were a property, the sensible response would be a program: fix the data, then start the projects. Because readiness is a relationship, the sensible response is to name the question, check the six conditions below against that question, and repair only what that question requires. The first approach takes eighteen months and frequently ends without a single shipped system. The second takes two weeks and ends with a decision.
It is also worth saying plainly that the state of your data is not evidence of mismanagement. Operational systems are built to run a business, not to be analyzed. Fields get repurposed because a release was due. A company gets acquired and inherits a second version of every entity. Someone adds a dropdown option in 2023 and the meaning of a column quietly changes underneath four years of history. Every company that has been operating for a decade has all of this. The ones that appear not to have it have simply not looked yet.
You are probably here because
- A vendor told you the data needs to be fixed before anything can start
- Two reports disagree and nobody can say which is right
- You are being asked to fund a multi-year data program with no shipped output
- A project stalled at the point where somebody had to actually pull the data
The six conditions are the diagnostic. The section on fixing in the path rather than in the warehouse is the one that saves the most money.
Six conditions, checked against one question
Write down the question you want answered, precisely enough that two people would compute it the same way. Then check these six. Every genuinely blocked project we have seen was blocked on one of them, and most projects that people believe are blocked turn out to satisfy all six for a slightly narrower version of the question.
Existence. Is the thing you need actually recorded anywhere? Not “could it be derived” — recorded. The single most common hard stop is that the outcome the business wants to predict was never written down: the reason a deal was lost, whether the technician's fix worked, which alert was a false alarm. No amount of engineering conjures a field nobody ever filled in.
Coverage. Does it exist for enough of the population, and for the part of the population you care about? Data present on 95 percent of small accounts and 40 percent of large ones is not 88 percent coverage in any useful sense if the large accounts are the point.
Identity. Can you tell that this record and that record are the same customer, part, property, or patient? This is the defect that most often masquerades as everything else, and it is the most expensive one to fix late because every downstream number depends on it.
Timeliness. How old is the data when it arrives, and is that inside the window the decision operates in? A nightly batch is fine for weekly planning and useless for a decision made at the counter.
Ground truth. For anything that learns or is evaluated, is there a recorded correct answer, and is it recorded consistently? A label that three different teams filled in with three different meanings is a moving target rather than a target.
Access. Can the people doing the work legally and practically get the data, on a path that will still exist in production? A one-off extract emailed by a friendly analyst is not an access path; it is a favor that will not scale and cannot be automated.
What “bad data” usually turns out to be
When we open a dataset that a company has apologized for, the same handful of defects come up. They have very different costs to fix and very different consequences if ignored, and it is worth being able to name them because “the data is messy” hides all of it.
No stable identity. The same company appears as four rows with four spellings, or a person exists once per system with no shared key. This is entity resolution, it is a real engineering problem, and it is usually the largest single line in a data project. It is also solvable to a useful standard in weeks rather than years, if you accept a match rate rather than perfection.
Meaning drift. A field means one thing before a certain date and another thing after, and nothing marks the boundary. Status codes get reused. A product is renamed. A team changes what counts as “active” and nobody updates the six-year-old report that uses it. Any analysis that spans the boundary is wrong in a way that looks completely plausible.
The real information is in the free-text field. The structured column says “Other” on 40 percent of records and the note field says what actually happened. This is annoying and it is also good news, because reading unstructured text is one of the things current tools genuinely do well.
Missingness that is not random. Fields are blank precisely when the situation was unusual, or when a particular team was busy, or for one region. Filling those blanks with an average bakes the bias in and hides it. Missingness is data; the fact that a field is empty is often the most informative thing on the row.
Duplicates from process, not error. Three rows because the order was amended twice. Two records because a system retried. These are not mistakes to be deleted, they are events to be understood, and deleting them silently changes the totals that finance already reconciled.
The system of record is not where the truth is. The official system says one thing and the team runs off a spreadsheet that a supervisor maintains, because the official system cannot express something the work requires. That spreadsheet is a specification. Find it before designing anything.
| Symptom | What it usually is | Cheapest honest fix |
|---|---|---|
| Two reports disagree | Two definitions of the same word, both defensible | Write down one definition, name an owner, keep both numbers visible until people trust the new one |
| Customer counts look inflated | No stable identity across systems | Match on a scored rule set, publish the match rate, review the uncertain band by hand |
| History looks strange before a date | Meaning drift, unlabelled | Find the boundary, record it, restrict the analysis window or map the old values explicitly |
| Half the category field says “Other” | The real information is in the notes | Extract structure from the text for the specific fields you need, not all of them |
| The model does well in testing and badly live | The test data was cleaned by hand in a way production data is not | Re-run the evaluation on data pulled through the production path, unedited |
| Nobody can produce the extract twice | The pipeline is a person | Write the query down, put it in version control, schedule it |
Fix in the path, not in the warehouse
The instinct when data is wrong is to go and correct it at the source, across the whole history, for everyone. Occasionally that is right. Usually it is a much larger project than the one you were trying to do, it requires agreement from every team that touches the system, and it competes with the roadmap of an operational system that is busy running the business.
The alternative is to put the corrections in the path from source to use: a defined transformation layer, in version control, that maps the messy reality to the clean model the work needs, with the rules written as code rather than performed by an analyst each month. This has three advantages that matter more than they sound. The rules become visible and arguable rather than living in somebody's head. The raw data stays raw, so a rule that turns out to be wrong can be changed and the history reprocessed. And the work can start next week.
Two rules keep this from turning into a mess of its own. Never overwrite the source values — keep the original alongside the corrected one, so the correction can be audited and reversed. And test the assumptions in the pipeline rather than assuming them: if a rule depends on that field never being null, assert it, so the day it changes you find out from a failed check instead of from a customer.
Where fixing at the source is genuinely worth it is at the point of entry, for a small number of fields, going forward. Making one dropdown required and unambiguous today is worth more than reconstructing five years of what it should have said, and it is a change an operational team will usually agree to because it is small.
How much each defect actually blocks work — our read
Our judgment, not a survey. The bottom row is what people apologize for; the top two are what actually ends projects.
The two-week assessment that replaces the two-year program
Before committing to any large data investment, run a short, bounded look at the actual tables against the actual question. Two weeks, one or two engineers, and access to the real systems. It ends with a document rather than a pipeline, and it is the cheapest decision-quality information available in this field.
What it produces: a written statement of the question and its tolerance. A profile of the relevant tables — row counts, fill rates by field, distinct values, date ranges, and where the shapes change over time. A measured identity match rate between the systems that need joining. A list of the definition conflicts found, with the people who own each side. A working end-to-end extract of a small sample, pulled through a path that could be automated. And a verdict on each of the six conditions, with the specific work each one would need.
Cost is typically $20,000 to $60,000 of outside work, or two weeks of an internal engineer if you have one who can be freed. Set against a data platform program with a budget in the hundreds of thousands, it is close to free, and it routinely finds either that the project can start immediately on a narrower question or that one specific blocker has to be solved first. Both answers are worth having before the platform is bought.
What is genuinely a stop
Four findings mean the project as posed does not work, and the honest thing is to say so early rather than to bill through the discovery.
The outcome does not exist. You want to predict something nobody recorded. The only route forward is to start recording it, which means the project is a process change first and a system in twelve to eighteen months. That is a legitimate plan; it is just not the plan that was funded.
There is no lawful or practical access path. Consent does not cover the use, a contract forbids it, or the data lives in a system whose owner will not integrate. This is a negotiation, not an engineering task, and no amount of technical cleverness resolves it.
The volume is too small for what is being asked. Forty examples of the rare event you want to detect will not support a learned model, no matter how the question is framed. Sometimes the right answer is a rules system that encodes what your experts already know, shipped in six weeks, which is not what anyone hoped to hear and is frequently the correct move.
The definition is contested and nobody will decide. If two departments hold incompatible definitions of the central word in the question, and no executive will choose between them, the technical work cannot land. Every number produced will be wrong in one of the two frames, and the project will die in review rather than in engineering.
The mistakes we see most
- Funding a full cleanup before naming a question, so completeness has no definition and the program has no end
- Cleaning the extract by hand and evaluating on data the production system will never see
- Overwriting source values, so nobody can audit or reverse a correction
- Deleting duplicates that were events, and quietly changing reconciled totals
- Imputing missing values that were missing for a reason, and hiding the reason
- Ignoring the shadow spreadsheet where the actual process lives
- Buying a platform before profiling the tables it is supposed to hold
- Treating identity matching as a preprocessing step rather than the largest task in the project
When you do not need us
Three situations where the right answer is to do it yourself, and we would tell you so.
If the question can be answered from one system with a query, write the query. A surprising share of requests that arrive framed as data projects are one join and a filter away from being answered by an analyst in an afternoon, and hiring an outside firm to do that is an expensive way to get an afternoon.
If the problem is that two departments disagree about a definition, that is a management decision and it costs nothing but a meeting and a memo. Bringing in engineers to arbitrate a definition produces a technically elegant version of an unresolved argument.
If you have a competent analyst and the main obstacle is that nobody has given them a week of uninterrupted time and access, give them the week. That experiment costs one week and it frequently ends the conversation about hiring anyone, which is a good outcome for you and an honest one for us to recommend.
Before you fund a data program
- The question is written down precisely enough for two people to compute it the same way
- The tolerance is stated: how wrong can the answer be and still be useful
- All six conditions have been checked against that question, on the real tables
- The identity match rate between systems has been measured, not assumed
- An access path exists that could run in production, not a favor from an analyst
- Corrections are planned for the path, with raw values preserved alongside
- The shadow spreadsheets have been found and read
- Someone with authority has settled any contested definition, in writing
- The first shipped output is weeks away, not quarters
Bottom line
Your data is not ready for every question, which puts you in the company of every organization we have ever worked with. It is very likely ready for a narrower version of the question you asked, and finding that version is a two-week exercise rather than a two-year program. Check existence, coverage, identity, timeliness, ground truth and access against one precisely stated question. Fix what that question needs, in the path rather than in the source, keeping the raw values. Ship something small, watch what breaks, and let the real failures tell you where the next cleanup should go. The companies that get value out of their data are not the ones with clean data. They are the ones that shipped something narrow early enough to learn which mess actually mattered.
Frequently asked questions
Usually not first, and not for one project. A warehouse is the right investment when several teams need consistent access to the same definitions repeatedly. For a single question, a defined extract and a version-controlled transformation layer gets you an answer in weeks and tells you what the warehouse should actually contain. Building the warehouse first commonly produces a well-modeled home for data that nobody has yet proved they need.
Less clean than most people expect on formatting and inconsistency, and far more consistent than most people expect on labels and identity. Models tolerate noisy inputs reasonably well. They do not tolerate a target variable that means different things in different years, or a training set where the same entity appears as three different entities. Spend the cleanup budget on identity and on the definition of the thing being predicted.
Yes, and you should, with one condition: the sample must come through the same path the production system will use, unedited. A hand-curated sample answers a question about a dataset that will never exist again. A random sample pulled through the real path, including the malformed records, tells you what you are actually going to face.
Almost always with definitions rather than with engineering. Take one specific number, have each side write down exactly how they compute it, and compare. In our experience the great majority of these disputes resolve to two defensible definitions of a word like “active” or “booked” rather than to a broken pipeline. Pick one, name an owner, and keep both numbers visible for a while so people can see the bridge between them.
Typically $20,000 to $60,000 for two weeks of outside work, or two weeks of an internal engineer's time if you can free one. It should end with a written verdict on each of the six conditions, a measured identity match rate, a profile of the relevant tables, and a working extract pulled through an automatable path. If a proposed assessment does not produce those, it is a discovery meeting with a longer invoice.