Free data is not cheap data
Somebody on your team found the dataset. It is published on a schedule, it goes back fifteen years, the license is permissive, and the download is a few gigabytes. The business case wrote itself in an afternoon: we score the entities in that file, we sell the score, the underlying data costs nothing. Then the build starts, and eight months later the model is fine and the product still cannot answer a customer's question about a specific company, because the file does not actually say which company it means. That is the shape of this failure, and it repeats across FDA adverse events, SAM.gov and USAspending, CMS provider and claims files, EPA ECHO, FAA service difficulty reports, NHTSA complaints, OSHA inspections, and every state analog of each.
The acquisition cost of federal records is close to zero. The cost of making them decidable is the entire project. A useful rule from work on files of this kind: budget one dollar of engineering for every zero dollars of data, and expect sixty to seventy percent of the total build hours to land before anyone trains a model or draws a chart. Teams that budget the reverse are the ones that show up eighteen months in with a demo that works on the five records they hand-picked.
This piece is about where those hours go. It assumes you are a director or VP with a real budget, a product that needs to ship, and enough experience to be suspicious of anyone who says the data part is straightforward.
Where the hours actually go on a federal-records build — our weighting
Weights sum to 100. Our ordering from builds on public federal and state record sets. The ordering is the useful part, not the decimals.
Read the record layout before you write the business case
Every serious federal file publishes a record layout, a data dictionary, or a submission specification. It is usually a PDF, it is usually dull, and it usually contains the sentence that kills or saves the product. Two hours with that document ahead of the pitch is the highest-return time on the whole project.
What you are looking for is not the field list. It is the rules of submission. Who is required to report, under what trigger, on what deadline, and what happens when they do not. FDA's adverse event system takes voluntary reports from consumers and clinicians alongside mandatory reports from manufacturers, and those two streams behave nothing alike. OSHA inspection records reflect where inspectors went, which is driven by complaints and by a targeting program, not by where hazards are. USAspending reflects what a contracting officer entered into an award system, on a lag, with modifications that restate history.
The practical consequence is that most of these files measure reporting behavior at least as much as they measure the world. A rise in reported events can be a rise in events, a change in a submission form, a new required field, an enforcement campaign, or one large filer changing a workflow. A product that reads all five as the same signal will be wrong in public, in front of a customer, on a specific named entity, which is the only kind of wrong that matters commercially.
The identifier problem is the whole problem
Federal record systems were built to administer a program, not to be joined to your customer list. So the same company appears as a manufacturer name typed by a hospital clerk in one file, a UEI in another, an EIN nowhere, a facility ID that belongs to a plant rather than a corporate parent in a third, and a slightly different legal name in a fourth after an acquisition that the older records never learned about.
Getting this right is not a fuzzy-match library call. It is a system with four parts. A blocking strategy so you compare a few million candidate pairs instead of a few trillion. A scoring function over name, address, geography, and behavior. A threshold policy with a deliberate middle band that no automated rule decides. And a human review queue with an audit trail, because on the entities your customers care most about, the accuracy you need is higher than any unsupervised method delivers.
Expect this to be six to fourteen weeks of a two-person team on a first build, and expect it never to be finished. Corporate structure changes continuously. Plan for a standing quarterly reconciliation, not a one-time cleanse. On the tooling side, Splink and dedupe are real and worth using; the reason the work is still expensive is that the labeled training pairs and the review policy are yours to produce, and nobody can hand them to you.
Once you can tell that two rows are the same facility, you still have to decide what a customer means by "the company." A rating on a legal entity, a rating on a manufacturing site, and a rating on a corporate family are three different products with three different defensibility profiles, and a buyer will assume you built the one they wanted.
Roll-up is where a decision tool quietly loses trust. Attribute a subsidiary's violations to a parent that acquired it two years after the fact, and you will get a legal letter. Fail to attribute them, and a risk analyst will notice within a month and tell you the product misses the obvious. Neither answer is universally right, which is why the rule has to be written down, dated, visible in the interface, and applied the same way every run. Pick an as-of convention, state it on the screen, and give the user a control to see it the other way.
The archive will not match the new files
Fifteen years of history is the asset that makes the product worth buying and the thing that consumes the schedule. Federal submission formats change. Code sets get revised. A field that was free text becomes an enumeration. A form adds a question in 2019 that makes every prior year structurally missing, not zero. Facility identifiers get renumbered. A file that used to arrive quarterly starts arriving monthly with a different key.
Two engineering commitments hold this together. First, keep the raw file exactly as it landed, forever, with its checksum and retrieval timestamp, and treat every transformation as code that runs against it. Storage is a rounding error against the cost of a reload you cannot reproduce. Second, version the transformation and stamp each output row with the code version that produced it. When a customer asks in March why a score changed since November, you either can replay November or you cannot, and that answer decides whether an enterprise contract renews.
| What changes in the source | What it does to a naive build | What handles it |
|---|---|---|
| A field becomes an enumeration Free text in 2016, coded list in 2021 | Trend breaks at the boundary and looks like a real-world shift | Map old values forward in versioned code, and flag the boundary in the interface |
| A new required question Added mid-series | Prior years read as zero rather than unknown | Three-state nulls throughout: absent, reported empty, reported zero |
| Restated or amended records Same event, later correction | Yesterday's answer silently changes and nobody can explain it | Append-only history with effective dates, never an in-place update |
| Identifier renumbering Facility or award keys reissued | One entity splits into two, or two merge into one | A crosswalk table you own, with the evidence for each mapping |
| Publication cadence change Quarterly becomes monthly | Ingest job fails quietly or double-counts a period | Expected-arrival monitoring, and a load that is idempotent by period key |
Say what the data does not cover
Every one of these files has a shape of absence, and the absence is the part that gets a product sued or dropped. Small employers below a threshold do not report. Some categories are exempt. Certain facilities are covered by a state program that reports differently or not at all. Records under investigation may be withheld until the investigation closes. Redactions under FOIA exemption 4 remove exactly the commercially interesting field.
Write the coverage statement early, in plain language, and put it in the product rather than in an appendix. "This score reflects federally reported inspections only. Facilities in twenty-one state-plan states report through a state program with different thresholds, and coverage there is partial." That sentence costs you nothing in a sales conversation with a serious buyer and it saves the product from the single worst outcome, which is a customer discovering the gap on their own after they relied on it.
The counting discipline behind that sentence is a denominator. A raw count of events per company is a measure of size. What a user wants is a rate, and a rate needs a base the source file usually does not carry: employees, units shipped, patient days, flight hours, revenue. Building that denominator from a second source is frequently the hardest analytic step in the project and is the thing that separates a real product from a leaderboard of large companies.
Build the explanation screen before the model
The user of a decision tool has one question after the score, and it is always the same: why. If your answer is a feature-importance plot, you have not answered it. The answer that works is the underlying records, listed, with dates and identifiers, and the arithmetic that turned them into the number.
Design that screen first, because it constrains everything upstream. It means the score has to decompose. It means you have to carry the record grain all the way through instead of loading a pre-aggregated summary. It means the entity resolution has to be inspectable, since the first thing a challenged customer says is that one of those records is not theirs, and they are right about ten to twenty percent of the time on the disputes that get raised.
It also means you need a dispute path with a service level. Somebody reports a mismatch, a person looks at it inside a stated number of business days, the correction lands in the crosswalk you own, and the affected scores recompute with a note in the history. That loop is a product feature, not a support cost, and it is the mechanism that converts your entity-resolution debt into an asset nobody else has.
Governance if a model makes the call
If the output influences a decision about a person, a loan, a claim, or a supplier, three frameworks are worth reading before you design rather than after. NIST AI RMF gives you the Govern and Map vocabulary that federal and enterprise buyers already recognize, and its Map function is essentially a demand that you write down the intended use and the out-of-scope uses. ISO 42001 gives you a certifiable management-system shell if a customer's procurement team asks for one. SR 11-7, written for bank model risk, is the oldest and most practical of the three: independent validation, documented conceptual soundness, ongoing monitoring, and a clear owner.
If the tool uses a language model anywhere in the chain, add OWASP LLM Top 10 for the application-layer failures and MITRE ATLAS for the adversarial ones. The realistic threats on a records product are prompt injection through ingested document text and data poisoning through a public submission channel, and both are cheap to test for and expensive to discover in production.
None of this requires a certification to start. It requires that the documentation exists, is dated, and matches what the system actually does. A validation memo that describes a model you replaced last quarter is worse than no memo.
The requirements change shape entirely the moment an agency wants to host the tool or feed it non-public records. Controlled Unclassified Information puts you under NIST 800-171 and, on a defense contract, CMMC assessment. A system running in a federal boundary needs an authorization built on NIST 800-53 controls, and if it is delivered as a cloud service, FedRAMP. Those are program-length efforts measured in quarters and hundreds of thousands of dollars, not features you add at the end.
The design decision that saves the most money is a boundary drawn early. Keep the public-records processing in your commercial environment, where it belongs, and put only the part that touches sensitive government data inside the controlled environment. Teams that skip that separation end up dragging a fifteen-year public archive and its whole toolchain through an authorization process it never needed to enter.
Why a records product loses a customer — our ranking by how often it is the cause
Our ordering from products we have been called in to repair after launch. Note the last row: the model is rarely the thing that lost the account.
What it costs and how long it takes
For a single federal record set, one entity type, a defensible score, an explanation screen, and an API, a realistic first release is four to seven months and somewhere between $250,000 and $700,000 of engineering. The wide range is almost entirely entity resolution: a file with a clean, stable, government-issued identifier lands at the bottom, and a file where the entity is a typed manufacturer name lands at the top or above it.
Two or three sources joined together roughly doubles it, because the join is a new resolution problem rather than an addition. Running cost is modest and predictable, usually $4,000 to $15,000 a month in compute and storage for a set of this size, plus the standing human review that never goes away.
The schedule risk is not technical. It is the week in month three when the client's subject-matter expert looks at fifty resolved entities and rejects six of them, and the threshold policy has to be rebuilt. Plan that review into the schedule as a milestone with a date. Teams that discover it late lose a month; teams that plan it lose a week.
How to tell whether a firm has done this before
Ask four questions in the first meeting, and listen for whether the answers are specific.
Ask what the entity is and how they will know two rows are the same one. A firm that has done it will start talking about blocking keys, a labeled sample, and a review queue within a minute. A firm that has not will say they will use fuzzy matching.
Ask what happens when the source publisher changes the file format. The answer you want involves raw retention, versioned transformations, and replay. The answer you do not want is that they will handle it when it happens.
Ask what the tool will say about the entities it has no data on, which is the majority of them in most of these files. Absence handled as absence is a sign of maturity; absence rendered as a low score is a lawsuit.
Ask them to walk one specific real record, end to end, from the published file to the number on the screen. This is the question that separates people who have read the record layout from people who have read a description of the dataset. Give them a company name and a date and watch what happens.
Bottom line
The records an agency already has are a genuine commercial asset, and the reason more products are not built on them is not that the data is hard to get. It is that turning administrative records into a defensible statement about a named company requires identity work, coverage honesty, versioned history, and an explanation the customer can audit. Those four things are the product. The score is the easy part, and the team that treats it as the hard part will ship something nobody trusts twice.
Frequently asked questions
USAspending and SAM.gov for contracting, CMS provider and utilization files for health, FDA adverse event and enforcement data, EPA ECHO for environmental compliance, OSHA inspections, NHTSA complaints and recalls, and FAA service difficulty reports are all published, documented, and licensed for reuse. Usability varies almost entirely by whether the file carries a stable entity identifier.
Six to fourteen weeks of a two-person team when the entity is a company name typed by a third party. Considerably less when the file carries a UEI, NPI, or facility ID that the publisher maintains. It is a continuing quarterly cost either way, because corporate structure keeps changing.
No. Public records processed in your own commercial environment carry no federal authorization requirement. FedRAMP, NIST 800-53, and 800-171 enter the picture only when a federal customer hosts the system or supplies non-public data, which is a reason to draw that boundary deliberately and keep the public pipeline outside it.
Not at the accuracy a commercial buyer expects on named companies. The model handles the confident majority; the deliberate middle band is where the disputes and the reputational risk live, and reviewing it is cheaper than defending a wrong attribution.
Compute and storage for a single large record set typically runs $4,000 to $15,000 a month. The larger standing cost is the periodic reconciliation of entity mappings and the dispute-handling loop, which is people rather than infrastructure.