The mistake is treating it as a data problem
A team decides its product needs government data. Someone finds the relevant API, reads the documentation, and writes a working prototype in an afternoon. That prototype is real. It is also the cheapest hour anyone will ever spend on the project, and it sets an expectation that the rest of the work will be proportional. It will not be. The prototype tested whether the data can be fetched. The remaining cost is entirely about whether it can be depended on, and nothing in the first afternoon touches that question.
The failure is predictable enough to describe in advance. The pipeline runs for four months. Then a quarterly file arrives with two columns renamed and one code list extended, and every downstream aggregate shifts by a few percent without erroring. Nobody notices for six weeks because the numbers still look plausible. A customer notices first. That is the characteristic failure of public data in a commercial product: not an outage, which you would catch, but a silent semantic drift in a source you cannot control and did not instrument.
So the useful framing for a build decision is not "is the data available." It is available. The question is what recurring engineering cost you are signing up for, and that cost is a function of four things: how the publisher exposes the data, how often the schema moves, whether the licensing supports what you intend to do commercially, and whether anything about your intended use crosses into a regulated category. Those four are what this piece is about.
Where the hours actually go on a government data integration
Relative weighting from delivery practice, not a measured statistic — shown to rank where attention belongs.
Read that ranking against the prototype. The afternoon that felt like progress covered the bottom bar. Everything above it is the actual engagement, and almost all of it is recurring rather than one-time.
Four distribution patterns, and only one of them is cheap
Public data reaches you in one of four shapes, and which shape you are dealing with predicts your cost structure more reliably than the subject matter does.
A documented query API with a stable schema. The Census Bureau's API is the reference example: versioned by dataset and vintage, with a machine-readable variable list per endpoint, so a schema change arrives as a new vintage rather than as a mutation of the old one. USAspending publishes an OpenAPI-described interface over federal award data. FDA's openFDA covers drug, device and food endpoints with documented fields. When you get this, the integration is close to what your team already knows how to build.
A bulk extract that is the real product, with an API bolted on for convenience. This pattern is more common than the documentation suggests, and missing it is the most expensive routine mistake in this domain. SAM.gov's entity and opportunity data, SEC EDGAR's full-text and structured filing archives, and most state licensing boards fall here. The API exists, carries a rate limit sized for interactive lookups, and will take weeks to walk if you use it to build your initial corpus. The bulk file next to it, often less prominently linked, holds the same content and downloads in an hour. If you find yourself planning a multi-day paginated crawl, stop and go look for the bulk extract. It is almost always there.
A harvested catalog that points at files of wildly varying quality. Data.gov is an index, not a store. It aggregates DCAT-US metadata records that agencies publish about their own holdings, and the record's quality is entirely the publishing agency's. A catalog entry may point at a live service, a static CSV last touched in 2019, or a dead link. State and municipal portals on Socrata, CKAN and ArcGIS Hub work the same way. Budget for the catalog itself being a dataset you have to validate before you can trust any single row of it.
A PDF, or a system with no programmatic interface at all. Some of the most commercially valuable material is here: inspection reports, adjudications, enforcement actions, technical evaluations. There is no cheap path. Extraction from this shape is its own project with its own accuracy target, and treating it as a preliminary step to the real work is how a six-week estimate becomes six months.
What the free data actually costs
Useful ranges, for a single source of moderate complexity feeding a production system. A clean API-backed source with a stable schema runs roughly 60 to 120 engineering hours to reach production quality, meaning validated, monitored, documented and reproducible rather than merely working. A bulk-extract source with quarterly releases and a code list that moves runs 150 to 300 hours. A source requiring extraction from documents starts around 400 hours and is bounded mostly by the accuracy you are willing to accept.
The number that governs your budget is not any of those. It is maintenance, which for public sources tends to run 15 to 25 percent of the original build annually, per source, indefinitely. That is the line people omit. A team integrating twelve public sources is signing up for something in the range of a permanent half-time engineer just to keep what it already built from quietly going wrong. Knowing that number before the commitment changes which sources you take and how many.
Two costs sit outside those ranges and are worth pricing separately. Backfill is one: reconstructing a consistent historical series across vintages where definitions changed is frequently harder than ingesting the current release, because you are reconciling a publisher's decisions rather than parsing their output. Reprocessing is the other: when a publisher revises prior periods, and many statistical agencies do so routinely, your stored history is wrong until you re-run it, so the pipeline has to be built for replay from the start. Retrofitting replay into a pipeline that assumed append-only is close to a rewrite.
The rate limit is a design constraint, not an obstacle
Federal APIs are sized for interactive use. A typical key allows something in the hundreds to low thousands of requests per hour, and the limit is usually per key rather than per IP, so parallelizing across machines buys nothing. Teams meet this constraint and reach for the wrong three answers: request a limit increase, which is slow and often refused; distribute the crawl, which does not help; or accept partial data, which quietly compromises the product.
The right answer is to change what you are asking the API for. Use bulk extracts for the corpus and reserve the API for the delta, which is what both are designed for. Where a bulk extract genuinely does not exist, cache aggressively against a content hash and re-request only what changed, and treat your cache as the system of record with the API as a refresh mechanism. This inverts the naive design, where the API is the source and the cache is an optimization. The inverted version is both faster and more resilient, because it keeps working when the upstream service is down, which it periodically will be.
One operational note that saves real money: many federal data services are noticeably slower during business hours in Eastern time and during the last week of the fiscal year, when agency systems are under internal load. Scheduling heavy pulls outside those windows is a free reliability improvement.
Licensing is not the same question as availability
Federal government works are generally not subject to domestic copyright under 17 U.S.C. 105, which is why most federal data can be redistributed and built on commercially. That covers a great deal, and it is why the licensing question feels settled to teams that have only worked with federal sources.
It is not settled at the state and local level, and the inconsistency is a real risk if you assume otherwise. Municipal catalogs frequently carry no machine-readable license identifier at all, which means the governing terms are whatever the site-wide terms of use say, and those vary. Where an identifier is present it is often a public domain dedication, but attribution and share-alike licenses do appear, and a share-alike term reaching a derived database is a genuine product constraint rather than a footnote. Read it before you build a commercial offering on top of it, not after.
Three sources of restriction survive even where copyright does not apply, and each has caught out teams that checked only the license field. Some federal datasets carry contractual terms from an underlying commercial provider, so the government's right to publish does not transfer to you. Some carry statutory use restrictions independent of copyright, most visibly the Driver's Privacy Protection Act on motor vehicle records and the Fair Credit Reporting Act on anything that functions as a consumer report regardless of what you call it. And some carry redistribution terms attached to how you received the data rather than to the data itself.
The practical discipline is small and pays for itself the first time a customer's counsel asks. Record the source URL, the retrieval timestamp, the license string as published, and the file hash for every dataset you ingest, inside the artifact rather than in a side note. This is perhaps a day of work to build and it converts an unanswerable question into a query.
Relative integration cost by distribution pattern
Relative weighting from delivery practice, not a measured statistic — shown to rank where attention belongs.
Joining public data to your own is the hard part
The join is where most of the value is and where most of the schedule goes. Government records identify entities by identifiers your systems do not carry: a UEI for a registered entity, a CAGE code, an NPI for a healthcare provider, an FEI for a regulated establishment, a facility identifier for a permitted site. Your customer table has a name and an address. Matching one to the other is entity resolution against a source you cannot correct, which behaves differently from the internal version of the same problem.
Internal resolution converges, because you fix the bad records and the fixes persist. External resolution does not converge. The source refreshes, reintroduces the variants you resolved, and occasionally reassigns its own identifiers without announcement. So the design that works is a maintained, versioned mapping rather than a one-time cleanup producing a merged table, and the match has to be stored as its own object with the supporting signals, a confidence value, the source versions on both sides, and any human adjudication attached. Merging into a golden record discards the evidence and makes every future dispute unanswerable.
Budget the ongoing adjudication explicitly. A meaningful share of matches in any cross-organizational resolution will sit in a band where the automated decision is not trustworthy, and someone has to look at them. Teams that plan for this ship. Teams that assume the matcher will handle it discover the queue in production, usually at the same moment a customer disputes a number.
When your use crosses into a regulated category
Some intended uses change the compliance picture regardless of how public the input was. If your product scores individuals for credit, employment, insurance or housing decisions, you are likely operating a consumer reporting function under FCRA and the source's public status is irrelevant to that. If you touch protected health information, HIPAA governs your handling even where the underlying statistical release was open. If you re-identify individuals from combinations of public releases, the fact that each input was published separately is not a defense.
Two adjacent categories catch commercial teams working near defense. Data that is public today can still be export-controlled in aggregate or in combination under ITAR or EAR, particularly for technical data about controlled items. And if you take on government work directly, the relevant frameworks become concrete requirements rather than background: NIST SP 800-171 for controlled unclassified information on a non-federal system, NIST SP 800-53 as the control catalog behind an authorization, and FedRAMP where a cloud service is offered to agencies. None of these is triggered by reading public data. All of them are triggered by what you do next, and they are far cheaper to design for than to retrofit.
If you are also putting a model in front of any of this, the governance frameworks worth reading early are the NIST AI Risk Management Framework, ISO/IEC 42001 for a management system, the OWASP LLM Top 10 for application-layer threats, and MITRE ATLAS for adversarial technique coverage. In financial services, SR 11-7 model risk management expectations will reach the model whether or not its inputs were public.
How to scope this before you commit
A short, bounded assessment answers the build decision far better than a longer discussion does, and it is cheap enough to run before anyone commits a roadmap slot. It takes about two weeks for a typical multi-source integration.
Pull one full release of each candidate source, not a sample, and diff it against the prior release to see what actually moved. That single measurement tells you more about your maintenance cost than any documentation will. Attempt the join against a real extract of your own records and measure the match rate honestly, including the ambiguous band. Read the license and terms for each source and write down the answer with its retrieval date. Then build one end-to-end slice, from ingest through the join to a single output number, and check that number against a published figure you did not compute.
The output is a decision with numbers behind it: which sources are worth the recurring cost, which are not, what the first year looks like, and what the steady state looks like. That is a defensible input to a roadmap conversation in a way that a working prototype never is, because the prototype only ever proved the cheap part.
What separates a working pipeline from a dependable one
Four properties, and they are what we build toward on this kind of work. Every dataset carries its provenance in the artifact: source, retrieval time, license, hash. Every refresh is validated against the prior release with an explicit schema contract, so a renamed column stops the pipeline instead of silently changing an aggregate. The whole history is replayable from raw stored inputs, so a publisher's revision is a re-run rather than an investigation. And every match is reviewable, with the evidence retained, so a disputed number has an answer.
None of that is exotic. It is the difference between a pipeline that works on the day it ships and one that still produces defensible numbers eighteen months later, after two schema changes, a revision to prior periods, and a customer asking where a specific figure came from. The first is an afternoon. The second is the engagement.
Bottom line
Government data is genuinely free to acquire and genuinely expensive to depend on, and the cost lands almost entirely in the parts a prototype does not touch: schema drift, revision handling, entity resolution, provenance, and the regulated status of what you build. Price the maintenance before the build, find the bulk extract before you write the crawler, record provenance from the first ingest, and check whether your intended output falls into a regulated category regardless of how open the input was. Teams that do this ship products that hold up. Teams that skip it ship a demo and then spend a year discovering the bill.
Frequently asked questions
Usually yes. Federal government works are generally not subject to domestic copyright under 17 U.S.C. 105. Check three exceptions: datasets containing licensed commercial content, statutory use restrictions such as DPPA or FCRA that apply to your use rather than the data, and terms attached to how you received it. State and local data is a separate question with no national default.
For one moderately complex source reaching production quality: roughly 60 to 120 hours for a clean documented API, 150 to 300 for a bulk extract with a moving schema, 400 and up where extraction from documents is required. Then 15 to 25 percent of the build annually, per source, for maintenance. The maintenance line is the one that governs a multi-source budget.
Almost always because the API is being used for a job the bulk extract is meant for. Rate limits are typically per key and sized for interactive lookups, so parallelizing does not help. Load the bulk file for the corpus and use the API only for the delta.
Through a maintained, versioned mapping rather than a one-time merge. Government identifiers such as UEI, CAGE, NPI and FEI will not exist in your systems, and the external source refreshes without regard for your prior cleanup. Store each match as its own object with signals, confidence, source versions and any human adjudication, and plan for an ongoing review queue.
Not by itself. Your use can. FCRA reaches anything functioning as a consumer report, HIPAA reaches protected health information, and export control can reach aggregations of individually public technical data. Frameworks like NIST 800-171, 800-53 and FedRAMP apply when you take on federal work or serve agencies directly, and cost far less to design for than to retrofit.