The stall that does not look like a stall
A federal AI effort rarely dies loudly. It dies in week nine, when the dataset that the kickoff meeting described as "available" turns out to need four signatures, one of which belongs to a privacy officer who has not yet heard the project exists. Nothing is formally late. The engineering team keeps building, because there is always infrastructure to build. Status reports stay green. Then the period of performance closes and the final report has to explain why the accuracy figures came from four hundred records someone exported by hand into a spreadsheet.
This is the most common failure mode in applied federal AI work, and it is almost never described that way afterward. The retrospective says the scope was too ambitious, or the problem was harder than expected, or the model did not generalize. Underneath those explanations, the same fact usually sits: the team had roughly one-third of the data-months it planned for, and every downstream decision was made under that compression.
The useful thing about this failure is that it is structural. It is not caused by an unhelpful government team, and it is not fixed by pushing harder on email. It is caused by a project plan that treated a multi-party legal and security approval as a task rather than as a dependency chain with its own calendar. Chains like that can be scheduled, priced, and hedged, once you can see the whole thing.
Where the schedule risk actually sits
Ask a technical team where a data-access effort will slip and most will say "the extract." Ask anyone who has run one and the answer is different. The engineering work of writing a query, pulling a date range, and shipping a file is usually a few days of one person's time. The delay lives entirely in the approvals in front of it, and in the human-onboarding process running in parallel that nobody put on the Gantt chart at all.
Where Data-Access Schedule Risk Concentrates
Editorial weighting from public guidance and practitioner reading, not a measured statistic.
Who actually has to say yes
The sponsor who wrote the requirement is not the person who releases the data. That is the single most expensive misunderstanding in this whole area. A program office can want an AI capability badly, fund it, and still have no authority to hand over a single row.
The release decision is typically distributed across five or six roles, each answering a different question, each with its own queue. They do not coordinate with one another by default. If nobody on the project explicitly runs the chain, the requests sit in five inboxes in serial rather than in parallel, and the elapsed time becomes the sum instead of the maximum.
| Approver | The question they are answering | What unblocks them |
|---|---|---|
| Data / system owner | Is this use consistent with why we collected the data, and will the extract disrupt the production system? | A written description of the exact tables, fields, and date range, plus an off-hours extract window. |
| Privacy office | Does the Privacy Act system-of-records notice already cover this use, and is a privacy impact assessment required or current? | The published SORN citation and a purpose statement that fits an existing routine use. |
| Security (ISSO / ISSM, AO) | Where will the data live, is that boundary authorized for this sensitivity level, and how does it move? | A named destination environment with an existing ATO or FedRAMP authorization, and a transfer method. |
| Counsel | What does the agreement obligate us to, who is liable on a breach, and who owns what comes out? | A data use agreement the agency has signed before, edited rather than drafted fresh. |
| Records / program compliance | Is this a records-schedule question, a human-subjects question, or a re-disclosure question? | A clear statement that the work produces a model and a report, not a new system of record. |
| Contracting officer | Is the government furnishing property or information, and does the contract say so? | Government-furnished data listed as an attachment, with a delivery date in it. |
Two of those rows do more work than the rest. The privacy question is governed by the Privacy Act of 1974 (5 U.S.C. § 552a), which permits disclosure of records about individuals only under an enumerated condition, most often a published routine use. If the proposed analysis does not fit an existing routine use, the fix is a modified notice published in the Federal Register, and OMB Circular A-108 sets the process for that, including advance notice to OMB and Congress and a public comment period on new routine uses. That is a multi-month path. Discovering it in month three is the difference between a delivered model and a lessons-learned memo. Section 208 of the E-Government Act of 2002 adds the privacy impact assessment requirement when an agency develops or procures IT that handles identifiable information, which is a shorter path but still a queue.
The security question is governed by where the data lands. If the destination is a contractor environment, the practical bar for controlled unclassified information is NIST SP 800-171, which agencies apply to nonfederal systems under 32 CFR Part 2002; for DoD work the flow-down clause is DFARS 252.204-7012, which also carries a 72-hour cyber incident reporting obligation and a FedRAMP Moderate equivalence requirement for cloud. If the destination is a government environment, the bar is different and usually easier, and the constraint moves to getting people credentialed into it.
PHI, PII, and what de-identification really buys
When the data includes health information, a second regulatory chain opens. Protected health information under HIPAA is governed by 45 CFR Parts 160 and 164, and a contractor handling PHI on a covered entity's behalf is a business associate, which requires a business associate agreement meeting the elements at 45 CFR 164.504(e). That agreement is separate from the contract and separate from the data use agreement, and it has its own review path.
The rule offers three doors, and choosing the wrong one costs months. Safe Harbor de-identification at 45 CFR 164.514(b)(2) removes eighteen categories of identifiers, including all geographic subdivisions smaller than a state, all date elements more specific than year, and ages over 89. Data that passes Safe Harbor is no longer PHI and the transfer becomes far simpler. Expert determination at 164.514(b)(1) lets a qualified statistician certify that re-identification risk is very small, which preserves more analytic value and takes longer to arrange. The limited data set at 164.514(e) keeps dates and geography down to town or ZIP, still counts as PHI, and requires its own data use agreement.
Here is the part that matters for modeling. Safe Harbor destroys temporal resolution. If the analytic question is about sequence, timing, or intervals between events, year-only dates will not answer it, and a team that accepts a Safe Harbor extract to move faster has quietly changed the problem. The right move is to decide the door before the schedule pressure arrives: write down which fields the model genuinely needs at what granularity, and let that drive whether to pursue expert determination or a limited data set rather than discovering the loss after delivery.
Other data classes carry their own gates. Federal tax information falls under IRS Publication 1075, which requires a safeguard security report and advance notification to the IRS Office of Safeguards before certain changes in how the data is handled. Student records fall under FERPA (20 U.S.C. § 1232g, 34 CFR Part 99), where the studies exception at 34 CFR 99.31(a)(6) requires a written agreement with specific terms. Criminal justice information falls under the FBI CJIS Security Policy, which requires fingerprint-based screening for every individual with access. None of these are unusual. All of them add weeks that a naive plan does not have.
The gate nobody schedules: getting a person in the door
Even when the data is cleared, a human still has to touch it, and that human needs credentials. Under HSPD-12 and FIPS 201, a contractor working on a federal site or in a federal system needs a favorable fitness determination before a PIV credential is issued. For low-risk positions that means a Tier 1 investigation: fingerprints, an electronic application, and adjudication. The investigation itself is often not the slow part. The slow part is scheduling fingerprinting, waiting on a sponsor to initiate the package, and the handoffs between the program office and the security office.
Two practical consequences follow. First, start the credentialing package at award, not when the data agreement is signed, because the two tracks are independent and running them in serial adds a month or more for no reason. Second, name the specific individuals in the proposal. A team that says "our data scientist" gives the security office nothing to act on. A team that names the person, states their citizenship or lawful permanent resident status where relevant, and confirms that the paperwork can be submitted in week one is describing a schedule the government can believe.
A realistic timeline
The steps below overlap, and a mature agency with a template agreement and a warm privacy office moves faster than this. A first-time requester at an agency that has not released this dataset before should plan for the long end. The important number is not any single row. It is the total, and the fact that a six-month effort cannot absorb it without a bridge.
Typical elapsed time, first-time data release
Run well and in parallel, that lands somewhere around ten to sixteen weeks. Run in serial with a two-week reply latency at each hop, it becomes six to nine months. The difference between those two outcomes is almost entirely project management, and it is worth pricing explicitly rather than absorbing quietly.
Bridge data: synthetic, surrogate, and public analogues
A team that cannot start until the real data lands has given away its most valuable months. The alternative is to build against a stand-in that has the same shape, so that the day the real extract arrives the pipeline already runs and the only open question is whether the numbers hold. Three kinds of stand-in are worth distinguishing.
Public analogues. Real data, different population. Health work has MIMIC-IV on PhysioNet, which requires human-subjects training and a credentialed-access agreement but is genuine clinical data, plus the CMS synthetic public use files for claims structure. Transportation work has NHTSA's Fatality Analysis Reporting System and the Bureau of Transportation Statistics releases. Federal spending work has USASpending.gov bulk downloads. Population work has Census PUMS microdata. These are the strongest bridge because the messiness is real.
Structural surrogates. Generators built to mimic a domain's schema and clinical or operational logic. Synthea, the open-source synthetic patient generator from MITRE, is the widely used example. Surrogates get the joins, the cardinalities, and the null patterns close enough to exercise a pipeline end to end.
Statistical synthesis. Model-based generation from a real sample, using tools such as the Synthetic Data Vault library or a CTGAN-style tabular generator, with differential privacy applied when the sample is sensitive. The Census Bureau's use of a differentially private disclosure avoidance system for the 2020 decennial data is the reference implementation at scale. This is the right tool when a small real sample can be released quickly and a large one cannot.
Ask for a hundred rows before you ask for a million
A small, hand-reviewed validation sample often clears under a much lighter approval than a full extract, sometimes under an existing agreement. It settles schema, encoding, null behavior, and code sets, which is most of what a pipeline actually needs to be built correctly. Make it a separate milestone with its own date, ahead of the full delivery.
What bridge data cannot buy
Honesty about the limits is what keeps this technique from becoming self-deception. Synthetic and surrogate data validate that a pipeline runs, that features compute, that a training loop converges, and that an interface makes sense to a user. They do not validate accuracy on the real distribution. A model that scores well on Synthea output has demonstrated engineering, not performance.
So the reporting language has to stay clean. Results from bridge data get labeled as bridge results in every chart and every table, every time, with the real-data evaluation held as a separate and clearly pending line item. A reviewer who finds a synthetic-data number presented as a performance claim will discount everything else in the document, and they will be right to.
The second limit is distribution shift in the parts that matter most. Rare classes, data-entry idiosyncrasies specific to one office, upstream system migrations that changed a code set in 2019, and the free-text fields where the real signal often lives are the things a generator gets wrong. Plan for the first real extract to break something, and put a two-week reconciliation window in the schedule for it.
How to write a data-access milestone that holds
Most project plans contain a line that reads "Data access obtained" with a date and nothing else. That line cannot be managed, because it names no person, no artifact, and no failure condition. A milestone that works has seven parts.
- A named human who owns the release decision, by name and role, confirmed in writing at kickoff rather than assumed from the org chart.
- The exact data object: source system, tables or feed, field list, date range, and an estimated row count, so both sides can tell whether what arrives is what was agreed.
- The named legal instrument: data use agreement, business associate agreement, interconnection security agreement, or a government-furnished-information attachment to the contract.
- The destination and the transfer method: which authorized boundary the data lands in and how it moves, agreed with the security officer before the request goes forward.
- Two dates, not one: a validation-sample date roughly four weeks ahead of a full-delivery date, each with its own owner.
- A named fallback dataset and a trigger date, written into the plan before anyone is under pressure, so switching to it is a decision that was already made rather than an admission.
- Who does the extract engineering and whose hours it comes out of, since an unfunded government-side task is the quietest way for a delivery date to slip.
Written this way, the milestone becomes reportable. Every status meeting has a specific question with a specific answer: has the named person signed, and if not, what is the named blocker. That is a conversation a contracting officer's representative can act on.
What to do when access slips anyway
It will slip sometimes. Handled well, a slip is a scope conversation. Handled badly, it is a surprise at the final review.
Raise it the week the trigger date passes, in writing, in the monthly progress report, with three things attached: the specific blocker and who owns it, the fallback path the team is switching to, and what the technical outcome will be under the fallback compared to the original plan. Ask for one thing the government can actually grant, whether that is a named escalation, a smaller sample under an existing agreement, or a no-cost extension. A single clear ask lands; a list of grievances does not.
Then re-baseline the work honestly. If the real evaluation cannot happen inside the period of performance, say so early and reshape the deliverable around what can be proven: the pipeline, the interface, the validated design, and an evaluation protocol ready to execute the day data arrives. That is a defensible outcome, and program offices recognize it. What they do not forgive is a final report that describes a synthetic-data result in language that implies it was real.
Bottom line
Data access is not administrative overhead sitting in front of the interesting work. In federal AI it is the critical path, and it deserves the same rigor as the modeling: a named owner per approval, a written request that a privacy officer can evaluate on first read, a de-identification decision made before the schedule forces one, a bridge dataset chosen up front, and a milestone with two dates and a fallback. Teams that plan this way spend their period of performance building. Teams that do not spend it waiting, and then explain.
Frequently asked questions
Because the sponsor usually does not own the release decision. Privacy, security, counsel, and the system owner each answer a different question under different authority, and each has an independent queue. Funding the work does not pre-clear the data.
Yes, and on a short period of performance you generally must. Build against a public analogue, a structural surrogate such as Synthea, or statistically synthesized tables, so the pipeline is running when the extract lands. Label every bridge result as a bridge result.
It can shorten it substantially. Data de-identified under the HIPAA Safe Harbor method at 45 CFR 164.514(b)(2) is no longer protected health information. The cost is analytic: Safe Harbor strips dates below the year and geography below the state, which breaks most sequence and timing work.
A named approver, the exact tables and date range, the named legal instrument, the authorized destination and transfer method, a validation-sample date ahead of the full-delivery date, a named fallback dataset with a trigger date, and whoever is funded to build the extract.
For a first-time release at an agency, ten to sixteen weeks run in parallel is a reasonable plan, longer if federal tax information, criminal justice information, or a Privacy Act routine-use modification is involved. Budget the credentialing track separately and start it at award.