The pilot and the production system are not the same system
A pilot usually runs on a copy. Someone exported a slice of the warehouse, dropped it in a bucket the team could reach, and the model did well enough on it that the meeting went well. Production is a different animal. It reads from the record of authority, it writes back into it, and its outputs start shaping what happens to real people and real money. The model did not change. The obligations attached to the system did, and they attached the moment the data stopped being an extract and started being the record.
This is why so many teams experience compliance as an ambush. Nothing about the technical work was wrong. The pilot proved what it was supposed to prove. But the review that follows is not asking whether the model works. It is asking a different set of questions, and none of them were on the pilot's list: where does this data live now, who else can reach it, what did the system do on the third Tuesday of last month, how do you know it performs the same way for every group of people it touches, and if it was wrong about someone, what do you tell them.
The good news is that this is a finite list. Regulated systems are not mysterious. They are systems where you have to be able to prove things after the fact, to someone who was not in the room. Almost every requirement below is a consequence of that one property.
Two different things get regulated, and teams only plan for one
Regulation attaches at two separate points, and confusing them is the most common planning error we see. The first point is the data. Confidentiality rules govern custody: who may hold the information, under what agreements, with what controls, and for how long. Teams generally see this one coming, because it looks like security work and security work has a budget line.
The second point is the decision. When a system's output determines whether someone gets credit, a benefit, a job interview, a clinical pathway, or a place in a queue, a different body of law turns on. That law does not care much about your encryption. It cares whether the person can be told a specific reason, whether they can contest it, and whether you can show the reason was actually the reason. Teams almost never plan for this, because it does not look like security work at all. It looks like a product feature, and by the time anyone notices, the architecture already made it hard.
Credit is the cleanest illustration because the rule is unusually explicit. Under Regulation B, 12 CFR 1002.9(a)(1), a creditor has to notify an applicant of action taken within 30 days of a completed application. Section 1002.9(a)(2) requires that an adverse action notice contain a statement of specific reasons, or disclose the right to request them. And 1002.9(b)(2) closes the escape hatch: the statement of reasons must be specific and indicate the principal reasons, and a generic reference to internal standards or policies is insufficient. The CFPB restated how this applies to complex algorithmic models in Consumer Financial Protection Circular 2023-03, published September 19, 2023.
Read that as an architectural requirement, because that is what it is. A model that produces a score and nothing else cannot satisfy it. Reason attribution has to be produced by the same computation that produced the decision, at decision time, and stored with it. Bolting on a post-hoc explainer six months later gives you a plausible story rather than the actual principal reasons, and the difference matters when someone disputes it.
| Regime | What it attaches to | The obligation that surprises teams |
|---|---|---|
| HIPAA Security Rule 45 CFR Part 164, Subpart C | Electronic protected health information, wherever it sits | § 164.308(b) requires satisfactory written assurances from every business associate, and requires business associates to obtain the same from subcontractors. The specification at § 164.308(b)(3) is Required, not Addressable, and § 164.314(a) sets what the contract must contain. |
| FERPA 34 CFR Part 99 | Personally identifiable information in education records | § 99.31(a)(1)(i)(B) treats an outside party as a school official only if it is under the institution's direct control with respect to the use and maintenance of the records, and is bound by the redisclosure limits at § 99.33(a). "Direct control" is a contract term you have to actually accept. |
| GLBA Safeguards Rule 16 CFR Part 314 | Customer information held by a financial institution | § 314.4 names the controls individually: a designated Qualified Individual, a written risk assessment, encryption in transit and at rest, multi-factor authentication for anyone reaching an information system, monitoring of authorized user activity, contractual safeguards on service providers, a written incident response plan, and an annual report to the board. |
| Federal tax information IRS Publication 1075 (Rev. 11-2021) | FTI at a recipient agency, its agents, and its contractors | Safeguards required under IRC 6103(p)(4) extend to contractors, and the Office of Safeguards operates a 45-day advance notification process before FTI reaches a new contractor or cloud environment. That lead time sits on the critical path of any deployment. |
| Controlled unclassified information NIST SP 800-171 | CUI handled under a federal contract | Revision 3 was published in May 2024 and supersedes Revision 2. For defense work, the CMMC program rule at 32 CFR Part 170 took effect December 16, 2024, and the DFARS acquisition rule that puts CMMC into contracts was published September 10, 2025 and took effect November 10, 2025. |
| Credit decisions ECOA / Regulation B, 12 CFR 1002.9 | The decision itself, not the data | Specific principal reasons, within 30 days, and § 1002.9(b)(2) says pointing at internal standards or policies does not count. This is a constraint on the model's output format, not on its storage. |
The boundary moves outward, and it moves through your copies
Ask an engineering team where the regulated data lives and you will usually get one answer: the database. Ask where copies of it exist and the conversation gets much longer. Modern AI systems are copy factories, and every copy inherits the classification of its source.
Embeddings and vector indexes. An embedding derived from a medical note is derived from a medical note. Treating the vector store as ordinary infrastructure because it holds floats is a scoping error, and it is the one we see most often. The index needs the same access controls, the same retention rules, and the same deletion path as the source.
Prompt and completion logs. The most useful debugging artifact you have is a verbatim record of regulated content leaving your boundary and coming back. Log retention that made sense for an application server is a liability here, and log destinations that ship to a third-party observability tool are an unreviewed disclosure.
Evaluation fixtures and golden sets. The held-out set someone built to measure quality is a permanent copy of real records, usually stored where the team can reach it easily, which is the opposite of where it should be. It also tends to outlive every other artifact in the project.
Fine-tuning corpora and cached context. Training data persists inside model weights in ways that are hard to characterize and harder to delete. If a deletion request has to be honored, a fine-tuned model built on the deleted record is a problem with no clean technical answer, which is a strong argument for retrieval over fine-tuning in regulated settings.
Error queues and human review interfaces. The triage tool a team builds in a week to look at failures is a full-fidelity window into regulated records, usually with no access model beyond a shared link.
The practical fix is an enclave: one clearly drawn boundary that holds the data, the index, the logs, the fixtures, and the review tooling, with a small number of audited paths in and out. Enclaves are not glamorous, and they are the single highest-value architectural decision in this whole category, because they convert an open-ended question about your entire environment into a bounded question about one system.
Your vendor list became a compliance artifact
The moment regulated data is in play, every service in the request path is a party to the arrangement, whether or not anyone papered it. The model API provider, the vector database host, the observability platform, the queue, the storage layer, and the evaluation tool are all handling the data or its derivatives. Each regime says this in its own vocabulary, and each says it.
- Written assurances that flow all the way down. HIPAA § 164.308(b) requires assurances from the business associate and, in turn, from its subcontractors. A signed agreement with your platform vendor does not cover the inference provider behind it unless the chain is complete.
- A current subprocessor list you have actually read. Vendors publish these and change them on notice. The notice period is the window in which you are supposed to object, and almost nobody uses it.
- Contractual control over training use. Whether your inputs and outputs may be used to improve a vendor's models is a contract term, not a setting, and it should be pinned in writing rather than in a console toggle that a future admin can flip.
- Direct control language where FERPA applies. § 99.31(a)(1)(i)(B) conditions the school official designation on direct control over use and maintenance of the records. A standard commercial terms-of-service does not give an institution that, which is why education deals stall on paper rather than on function.
- Service provider safeguards under the Safeguards Rule. § 314.4 requires financial institutions to contractually require their service providers to implement and maintain safeguards, and to periodically assess them based on risk. "Periodically assess" means you own an ongoing obligation, not a one-time diligence pass.
- Lead time for federal tax information. The 45-day notification process for a new contractor or cloud environment holding FTI is a schedule constraint. Discover it in month one, not the week before go-live.
- Data residency and support access. Where the data sits is one question. Which support engineers, in which countries, can reach the tenant to troubleshoot is a separate question, and it is the one that fails an audit.
- An exit path. Deletion certificates, export format, and how long the vendor retains backups after termination. This gets negotiated at signing or never.
None of this is exotic. It is the same diligence a prime contractor runs on a subcontractor, applied to the software supply chain instead of the labor one, and it takes days rather than months when someone owns it from the start.
Logs stop being telemetry and become evidence
Application logs exist to help engineers. Compliance records exist to answer a question from someone who does not trust you, about an event that happened a year ago, in a form that would survive a challenge. Those are different artifacts with different requirements, and the second one is rarely a superset of the first.
What an evidentiary record has to carry: the exact input, the exact output, the identity of the model and its version, the versions of the retrieval index and any prompt template, the identity of the human who reviewed it and what they saw at the time, the timestamp, and enough integrity protection that the record cannot be quietly edited. The GLBA Safeguards Rule at 16 CFR 314.4 makes the monitoring obligation explicit, requiring procedures designed to monitor authorized user activity and detect unauthorized access or tampering. Most of the other regimes get to the same place through their audit control requirements.
Model version pinning deserves particular attention, because it is the requirement that most often gets discovered late. If a hosted endpoint silently moves to a newer model, and your logs record only the endpoint name, you have lost the ability to say which system produced last quarter's outputs. That is not a minor gap. It undermines every reproduction attempt, every incident investigation, and every claim you make about measured performance. Pin the version, record the version with each call, and treat a version change as a change that requires re-measurement rather than a routine upgrade.
One accuracy number stops being an answer
Pilots report an aggregate. Regulated systems get examined by slice, because harm concentrates in slices. The question is not what your accuracy is; it is how the system performs on the smallest group of people anyone might ask about, and whether you knew.
The second shift is that error types separate. In a pilot, a false positive and a false negative are both errors and often get averaged into one figure. In a regulated workflow they have different consequences, land on different people, and carry different legal weight. A benefits system that wrongly denies is doing something categorically different from one that wrongly approves. Write the error taxonomy before you write the evaluation, and report the two rates separately, with the operating threshold that produced them stated next to the numbers.
What a successful pilot usually already has when the first compliance review starts
Editorial weighting from published regulatory requirements and practitioner reading. Illustrative of where the gaps cluster, not a measured statistic.
The shape of that block is the whole point of this article. The gap is never at the top. Teams arrive at review with the hard technical part finished and the evidentiary part untouched, and the evidentiary part is what the review is about.
Human oversight has to be real, or it makes things worse
"A human reviews every output" is the most common answer to a governance question and the weakest one, because it is easy to say and hard to mean. Oversight counts when the reviewer can see the inputs the model saw, has enough time per item to form an independent judgment, has a genuine ability to override without penalty, and when overrides are recorded and studied.
A review step that approves nearly everything is not a control. It is a record of a control that did not work, and it is now in your logs, which makes it worse than having no step at all. Measure the override rate from the first week. If it sits near zero, either the model is extraordinary or the review is a formality, and the honest way to find out is to seed known-bad items and see whether they get caught.
The compliance calendar is moving, so do not build the plan on a date
Several of the frameworks that govern this work changed in the last eighteen months, and some of them changed direction. Reading the current state accurately is part of the job.
The EU AI Act timeline moved. The Act entered into force August 1, 2024. Prohibited practices applied from February 2, 2025, and general-purpose AI and governance obligations from August 2, 2025. The high-risk obligations then shifted: the AI Omnibus was adopted on November 19, 2025 and entered into force on July 27, 2026, moving the rules for systems in high-risk areas such as biometrics, critical infrastructure, education, employment, and migration and border control to December 2, 2027, and the rules for AI embedded in regulated products to August 2, 2028. Teams that paced their work to the original August 2, 2026 date now have room, and teams that used the delay as a reason to stop have simply moved the same work later.
State AI law has already slipped once. Colorado's AI statute was signed May 17, 2024 with obligations set to begin February 1, 2026. A 2025 special-session bill approved August 28, 2025 moved that date to June 30, 2026. California took a different route: the CPPA's regulations covering automated decisionmaking technology, risk assessments, and annual cybersecurity audits were adopted by the board July 24, 2025, approved by the Office of Administrative Law September 22, 2025, and became effective January 1, 2026. Confirm operative dates with counsel before they drive an engineering schedule, because in this layer they move.
Some federal rules are still proposals. HHS published a proposed rule to strengthen the HIPAA Security Rule for electronic protected health information on January 6, 2025, with comments closing March 7, 2025. It remains a proposal. Building to a proposed rule is a defensible choice, but describing it as a requirement is not.
Federal cloud authorization is being rebuilt. FedRAMP 20x replaces the static annual assessment with automation-based validation, machine-readable packages, and continuously reported Key Security Indicators. Phase One piloted Low impact with 26 submissions and Phase Two piloted Moderate with 14; Phase Three is the wide-adoption phase, with the submission pipeline opening in July 2026. If a federal customer is anywhere in your two-year plan, the artifacts you build now should be machine-readable from the start.
The lesson across all four is the same. Dates move; capabilities do not. A system that can produce a versioned, per-slice, reproducible record of its own behavior satisfies whichever version of the rule lands. A system that cannot will fail all of them.
From working pilot to a system that survives a review
The order matters more than the speed. Classification before architecture, architecture before paperwork, paperwork before instrumentation. Teams that invert this rebuild twice.
The sequence that works, and roughly what each step costs
Two notes on the sequence. Step three runs alongside the technical work rather than after it, because contract cycles are the longest pole and they are not yours to compress. And step one is worth doing carefully even when it feels obvious, because the classification decision determines everything downstream, and re-drawing an enclave after the data has moved is the single most expensive rework in this category.
A common framework is cheaper than a stack of them
Companies operating across health, finance, education, and government tend to accumulate one compliance program per regime, each with its own inventory, its own risk register, and its own owner. The overlap is enormous. Access control, encryption, logging, vendor management, incident response, and change control appear in nearly all of them, in nearly the same form.
The NIST AI Risk Management Framework, released as version 1.0 on January 26, 2023, organizes this into four functions: Govern, Map, Measure, and Manage. NIST added a Generative AI Profile, NIST AI 600-1, on July 26, 2024, addressing risks specific to generative systems. Neither is binding on a commercial company, and that is precisely why they are useful as the spine: you can map a common control set to whichever regimes apply and produce regime-specific evidence from one underlying inventory rather than maintaining several.
This also happens to be the cheapest bridge into federal work. The control families in NIST SP 800-171, the audit and monitoring expectations in the Safeguards Rule, the assurance chain in HIPAA, and the evidence a FedRAMP package wants are overwhelmingly the same controls described in different dialects. A team that builds them once, with machine-readable artifacts, walks into a federal conversation with most of the answer already written.
Bottom line
A model that works is roughly half of a regulated system. The other half is the ability to prove, later, to someone who was not there, what the system did and why. That half is not a document you write at the end. It is a property of the architecture: a bounded enclave that contains every copy, a papered chain of vendors, logs that pin a model version to an output and cannot be quietly edited, measurement by slice with separated error types, oversight that can actually say no, and a reason specific enough to give a person who wants to argue with it. Build those, and the specific regime becomes a mapping exercise. Skip them, and every new regulator is a new project.
Frequently asked questions
If regulated data or anything derived from it reaches the vendor, yes. Under HIPAA, § 164.308(b) requires a covered entity to obtain satisfactory written assurances from a business associate and requires that business associate to obtain the same from its subcontractors, with the written contract terms set by § 164.314(a). The GLBA Safeguards Rule at 16 CFR 314.4 requires financial institutions to contractually require safeguards from service providers and to assess them periodically. Whether the vendor is a household name has no bearing on either requirement.
Encryption is a control, not a scope exit. The business associate obligation in § 164.308(b) turns on the arrangement, and the specification at § 164.308(b)(3) is Required rather than Addressable. Key custody genuinely reduces risk and is worth doing, but the written agreement is still owed. Treat encryption as one of the safeguards you describe in that agreement rather than as a reason not to have one.
You need enough to reconstruct a specific decision on demand, which in practice means input, output, model and index version, prompt template version, reviewer identity, and timestamp, with tamper evidence. What you do not need is those records living in a general observability platform outside your boundary, or retained forever by default. Decide the retention period deliberately, apply it to the derived copies as well as the source, and make sure a deletion request reaches the vector index and the evaluation fixtures, not only the database.
Only if the review is real. The reviewer needs the same inputs the model had, enough time per item to reach an independent judgment, and a genuine ability to override. Track the override rate from day one. A rate close to zero usually means the step is a formality, and because the approval is recorded, a formality that is documented is worse than no step at all.
Later than the original text said. Prohibited practices applied from February 2, 2025 and general-purpose AI and governance obligations from August 2, 2025, but the AI Omnibus adopted November 19, 2025 entered into force July 27, 2026 and moved the high-risk rules. Systems used in high-risk areas including biometrics, critical infrastructure, education, employment, and migration and border control now come under the rules from December 2, 2027, and AI embedded in regulated products from August 2, 2028. Confirm the current position before it drives a schedule; this timeline has already moved once.
Very little of it is. The control families in NIST SP 800-171, whose Revision 3 was published in May 2024, overlap heavily with what the Safeguards Rule and the HIPAA Security Rule already ask of you. FedRAMP 20x is moving federal authorization toward machine-readable packages and continuously reported indicators rather than annual static audits, which rewards teams that instrumented properly for commercial reasons. Building the evidence layer once, in a form a machine can read, serves both markets.