The two ways companies your size get this wrong
A company with 300 to 3,000 people has a specific AI governance problem, and it is not the one the consulting decks describe. The first failure mode is the binder: legal writes a policy modeled on something built for a bank with 40,000 employees, it lands as a PDF nobody opens, and a year later a customer questionnaire asks which of your products use AI and nobody can answer. The second failure mode is nothing at all. The model in the pricing engine was built by an analyst who left, support wired a language model into ticket triage over a long weekend, and the first time anyone assembles the list is after something breaks.
Both failures come from treating governance as a document instead of a set of running processes. The document is the smallest part. A program has to produce a list you trust, a way to sort it by consequence, a gate priced to the risk, records that update when the system changes, a human whose review means something, numbers that show when behavior has moved, and a phone tree for when it goes wrong.
Where AI exposure concentrates · mid-size company, by entry point
Editorial weighting from public sources and practitioner reading. Illustrative, not a measured statistic.
One: an inventory, because you do not have one
Every program starts here and most stall here. "Where are we using AI" sounds answerable and is not, because the interesting uses never went through procurement. They arrived when a vendor enabled a feature in a quarterly release, or when a manager expensed twenty seats of a writing tool on a corporate card.
Four discovery channels find most of it in two weeks. Read every app added to the single sign-on list in the last twenty-four months. Search card and expense data for the obvious vendor names. Pull egress or proxy logs for traffic to model API endpoints. Then run a short amnesty: anyone who registers a tool in the next fourteen days faces no consequence, and anything found afterward gets switched off. The amnesty finds what the logs miss, and it costs nothing.
Keep the record short enough that people fill it in. Ten fields is the working ceiling.
Owner. A named person, not a department. Somebody answers the phone when the model misbehaves.
Purpose and decision affected. One sentence on what it does, one on what changes because of its output. If the answer to the second is "nothing," you have found a demo.
Data in. Categories, not schemas. Whether it touches personal data, protected health information, or contract-controlled information decides most of the rest.
Model, provider, and human position. Vendor and model family, whether inference runs on your infrastructure or theirs, and whether a person reviews before the action, after it, or never.
Tier, evidence, monitoring, review date. The tier, a link to whatever evaluation exists, what is watched in production, and when this record gets read again.
Store it where keeping records current is already someone's job: the configuration management database, the service catalog, the GRC platform. A standalone spreadsheet decays within two quarters. A record attached to an asset IT already reconciles does not.
Two: three tiers, set by two questions
Five-tier schemes look rigorous and produce arguments. Three tiers produce decisions. The tier is set by two questions asked in order.
What happens to a person if the output is wrong? Denial of a loan, a job, a benefit, a clinical recommendation, a safety action, or a legal consequence puts the system in the top tier regardless of how good the model is. Inconvenience puts it at the bottom. This tracks how regulators already think. The EU AI Act, in force since 1 August 2024 with obligations phasing in through 2027, defines its high-risk category almost entirely by consequence to individuals, and Colorado's SB 24-205 uses the same logic for "consequential decisions" in employment, lending, housing, insurance, health care, education, and legal services. Borrowing those category lists saves an argument and matches what a customer's counsel will ask about.
How much room does the system have to act on its own? A model that drafts something a person then sends is different from one that sends it. Autonomy raises the tier one step; a mandatory, real human review lowers it one step. That single rule handles most edge cases without a committee.
| Tier | What qualifies | Gate | Ongoing |
|---|---|---|---|
| Tier 1 · High | Decides or materially shapes an outcome for a person; or acts without review in a safety, financial, or regulated path | Documented impact assessment, subgroup performance testing, legal and privacy sign-off, named accountable executive. Four to eight weeks | Monthly metric review, quarterly re-test, annual reassessment |
| Tier 2 · Moderate | Informs a decision a person makes; touches confidential or personal data; customer-visible output | Written evaluation against a held-out set, named reviewer, security review of the data path. One to two weeks | Quarterly metric review, drift alerts to the owner |
| Tier 3 · Low | Internal productivity, drafting, summarizing, search over non-sensitive material | Register it. Business owner approves. One to two days | Annual confirmation that it still exists and is still low |
| Prohibited | Uses your company will not permit at any tier | Written, short, specific. Most lists run five or six lines | Reviewed when the law or the business changes |
The prohibited list matters more than its length suggests. Writing down five things you will not do makes the other three tiers read as permissive, which is the difference between a policy people route around and one they use.
Three: gates that cost what the risk is worth
Review processes get bypassed because they charge the same price for a summarization tool and a credit model. Price the gate to the tier and the bypassing mostly stops. A Tier 3 gate is a form with eight fields and a one-business-day service level; if registering a low-risk tool takes longer than installing it, people install it. A Tier 2 gate adds a written evaluation: what was measured, on what held-out data, against what threshold, signed by whom. A Tier 1 gate is a real assessment, and that is where the effort belongs.
Put the gate at a chokepoint that already exists. Vendor onboarding, the security review before any new data connection, the change-approval step before production deployment, and the annual SaaS renewal are four places where an AI question becomes three lines on a form somebody already fills out. A parallel "AI review" queue collects dust. One rule keeps it honest: the gate has an exception path, and exceptions are written down with an expiry date. No exception path generates lies; unlimited undated exceptions generate nothing.
Four: documentation that stays current
Documentation fails because it is produced once, for an approval, and then diverges from the system within a release or two. The fix is to attach it to change events instead of calendar dates, and to keep it short enough that updating it is cheaper than avoiding it.
For a Tier 1 or Tier 2 system the record has five parts and fits on two or three pages: intended use and explicit out-of-scope uses; training or grounding data with provenance and known gaps; measured performance overall and on the subgroups that matter; known failure modes with what was done about each; and the human review design. Model cards and datasheets, the formats introduced in the research literature in 2019 and 2021, are the right shape. Nothing longer gets maintained.
Four events trigger an update: a model version change, a prompt or retrieval configuration change (which teams routinely forget is a model change), a material shift in input data, and any incident. If your deployment pipeline can fail a build when the model version in the manifest does not match the version in the record, do that. Enforcement in the pipeline is worth more than enforcement in a policy.
The federal side has converged on the same shape. OMB's memoranda governing agency AI use, most recently the April 2025 guidance that replaced the 2024 versions, require agencies to inventory AI use cases, designate a Chief AI Officer, and apply minimum practices to high-impact uses before deployment. If you sell to agencies, or subcontract to a prime that does, your documentation gets read against that standard.
Five: human review that means something
"Human in the loop" is the most oversold control in AI governance because it is usually asserted and almost never designed. A reviewer needs three things or the control is decorative.
Authority. The reviewer can reject the output without asking anyone. If overriding requires a manager's approval, the manager is the control and the reviewer is a formality.
Information. The reviewer sees the inputs, the output, a confidence signal, and the two or three factors that drove the result. A reviewer given a score and no context approves the score.
Time. The queue and the staffing math allow real review. A team asked to clear 400 items a shift is not reviewing them.
Then instrument it. Override rate is the most useful number in the program. Under about two percent, the review is a rubber stamp that will not survive an audit or a deposition. Over about forty percent, the model costs more than it saves. Track it monthly per reviewer and per queue. If overridden cases are not routed back into evaluation data, you are paying for judgment and discarding it.
Where a decision is legally consequential, the review design has to satisfy the rule that already applies. Under Regulation B, 12 CFR 1002.9, a creditor must give specific principal reasons for an adverse action, and the CFPB stated in Circular 2022-03 that algorithmic complexity is no defense for failing to do so. For employment selection, the Uniform Guidelines at 29 CFR Part 1607, including the four-fifths convention in 1607.4(D), are the yardstick a plaintiff's expert will use. New York City's Local Law 144 has required annual independent bias audits of automated employment decision tools since enforcement began in July 2023.
Six: drift monitoring, split three ways
"Monitor for drift" hides three different measurements with three different costs.
Input drift is cheapest and comes first. Compare the live feature distribution against the training or reference window. Population stability index is the workhorse: under 0.10 means nothing has moved, 0.10 to 0.25 warrants a look, above 0.25 means the model is operating on a different population than it was built for. Kolmogorov-Smirnov and chi-square do the same job on continuous and categorical fields. This needs no labels, so it runs daily from day one.
Output drift is nearly as cheap. Watch the score distribution, the approval or flag rate, the fraction of low-confidence results, and for language systems the refusal rate and mean response length. A ten-point shift in an approval rate with no product change is a signal whether or not you yet know which direction is wrong.
Outcome drift is the one that matters and the one nearly everyone skips, because ground truth arrives late. Defaults show up in months. A hiring decision resolves in a year. Build the label pipeline anyway, accept the lag, and use proxies meanwhile: downstream correction rates, complaint volume, override rate, rework hours. A system with no outcome measurement is unfalsifiable, and unfalsifiable systems fail quietly for eighteen months.
Route alerts to the named owner, not a shared mailbox, and give every alert a defined response: investigate, retrain, restrict, or roll back. An alert with no defined response trains the team to ignore alerts.
Ninety-day stand-up · the sequence that works
The seventh thing: what happens on a bad Tuesday
Incident response for AI is an annex to the plan you already have, and the work is in three definitions.
Define what counts. An AI incident is a model producing materially wrong output that reached a person or a system of record; a confidentiality event such as sensitive data entering a prompt or leaving through a response; a control failure such as a model deployed without its gate; or a third-party event such as a provider outage or a silent version change. Write those four down, because the first argument in any real incident is whether it is one.
Define severity by blast radius and reversibility: how many people were affected, whether any of them have a legal remedy, and whether it can be undone. Then define who can stop the system. Somebody on call at 2 a.m. needs standing authority to disable a model without a change advisory board meeting, and that authority is written down in advance or it does not exist.
Clocks come from the contract and the sector, not from the AI. Defense contractors report cyber incidents affecting covered defense information to DoD within 72 hours under DFARS 252.204-7012, and critical-infrastructure entities will report substantial incidents within 72 hours once CISA's rule under CIRCIA is in force. HIPAA breach notification, state breach statutes, and customer contracts each carry windows, and the contracts are usually tightest. Map them once, keep the map in the annex, and use the NIST SP 800-61 handling structure you already follow.
Third-party AI is most of your exposure
For a mid-size company most AI risk arrives through a renewal rather than a build. A vendor ships a feature, an administrator enables a checkbox, and your customer data now flows to a model nobody evaluated. Four questions in the vendor questionnaire cover most of it. Where does our data go and is it used for training. What is the retention period for prompts and outputs. What notice do we get before a model version changes. What happens to our data if you are acquired. Then push two clauses into the contract: notification before material model changes, and the right to disable AI features without losing the underlying service. The second is the one vendors resist and the one that gives you an off switch. ISO/IEC 42001:2023, the certifiable AI management system standard, is worth asking about; a vendor holding it has had an auditor look at exactly these processes. That is not proof the model is good, only evidence somebody outside the vendor has looked.
Where the NIST AI RMF fits
The NIST AI Risk Management Framework, published in January 2023, is the right spine for this and the wrong deliverable. Its four functions map onto the six components above: GOVERN is the tiering scheme and the accountable owners, MAP is the inventory and the impact assessment, MEASURE is the evaluation and the monitoring, MANAGE is the gates and the incident annex. The companion Playbook and the Generative AI Profile issued as NIST AI 600-1 in July 2024 supply specific practices when a control needs filling in.
Use the framework as a checklist against your own program and as the vocabulary in your customer answers, because it is the vocabulary they use. It is voluntary and outcome-oriented; it will not tell you what your thresholds should be. Do not produce a document that walks it category by category. Nobody will read it, and the auditor who asks wants the inventory and the evidence, not a restatement of the source.
What it costs and who runs it
At this scale a working program runs on roughly half of one person's time once standing, plus a few hours a month from security, privacy, and legal, plus a review group of four or five meeting monthly. The owner is usually already on staff: a director of engineering, a security lead, a risk manager. The title matters less than three attributes. They can say no, they report high enough that the no sticks, and they are close enough to the work to know when a tool ships. Discovery is the expensive part, and until the list exists every other control is theoretical.
If you are not regulated, build it anyway
Your customers are regulated. Enterprise procurement, insurance renewals, and prime contractor flow-downs now carry AI questions, and the company that answers them in a week wins deals against the company that takes a quarter. General consumer-protection authority also applies without any AI-specific statute: the FTC has brought actions against firms for overstating what their AI does, and the SEC settled charges with investment advisers in March 2024 on the same theory. The claim you make about your AI is regulated even where the AI is not.
Bottom line
The minimum that works is an inventory you trust, three tiers, gates priced to the tier, two-page records wired to the deployment pipeline, human review with real authority and an override rate you watch, three kinds of drift monitoring, and an incident annex with a named person who can pull the plug. That is a ninety-day build and half an FTE afterward. It answers a customer questionnaire, it survives an auditor, and when something goes wrong you hear it from your own dashboard rather than from your largest account.
Frequently asked questions
An inventory of AI uses, three risk tiers, review gates sized to each tier, short documentation for the top two tiers, human review with a tracked override rate, drift monitoring, and an AI annex to the existing incident response plan. Ninety days to stand up, roughly half a full-time person to run.
Four channels find most of it in two weeks: the single sign-on application list, expense and corporate card data, network egress logs showing traffic to model API endpoints, and a short amnesty window where anyone can register a tool without consequence. It is investigative work, and the part most programs underestimate.
Two questions. What happens to a person if the output is wrong, and how much the system can do without review. Consequence sets the base tier, autonomy raises it one step, mandatory human review lowers it one step. Borrow the high-risk category lists from the EU AI Act and Colorado's SB 24-205 rather than inventing your own.
Authority to reject without escalation, enough information to judge the output, and enough time in the queue to do it. Then measure the override rate. Below roughly two percent the review is a rubber stamp; above roughly forty percent the model is not helping.
It is voluntary, and it is the right structure to organize against because GOVERN, MAP, MEASURE, and MANAGE map onto inventory, tiering, evaluation, and response. Use it as the spine and the shared vocabulary, but do not produce a document that restates it category by category. That is not what an auditor or a customer asks to see.