What you are actually deciding
A health program office considering an AI tool is rarely deciding whether the technology is impressive. It is deciding whether this tool, trained on someone else's data, will perform on its own population, fail in ways its staff can catch, protect the people whose records pass through it, and still be affordable and replaceable in three years. A demonstration answers none of those questions. A short list of pointed questions, asked before money moves, answers most of them.
The questions below apply whether the buyer is a federal program, a state health agency, a county health department or a nonprofit running a screening program. The legal details differ, and we note where. The logic does not.
Ask them in writing, and ask for evidence rather than assurances. A vendor with a sound product will have most of the answers on file. The ones who do not are telling you something useful.
How we weight the questions
Not every question carries the same weight. Privacy and legal compliance are a gate: a tool that cannot meet them is out, however well it performs. Among tools that clear the gate, these are the weights we apply when helping a program office compare options.
How we weight the questions, once the legal gate is cleared
Relative weight, in our judgment from evaluation work. Not measured data. Adjust for your program's risk.
Performance on your population sits at the top because every other answer depends on it. A tool that does not work for the people you serve has no acceptable failure mode, no useful monitoring plan and no reasonable price.
What was it built and validated on?
Start with the data, because the model is a summary of it. Ask where the training data came from, what years it covers, what population it represents and how the outcome labels were assigned. A model trained on patients from a few academic medical centers learned the case mix, documentation habits, equipment and coding of those centers. A model whose labels came from billing codes learned to predict billing codes, which may or may not be the clinical outcome you care about.
Then ask separately about validation. Internal validation tests the model on held-out data from the same source it was trained on, which mostly confirms it did not memorize its training set. External validation tests it on data from different sites, populations or time periods, and is the only kind that tells you how it travels. Temporal validation, testing on data from a later period, tells you whether it survives changes in practice, coding and population over time. Ask which of the three the vendor has, and for the external validation, ask whose data it was.
Reading a vendor's validation claim
Validation claims arrive as a single impressive figure. Each common form hides a specific question, and the table pairs them.
| The claim says | What it leaves out | What to ask |
|---|---|---|
| High accuracy | Accuracy is dominated by the common case. A tool that never flags anyone scores 98% accuracy when 2% have the condition | Sensitivity, specificity and positive predictive value at the threshold you will use |
| High AUC | AUC ranks people across all thresholds; it says nothing about calibration or about performance at your operating point | A calibration plot, and results at your chosen threshold |
| Validated in a peer-reviewed study | The study may be internal validation, a different version of the model, or a population unlike yours | Which model version, which population, internal or external |
| Tested on a large dataset | Size is not representativeness. A large sample from one system is still one system | How many sites, which regions, and the demographic mix |
| Performs equally across groups | Often averaged over broad categories, or reported only for groups large enough to look stable | Subgroup results with confidence intervals, including the groups your program prioritizes |
The positive predictive value deserves its own example, because it depends on how common the condition is in the population where you deploy. Take a tool with 90 percent sensitivity and 90 percent specificity, figures chosen for illustration. Where 2 percent of people screened have the condition, only about 15 percent of the people it flags actually do. Where 20 percent have it, about 69 percent of flags are correct. Same tool, same published performance, very different workload for the staff who follow up on every flag. A validation number is a fact about the vendor's test set. Your job is to find out whether it is also a fact about your population.
How does it perform on your population and its subgroups?
The only reliable answer comes from testing the tool on your own data before committing to it. A local validation does not need to be large. A few hundred records with outcomes your clinicians agree on, run through the tool in a controlled environment, will tell you whether its performance holds. Write the right to do this into the evaluation phase of the procurement.
Report results for the subgroups that matter to the program: age bands, sex, race and ethnicity where recorded, language, rural and urban residence, insurance type, and any population the program is specifically funded to serve. Small subgroups will produce wide confidence intervals. That is a finding, not a reason to skip them, and it tells you where to watch closely after launch.
Look for the specific pattern where a tool performs well overall and poorly for the people with the least access to care. Records for those people are often thinner, which gives a model less to work with, and they are often the people a public program exists to reach.
How does it fail?
Every model is wrong some of the time. What matters is the shape of the errors and whether anyone will notice them. Ask the vendor to describe, with examples, the cases the tool gets wrong most often. A vendor who answers with a percentage has not studied the errors. One who answers with case types, such as atypical presentations, incomplete records or a particular device, has.
Ask what happens when the input is out of range: a missing field, an unfamiliar format, a value outside the training distribution. The answer you want is that the tool declines to score and says why. The answer to worry about is that it always produces a score.
For tools built on language models, ask how they prevent output that is not supported by the input, how outputs are traced back to source text, and what the reviewer sees when the model is uncertain.
Where does a human stay in the loop?
Define, before purchase, which decisions the tool informs and which decisions a person makes. A tool that ranks a caseload for outreach is a different risk from a tool that determines eligibility or recommends treatment. The design of the human role should match.
Human review only works if the reviewer has the time, information and authority to disagree. Ask what the reviewer sees alongside the score, how often reviewers override it in existing deployments, and whether overrides are recorded. A review step where staff approve nearly every recommendation within seconds is not oversight. It is a signature on the model's decision.
Some clinical software is regulated by the FDA as a medical device, depending on what it does and how it presents its output. Ask whether the vendor has assessed the tool's status under that framework and what they concluded.
Privacy posture: HIPAA, the Privacy Act and the rest
Which rules apply depends on who you are. HIPAA governs covered entities, meaning health plans, clearinghouses and providers who conduct standard transactions electronically, and the business associates who handle health information on their behalf. If your organization is covered, the vendor needs a business associate agreement before any protected health information reaches it. Many public health programs and nonprofits are not covered entities, which does not mean anything goes; state law, grant terms and your own commitments to participants still apply.
Federal agencies have the Privacy Act. When an agency keeps records about individuals that are retrieved by a personal identifier, those records form a system of records that requires a published notice, and a contractor operating that system on the agency's behalf is bound by the Act as well. Substance use disorder treatment records carry additional protection under 42 CFR Part 2. Several states have their own health data and consumer privacy laws.
Across all of these, ask the same practical questions. Where is the data stored and processed? Does any of it leave your environment? Is any of it used to train or improve the vendor's models, for you or for other customers? Who at the vendor can see it? How is it deleted at the end of the contract, and how is deletion confirmed? Federal buyers using a cloud-hosted tool should also confirm its FedRAMP authorization status.
Is our data used to train your models?
This question deserves its own line in the contract. Many AI vendors improve their products using customer data by default, sometimes in de-identified or aggregated form. That may be acceptable to your program or it may not, but it should be a decision you made, written into the agreement, rather than a default you discovered later.
Monitoring after launch
A model that worked at launch will drift. Populations change, coding changes, clinical practice changes and the data feeding the tool changes with them. Ask how the vendor will detect that, how often, and what you will see.
The minimum is a monthly or quarterly report of the tool's input data distribution against its training baseline, the share of cases flagged, override rates, and outcome-based performance where outcomes become available, broken down by the subgroups you validated. Agree in advance on the thresholds that trigger a review and who pays for retraining or recalibration. A monitoring plan without triggers is a dashboard nobody acts on.
Who owns the data and the model?
Your data is yours. Make sure the contract says so, including derived data such as scores, flags and annotations your staff create. For a tool built or fine-tuned for your program, decide who owns the resulting model weights and code, and what rights each party has to use them afterward. For an off-the-shelf tool, ownership of the model stays with the vendor, but you still need rights to your outputs and a clear statement of what the vendor may do with your inputs.
In federal contracts, data rights and software rights clauses determine much of this, and they should be negotiated deliberately rather than accepted by default.
Total cost and the exit plan
The license is rarely the largest cost. Add integration with your systems, the staff time for review and follow-up created by the tool's flags, training, local validation, ongoing monitoring, security review and renewal increases. Model the follow-up workload explicitly, using the positive predictive value at your prevalence, because a tool that flags many false positives shifts cost onto the staff who chase them.
Then plan the exit before you sign. Ask how your data and outputs will be returned, in what format, and on what timeline. Ask what transition support the vendor provides and at what price. Ask what happens to your data if the vendor is acquired or closes. A program that cannot leave a tool cannot negotiate its renewal price.
The question list
Bring these to the vendor in writing, ask for evidence with each answer, and score the responses before anyone sees a demonstration.
- Training data: source, years, population, and how outcome labels were assigned.
- Validation: internal, external and temporal, with whose data each used.
- Performance at our threshold: sensitivity, specificity, positive predictive value and calibration.
- Subgroup results with confidence intervals for the populations we serve.
- Local validation rights on our own data before commitment.
- Failure modes with examples, and behavior on out-of-range input.
- Human role: what reviewers see, override rates, and how overrides are recorded.
- Privacy: applicable law, storage location, training use of our data, deletion.
- Monitoring: metrics, frequency, triggers and who pays for retraining.
- Ownership: our data, derived outputs and any model built for us.
- Total cost including integration, staff follow-up and renewal terms.
- Exit: data return format, timeline and transition support.
Frequently asked questions
How the tool performs on your population, at the threshold you will use, broken down by the subgroups you serve. Every other question depends on that answer, and the most reliable way to get it is a local validation on your own data before you commit.
Accuracy is dominated by the most common outcome. When a condition is rare, a tool can be highly accurate while missing most cases. Ask for sensitivity, specificity, positive predictive value and calibration at your operating threshold instead.
No. HIPAA applies to covered entities and their business associates. Many public health programs and nonprofits fall outside it, but state law, the Privacy Act for federal agencies, grant terms and protections such as 42 CFR Part 2 may still apply. Determine which rules govern your data before choosing a tool.
Write the exit into the contract: return of your data and outputs in a usable format, a timeline, transition support and its price, and what happens to your data if the vendor is sold or closes.