The cost of a model nobody trusts

The failure that passes every test

A model can meet every obligation in the contract and still fail. It hits the accuracy target in acceptance testing. It runs inside the latency budget. It clears the security review and gets its production credentials. Then it goes live, and the people it was built for keep working the way they worked before. Six months later the service is healthy, the monitoring is green, and almost nobody is calling it. The owner is paying for infrastructure, an on-call rotation, and a maintenance agreement on a system that has changed no decisions.

This is a different failure from a model that does not work, and the two have different remedies. A model that does not work gets fixed by improving the model. A working model that is ignored usually does not, which is why the standard responses — another round of training, a redesigned interface, a message from an executive — produce a brief spike in usage followed by a return to the old baseline.

This failure has been studied for thirty years under a precise name, in domains where every instance of a human ignoring a computer is written to a log. The evidence is clean, and it points at causes an engineering team can act on.

Disuse is the technical term, and it has a cause

Raja Parasuraman and Victor Riley set the vocabulary in Human Factors in 1997 (volume 39, issue 2, pages 230 to 253). Their abstract defines four states, and the definitions are worth reading exactly as written: "Use refers to the voluntary activation or disengagement of automation by human operators. Misuse refers to over reliance on automation, which can result in failures of monitoring or decision biases. Disuse, or the neglect or underutilization of automation, is commonly caused by alarms that activate falsely."

That last clause is the diagnosis. Disuse is commonly caused by alarms that activate falsely. And the paper's explanation for why systems produce false alarms is arithmetic rather than psychology: designers set the detection threshold without accounting for the base rate of the event in the population the system will actually run on. A detector tuned against a balanced evaluation set, then deployed into a population where the event is rare, generates false positives that swamp its true positives. The operator learns the real hit rate, and adjusts.

Two things follow, and both are uncomfortable for the party that built the system. First, disuse is a design defect and it lives in the calibration, not in the user. Second, the user's behavior is rational. Once someone has learned what an alert is actually worth, ignoring it is the efficient response, and no amount of training will unlearn a correct inference.

The clearest measurements come from medicine

Clinical decision support has the longest instrumented record of humans ignoring a deployed model, because every override is captured with a timestamp and a reason code. The numbers have been stable for two decades.

Van der Sijs and colleagues reviewed 17 studies for the Journal of the American Medical Informatics Association in 2006 (volume 13, issue 2, pages 138 to 147) and reported that drug safety alerts are overridden by clinicians in 49 percent to 96 percent of cases. They also listed what they called the error-producing conditions in the alerting system: low specificity, low sensitivity, unclear information content, unnecessary workflow disruptions, and unsafe and inefficient handling. Every one of those is a property of the software, not of the clinician.

Fourteen years later, Tahmina Nasrin Poly and colleagues ran the same review over 23 studies published through March 2019 in JMIR Medical Informatics (2020, volume 8, issue 7, article e15653). The range of average override rates was 46.2 percent to 96.2 percent. Nothing had moved.

The finding an owner should sit with is the next one in that same paper. Poly's team adjudicated the overrides and classified between 29.4 percent and 100 percent of them as appropriate, with the rate varying sharply by alert type: drug-allergy 63.4 to 100 percent, dose 43.9 to 88.8 percent, renal 27 to 87.5 percent, geriatric 14.3 to 57 percent, drug-drug interaction 0 to 95 percent. The two most common reasons clinicians gave for overriding were "will monitor" and "patients have tolerated before."

Read those two reasons again. Neither is a rejection of the model's logic. Both are statements about context the model did not have. That is what the log of a distrusted system usually contains: not defiance, but local knowledge the system was never given access to.

Precision in the deployment population is the number that predicts use

Delphine Carli and colleagues went after the mechanism directly in JMIR Medical Informatics in 2018 (volume 6, issue 1, article e3), using positive predictive value as a proxy for clinical relevance. Across the studies that reported enough detail to compute it, positive predictive value ranged from 8 percent to 83 percent, with most results falling between 20 and 40 percent. Sensitivity ranged from 38 percent to 91 percent. The best-performing systems were the ones that incorporated patient characteristics such as comorbidity and laboratory results, rather than firing on a rule in isolation.

Put plainly: in the median system studied, three to four of every five alerts were wrong for that patient. A user facing those odds is not being difficult. If each flag costs thirty seconds to check and most are wrong, the system is a tax with an option attached, and the option is rarely worth the tax. The number that predicts use is not the accuracy in the test harness. It is the precision in the population the user actually sees, at the threshold the system actually ships with.

Four reasons a working model gets ignored

The precision was measured on the wrong population. A model validated on a curated set and deployed against the full stream will see a different base rate, and its precision will fall without any change to its weights. This is the single most common cause and the one most often missed, because the model's reported metrics never changed.

The output arrives at the wrong moment. A recommendation that appears in a separate dashboard, in an email digest, or after the user has already committed to an action competes with the cost of context-switching. Users who would accept the same output inline will ignore it in a second window. This is a placement defect that reads, in the logs, exactly like a trust defect.

The downside is asymmetric and it lands on the user. When following the model and being wrong is career-visible and ignoring the model and being wrong is invisible, the user is being paid to ignore it. No interface change fixes an incentive. Naming who owns the outcome when the model is wrong — in writing, before rollout — changes behavior more than any explanation feature.

There is no control surface. The user can take the answer or leave it, and nothing in between. This is the failure that has the best-evidenced and cheapest fix, which is the subject of the next section.

The modifiability finding, and why it is the cheapest fix on the list

Berkeley Dietvorst, Joseph Simmons and Cade Massey published the result that should shape how these systems are designed, in Management Science in 2018 (volume 64, issue 3, pages 1155 to 1170). Their abstract states it directly: "Participants were considerably more likely to choose to use an imperfect algorithm when they could modify its forecasts, and they performed better as a result."

Three details make this more useful than it first appears. The preference for a modifiable algorithm held even when participants were severely restricted in the modifications they could make. It was relatively insensitive to the magnitude of the modification allowed. And the authors read this as evidence of a desire for some control over the outcome, rather than a desire for greater control.

For an owner, that is the most actionable result in this literature, because a narrow adjustment control is a small piece of engineering. This does not mean handing users a text box and letting them rewrite the answer. It means a bounded, logged lever: a confidence floor the user can raise for their own queue, a category they can suppress with a reason code, a numeric estimate they can shift within a fixed band. The bound keeps the system's behavior defensible. The log turns every adjustment into a labeled example of where the model and the local expert disagree.

Aversion, appreciation, and telling which one you have

The literature here is genuinely unsettled, and pretending otherwise would be a disservice. The same three authors established the aversion result in the Journal of Experimental Psychology: General in 2015 (volume 144, issue 1, pages 114 to 126), under the title "Algorithm aversion: People erroneously avoid algorithms after seeing them err." The finding is that watching an algorithm make a mistake causes people to abandon it even when it still outperforms them.

Jennifer Logg, Julia Minson and Don Moore reported close to the opposite in Organizational Behavior and Human Decision Processes in 2019 (volume 151, pages 90 to 103). Across six experiments, lay people adhered more to advice they believed came from an algorithm than to the same advice attributed to a person — a pattern they named algorithm appreciation. Their own paper notes that researchers asked to predict the outcome predicted the opposite.

The reconciliation that matters operationally sits inside Logg's results. Algorithm appreciation waned in two conditions: when people were choosing between the algorithm's estimate and their own rather than an external advisor's, and when they had expertise in the forecasting domain. Both of those conditions describe an internal deployment to skilled staff. The practical read is that the default posture of a novice user is acceptance and the default posture of an expert user is audit, and a system fielded to experts has to be designed for audit — fast verification, visible evidence, a control surface — rather than for compliance.

What each intervention actually changes

The table below is the decision most owners are actually facing once the usage numbers come in. The failure mode is picking an intervention because it is cheap to schedule rather than because it addresses the mechanism in the logs.

Intervention	What it changes	When it works	When it fails
More training and rollout messaging	The user's knowledge of how to operate the tool	The tool is genuinely misunderstood and its outputs are right	Users already understand it and are declining on the merits
Better explanation of each output	The cost of verifying one output	Verification was the bottleneck and the evidence is checkable in one step	The explanation restates the model's confidence instead of showing evidence
Raising the firing threshold	Volume and precision, at the cost of recall	False positives are the complaint and the missed cases are cheap	The missed cases are the ones that justified the build
Narrowing the population it fires on	The base rate in the slice the user actually sees	Precision is low because the model runs everywhere by default	The defect is calibration rather than scope
A bounded, logged user adjustment	Whether the user has any control, and what you learn	Users are competent and hold context the model lacks	Adjustment becomes a way to route around a real defect unnoticed
A mandate to use it	Recorded usage, for as long as attention holds	Rarely, and never durably	As soon as the reporting stops being watched

Measuring the ignoring

Most owners discover the problem anecdotally, months late, because nobody instrumented the thing that was about to go wrong. A 2026 systematic review of systematic reviews in JAMIA by Cara Ray and colleagues (volume 33, issue 8, pages 1523 to 1531) found how thin this practice still is: across 22 included studies, only one article reported an operational definition of alert fatigue at all. The most common metrics reported were quantity, override rate, and acceptance rate. Their recommendation is worth adopting verbatim as a target definition — a significant, sustained decrease in appropriate response rates from an established baseline.

The phrase carrying the weight is from an established baseline. Without a baseline captured at launch, later measurement tells you the level and not the trend, and a level with no trend is very hard to act on. Four instruments, all cheap to build before go-live:

• Call rate against eligible events. Not raw invocations. The denominator is the number of cases where the model was available and relevant, which requires defining eligibility explicitly.
• Acceptance rate. The share of outputs that changed what the user did. This requires capturing the user's decision, not just their click.
• Appropriate-acceptance rate. Acceptance adjudicated against ground truth on a sampled subset. This is the only one of the four that distinguishes a user who is right to ignore the model from a user who is wrong to.
• Override reasons from a controlled vocabulary. A short closed list plus a free-text field. The value of this is settled: the two dominant reasons in the clinical literature were both statements about missing context, and they only became visible because someone recorded them in a structured field.

Segment all four by user and by case type from the first day. Disuse is rarely uniform. The usual pattern is that the model is trusted for one segment and ignored for another, and the aggregate number hides both facts.

A recovery sequence that starts with the logs

The sequence below is ordered so that every decision is made after the evidence that should drive it, which is the opposite of how most remediation projects run. The durations are typical rather than fixed; the ordering is what matters.

Step 6 does more work than it looks. Users who told you the system was wrong, then watched it get narrowed on the strength of their own override reasons, treat the next release differently. That loop is how a distrusted system recovers, and skipping it is why remediation projects that fixed the model still fail to recover usage.

The federal rulebook is the best free specification available

Two public documents describe this problem more precisely than most commercial governance frameworks do, and neither costs anything to read.

The NIST AI Risk Management Framework 1.0 (NIST AI 100-1, released January 26, 2023, and explicitly voluntary) puts end-user reality inside the measurement function rather than beside it. MEASURE 4.1 requires that "measurement approaches for identifying AI risks are connected to deployment context(s) and informed through consultation with domain experts and other end users." MEASURE 4.2 asks that measurement results "in deployment context(s) and across the AI lifecycle are informed by input from domain experts and relevant AI actors to validate whether the system is performing consistently as intended." MEASURE 3.3 calls for feedback processes for end users to report problems and appeal outcomes, "integrated into AI system evaluation metrics." MANAGE 4.1 puts appeal and override into the post-deployment monitoring plan alongside incident response and decommissioning.

OMB Memorandum M-25-21, Accelerating Federal Use of AI through Innovation, Governance, and Public Trust, issued April 3, 2025 by OMB Director Russell T. Vought, rescinds and replaces M-24-10 and implements Executive Order 14179, signed January 23, 2025. For what it calls high-impact AI, it sets seven minimum risk management practices: pre-deployment testing; a documented AI impact assessment; ongoing monitoring for performance and adverse impacts; adequate human training and assessment; human oversight, intervention and accountability; consistent remedies or appeals; and consulting and incorporating feedback from end users and the public.

Two provisions inside the impact assessment are worth copying into a commercial acceptance checklist word for word. The assessment must document "results of independent review," supported by "an independent reviewer within the agency who has not been involved in the development." And it must document "risk acceptance, supported by a signature from the individual accepting the risk." A named person signing for the residual risk of a system is the cheapest way ever devised to surface the disagreements that otherwise show up six months later as an override rate.

Scope, stated plainly: M-25-21 governs executive branch agencies' own use of AI, does not cover AI used as a component of a National Security System, and creates no obligations for a commercial buyer. Its practices are still the most specific public statement of what a responsible deployment looks like. For scale, GAO reported on July 29, 2025 (GAO-25-107653) that across the agencies it examined, reported AI use cases nearly doubled from 571 to 1,110 between 2023 and 2024, with generative AI use cases growing from 32 to 282.

When switching it off is the correct answer

M-25-21 says the quiet part out loud: "If a particular high-impact use case is not compliant with the minimum practices then the agency must safely discontinue use of the AI functionality." Its overview section is blunter still, requiring a plan to discontinue when high-impact AI is not performing at an appropriate level, and cessation where risk mitigation is not possible.

The commercial version runs like this. If adjudicated precision on the live population is low, and cannot be raised without cutting the cases that justified building the thing, then running it costs more than the hosting bill. It teaches skilled people that model output is noise, and that lesson does not stay inside one system — it is applied to the next one before it ships. Retiring a system honestly, with the measurement that justified the decision written down, protects the credibility the next deployment will need.

What this changes in a partner conversation

If you are a consultancy, an integrator or a prime whose delivered model is sitting unused in a customer's environment, the most important reframing is this: the remediation is an engineering scope, not a change-management scope. A remediation that begins with a training curriculum is treating a calibration problem as an education problem, and it will bill out cleanly and change nothing.

A remediation proposal worth signing contains six things: a log extract with an explicit eligible-event denominator; a re-measurement of precision on the live population at the shipped threshold; an adjudicated sample of the outputs users ignored; a named change to scope, threshold or placement that follows from the adjudication rather than from the complaints; a bounded control surface with an audit trail; and a re-measurement plan against the pre-change baseline. Refuse a proposal that promises to raise adoption without proposing to re-measure precision. That promise can only be kept by pressure, and pressure decays.

The same list works as a diligence instrument in the other direction. When we are asked to assess a delivered system that is not being used, these are the artifacts we ask for first, and the answer is usually visible in the override reasons before any modeling work starts.

Bottom line

A model nobody trusts is not a soft problem. It is disuse, it is caused most often by a threshold set against the wrong base rate, and it is measured by call rate, acceptance rate and adjudicated appropriateness against a baseline captured at launch. The strongest evidence-backed lever is not explanation and not training: it is giving a competent user a bounded, logged way to adjust the output, which raises usage, improves outcomes, and hands you a labeled record of every place the model and the expert disagree. Read the logs before choosing the fix. The users are usually telling you something true.

Frequently asked questions