Every transaction monitoring program is drowning in alerts and every vendor pitch leads with the false positive reduction number. It is the right problem. It is also the half of the problem that does not determine whether the system survives an examination.
The question an examination actually asks
The examiner's concern is not how many alerts were closed. It is whether anything that should have been escalated was closed instead — and whether the institution can demonstrate, rather than assert, that it was not.
That reframes the engineering. A triage model optimized purely for precision on historical dispositions inherits every judgment call in that history, including the wrong ones, and has no mechanism for showing what it suppressed or why.
Why historical dispositions are a contaminated label
The training signal in most programs is what analysts did, not what was true. Alerts closed under time pressure look identical in the data to alerts closed after genuine investigation. A model trained on that distribution learns the institution's throughput constraints as if they were risk judgments.
A model trained on closures learns how the team was staffed, not what was suspicious.
This is not a reason to avoid the model. It is a reason to hold back a deliberately re-investigated sample, disposed of without time pressure, and treat that as the evaluation set even though it is small and expensive to produce.
What has to exist for the answer to be constructible
An auditable triage system produces, for every alert it touches, a record that a reviewer can walk without access to the model.
- The features that drove the disposition, with the underlying transaction identifiers
- The typology or scenario the alert was matched against, named explicitly
- The threshold or rule that governed the outcome, with its version
- A confidence band, and a routing rule that escalates anything outside it
- Periodic re-scoring of closed alerts under the current model, so drift surfaces as a delta rather than a surprise
Examination burden by what the system is allowed to do
Relative weighting from delivery practice, not a measured statistic — shown to rank where attention belongs.
The narrative problem
Where language models enter this workflow is the SAR narrative and the investigation summary, and that is exactly where the fabrication risk is highest. A narrative that describes a pattern in slightly stronger terms than the transactions support is a document filed with a regulator that overstates the evidence.
The construction is the same as anywhere else the stakes are high: the drafting model sees a structured evidence object, never raw text, and every figure and date in the output must resolve to a field in that object. Anything that does not resolve is not a stylistic issue. It is removed.
Sequencing that tends to work
Programs that get this deployed usually do not start by closing alerts automatically. They start by ordering them, which requires no suppression argument at all, and by drafting the investigation record, which saves the analyst the majority of the time without removing the judgment.
Auto-disposition comes later, on the narrow bands where the re-investigated sample supports it, with the escalation path always open. That path is slower than the pitch deck and it is the one that reaches production.
Three dispositions, three different evidentiary burdens
Triage is usually discussed as though it were one decision. It is three, and they are not equally hard to defend.
| Disposition | What must be shown | Difficulty |
|---|---|---|
| Escalate | That the alert reached a human investigator | Low — no suppression argument exists |
| Prioritize | That ordering did not delay anything time-sensitive past its window | Moderate — requires queue-latency evidence |
| Close | That nothing requiring escalation was suppressed, demonstrably | High — this is the whole examination question |
Most of the claimed benefit in vendor material comes from the third row, and most of the deployable benefit in the first two. Programs that recognize this early ship something; programs that treat auto-closure as the objective spend a year in model risk review.
Building the holdout that carries the evidence
The re-investigated sample is the analytical backbone of the whole argument, and it has to be built deliberately.
- Sample across the risk spectrum, not just the alerts that were escalated — the informative cases are the closures
- Re-investigate without time pressure, by investigators who cannot see the original disposition
- Record the reasoning, so a disagreement between the original and the re-investigation can be adjudicated rather than assumed
- Refresh it periodically, because typologies move and a three-year-old holdout measures a world that no longer exists
- Keep it out of training entirely — it is worthless the moment the model has seen it
Institutions consistently underestimate how much this sample is worth. It is the only artifact in the program that supports a claim about false negatives, and false negatives are the entire subject of the examination. A few hundred carefully re-investigated alerts do more for the program's defensibility than a much larger volume of ordinary dispositions.
What model risk management will ask for
Institutions operating under model risk expectations will find that a triage model is treated as a model, with the documentation burden that implies. Anticipating it during the build is dramatically cheaper than reconstructing it afterward.
The recurring requests are conceptual soundness — why this approach for this problem, and what alternatives were considered; the development evidence, including the data used, its limitations, and the label contamination discussed above; independent validation by someone outside the development team; ongoing monitoring with defined thresholds and an escalation path; and a documented change control process governing what happens when the model is retrained.
None of this is unreasonable and most of it is work the team should want anyway. The failure mode is producing it retrospectively, at which point the development decisions that would have been easy to justify at the time have to be reconstructed from memory.
Threshold tuning is where programs get in trouble
Every monitoring program tunes thresholds. Done well it is sound risk management; done casually it looks like alert volume management, and the distinction lives entirely in the documentation.
The record has to show the analysis behind each threshold — the below-the-line testing that examined what falls just outside the current setting, the volume impact, and the risk rationale for accepting it. A threshold changed with an entry reading "reduce false positives" is the kind of finding that generalizes into a broader criticism of the program.
When a model is inserted into this process the same standard applies to its decision boundary. A model confidence cutoff is a threshold, it needs the same below-the-line analysis, and it needs it every time it moves.
The SAR narrative deserves its own controls
Narrative drafting is the most attractive automation target in the whole workflow — investigators spend enormous time on it, quality varies, and language models are genuinely good at the form. It is also the point where a fabrication becomes a filed document.
Three controls make it safe enough to deploy. The drafting model receives only a structured evidence object assembled by the investigation, never raw transaction text or free-form notes. Every figure, date, account reference, and counterparty name in the output is checked against a field in that object, and anything unmatched is stripped rather than flagged. And the investigator signs the narrative against the evidence object, with both retained together.
Built that way, the narrative is a formatting transformation of verified facts, which is a defensible thing for a machine to do. Built as summarization over case material, it is a filed regulatory document written by a system with no mechanism preventing it from overstating the evidence.
What good looks like after a year
The programs that succeed here do not report a dramatic reduction in alert volume. They report that investigators spend most of their time investigating rather than assembling, that queue ordering demonstrably reduced time-to-escalation on the cases that mattered, and that the institution can answer a question about any disposition without a research project.
That is a less exciting outcome than the pitch and a considerably more durable one.
Staffing changes before the technology does
Programs frequently justify triage automation as headcount avoidance and then discover the composition of the team has to change regardless of whether headcount does.
When the routine closures are handled or ordered by a system, what reaches investigators is denser. The skill mix shifts toward senior investigation and away from volume processing, and the training path that used to run through high-volume simple alerts disappears. That path was how junior analysts learned typologies, and removing it without replacing it produces a capability gap two years out that nobody connects back to the automation decision.
The programs that handle this well deliberately retain a sample of routine alerts for training purposes even though the system could dispose of them, and treat that as a cost of maintaining the bench rather than an inefficiency to be optimized away.
The adjacent programs this touches
| Program | Shared dependency | Why it matters |
|---|---|---|
| Sanctions screening | Resolved counterparty and ownership graph | An alert on a party with restricted ownership is a different alert |
| Fraud | Transaction and device signals | Overlapping typologies; findings on one side inform the other |
| KYC refresh | Customer risk rating and its inputs | Triage that ignores a stale rating is triaging against the wrong baseline |
| Model risk | Validation, monitoring, change control | Determines whether the model is deployable at all |
The KYC row is the one most often missed. A triage model that treats customer risk rating as a stable input inherits whatever staleness exists in the refresh program, and in many institutions that staleness is substantial. Surfacing the rating's age alongside the alert is a small change that prevents a class of confident-but-wrong dispositions.
What to build first if the program is under pressure
Institutions arriving at this after a finding or a consent order have less room to sequence patiently, and the instinct is to buy the largest thing available. That instinct is usually wrong, because the largest thing takes the longest to validate.
The fastest defensible improvements are the ones that add evidence without changing outcomes: recording the disposition rationale as structured data, surfacing the risk rating and its age, ordering the queue by time-sensitivity, and standing up the re-investigated holdout. None of these require model validation, all of them improve the examination position, and together they are the foundation any later model will need anyway.
Frequently asked questions
Yes, with a clean evaluation set. The contamination limits what the training data can prove, not what the model can do. The re-investigated holdout is what carries the evidentiary weight.
What is required is a reconstructable decision. An inherently interpretable model is one way to get there and often the pragmatic one, but a complex model with a complete evidence record can also satisfy the requirement.