Anomaly detection in practice: what separates a useful alert from an ignored one

The alert is the product

An anomaly detector is judged on one thing: whether a person acts on what it emits. Everything upstream of that (the algorithm family, the feature set, the training window, the hyperparameter search) is instrumentation in service of a queue that some human being has to work through on a Tuesday morning. Teams that start at the model and reason forward tend to ship detectors that are statistically defensible and operationally ignored. Teams that start at the alert and reason backward ship fewer alerts, of higher quality, into a queue that has a named owner, a response time, and a field where the outcome gets written down. The second kind gets renewed. This is a working guide to building the second kind.

The first question we ask on any anomaly engagement is not what the data looks like. It is: when the system fires, who receives it, what authority do they have to act, and how many can they handle in a day? If nobody can name the receiver, there is no detection problem yet. There is a governance problem wearing a modeling costume, and building a model first makes it worse by producing evidence that nobody has authority to act on.

The second question is what a miss costs relative to a false alarm. In a payment integrity program, a missed improper payment costs real money and a false alarm costs an examiner ten minutes. In a flight-critical telemetry monitor, a missed fault costs a platform and a false alarm costs a maintenance action. Those two ratios point at completely different operating points on the same curve, and no amount of model quality substitutes for stating the ratio out loud before training starts.

Three problems wearing one name

"Anomaly detection" covers at least three distinct problems, and confusing them is the most common source of wasted quarters. A point anomaly is a single observation far from the rest: a $40,000 charge on a card that never exceeds $400. A contextual anomaly is an observation that is unremarkable globally but wrong for its context: 200 kilowatts of draw at 3 a.m. in a building that is empty overnight. A collective anomaly is a subsequence in which no single point is unusual but the pattern is: a pump whose vibration signature slowly rotates through a harmonic it has never occupied before.

Method families map onto those categories unevenly. Distance and density methods (Local Outlier Factor, from Breunig and colleagues at SIGMOD 2000) and partition methods (Isolation Forest, from Liu, Ting and Zhou at ICDM 2008) are strong on point anomalies in tabular feature spaces and weak on context unless you engineer the context into features. Decomposition approaches (STL, seasonal-hybrid ESD) and forecast-residual approaches handle context naturally because the expectation is conditional on time. Subsequence methods (matrix profile, from Yeh and Keogh at ICDM 2016; autoencoder reconstruction over sliding windows) are the ones that find collective anomalies. Picking a family before naming the category is how a team spends three months proving that Isolation Forest cannot find a slow drift in a bearing.

The decisions that are expensive to reverse

Most of the choices in an anomaly project are cheap to change later. Six are not, and each one propagates into the data model, the storage layout, the evaluation code, and the operator's mental model of the system. We settle these in writing before any modeling work begins.

• The unit of alert. A transaction, an account-day, a device-hour, a flight. Everything downstream (labels, metrics, dedup, cost) is denominated in this unit.
• Time granularity and lag. Whether scoring happens per event, per minute, or nightly. Changing this later invalidates every threshold you tuned.
• Label policy. What counts as a confirmed anomaly, who confirms it, and how a "not enough information" disposition is stored. Unlabeled is not the same as normal.
• Training-window contamination. Whether known incidents stay in or are removed from the baseline. Both are defensible. Silence about it is not.
• Thresholding regime. A fixed score cut, a per-entity quantile, or an adaptive tail estimate. This determines how the system behaves the day traffic doubles.
• Feedback capture. Where the reviewer's verdict is written and how it flows back. A system with no return path cannot improve and cannot be audited.

Label policy deserves the most argument. In nearly every real program, the labels you have are the alerts someone already investigated, which means they are a biased sample drawn from whatever the incumbent rule set happened to catch. Training on those labels teaches a model to imitate the incumbent rules, including their blind spots, and then the evaluation looks excellent because the test set inherits the same bias. The honest move is to hold out a small randomly sampled review set, accept that it costs analyst hours, and use it to estimate what the incumbent misses.

Choosing a method

Method selection is mostly a function of data shape, label availability, and how much explanation the reviewer needs. Sophistication is not the axis. In our experience the cheapest method that meets the explanation requirement is usually the one that survives.

Method	Use it when	What it costs you
Control chart / robust z-score	One metric, stable process, reviewer needs a number they can recompute by hand	Blind to multivariate and contextual anomalies; assumes a usable center and spread
Seasonal decomposition + residual test	Strong daily or weekly cycles, business-calendar effects, telemetry with known rhythm	Needs enough history to estimate seasonality; holidays and outages must be handled explicitly
Isolation Forest	Tabular, tens of features, no labels, fast scoring on commodity CPU	Score has no calibrated meaning; a contamination parameter you must justify
Local Outlier Factor	Density varies across the space; clusters of different tightness	Neighbor search cost grows quickly; streaming variants add real complexity
Forecast residual (ARIMA, gradient boosting, sequence models)	Contextual anomalies where the expectation depends on covariates	Two systems to maintain: the forecaster and the residual test on top of it
Autoencoder reconstruction error	High-dimensional signals (spectra, imagery, dense telemetry), plenty of clean history	Training and serving cost; reconstruction error is hard to explain to an auditor

What "good" means, numerically

The arithmetic of rare events defeats intuition, so state it early and in the customer's units. Take two million transactions a month with a true anomaly rate of 0.05%, which is a thousand real events. A detector at 90% recall and a 1% false positive rate produces 900 true alerts and about 19,990 false ones. Precision is 4.3%. The reviewer opens twenty-three cases to find one. Getting precision to 50% at the same recall requires the false positive rate to fall from 1% to roughly 0.045%, a twenty-two-fold reduction. That is the actual engineering target, and it is worth putting on a slide in week one so nobody promises 99% accuracy and means it as a compliment.

Because of that arithmetic, ROC-AUC is close to useless as a program metric here. Davis and Goadrich showed at ICML 2006 that precision-recall curves separate models that ROC curves make look identical under heavy class imbalance, and rare-event detection is nothing but heavy class imbalance. Report precision-recall AUC, and report the two numbers the operator actually experiences: precision at the alert budget and recall at the alert budget. If two examiners can each work forty alerts a day, the budget is about 1,700 a month, so the only question is how many real events the top 1,700 scores contain. That is precision@k and recall@k with k set by staffing rather than by a round number.

For time-series work, add range-based scoring. Tatbul and colleagues (NeurIPS 2018) generalized precision and recall to intervals, which lets you credit a detector for catching an event late-but-within-window rather than scoring every timestamp independently. And report detection latency as a first-class number: median and 95th-percentile minutes from event onset to alert. A detector that finds every outage forty minutes after the pager already went off has a perfect F1 and no value.

How evaluation goes wrong

Three specific traps account for most inflated results. The first is point adjustment, a convention in which flagging any single timestamp inside a true anomaly window marks the whole window detected. Kim and colleagues showed at AAAI 2022 that a randomly generated score achieves strong F1 under point adjustment, beating published deep models on standard benchmarks. If a vendor's numbers rely on it, they have measured almost nothing.

The second is the benchmark itself. Wu and Keogh documented in IEEE TKDE (2021) that several widely used time-series anomaly benchmarks contain trivial anomalies solvable by a one-line rule, mislabeled ground truth, and unrealistic anomaly density. Results on those corpora do not transfer. The UCR Anomaly Archive was built specifically to correct those defects and is a better starting point than the older sets.

The third is temporal leakage. Random k-fold splits on time-series data let the model see the future, and normalization statistics computed over the full history quietly leak the anomaly into the training scale. Split forward in time, fit every transform on the training window only, and hold out a final period that no one touches until the operating point is frozen. Tuning a threshold on the same window you report is the most common way a promising pilot becomes a disappointing deployment.

Failure modes in the field

The detector learns the incident. Retraining on a rolling window that includes last month's outage teaches the model that the outage is normal, so the next one scores lower. Either exclude confirmed incidents from the baseline or use a robust estimator (median and median absolute deviation rather than mean and standard deviation) that a small contaminated fraction cannot move.

Multiple comparisons. A three-sigma Shewhart rule produces a false alarm roughly once every 370 points under normality. On a per-minute series that is about four a day. Across 500 monitored series it is about 1,950 a day, and the queue collapses on day one. Fixed thresholds do not scale to fleets; control the false discovery rate across series (Benjamini-Hochberg is the standard approach), or set per-entity budgets and enforce them.

Drift that is not an anomaly. Traffic doubles after a new region launches. Every static threshold fires. Population stability index and a Kolmogorov-Smirnov test on the input distributions, run on the same cadence as scoring, tell you whether the world moved or the system broke. Those are two different pages and they route to two different people.

Silent upstream change. A field changes units, a nullable column starts arriving null, an ETL job lands late and the window is half full. Data-quality assertions on the input contract catch these in seconds; a model catches them as a mysterious wave of alerts nobody can explain. Validate the input before you score it, every run.

Alert fatigue. This is the terminal failure and it is a volume problem long before it is a quality problem. Deduplicate on the entity, suppress repeat alerts inside a cooldown window, group related signals into one case, and cap the queue. A capped queue that returns the top 1,700 scores is honest about the constraint. An uncapped queue that returns 40,000 alerts is a way of transferring the modeling problem to the person least equipped to solve it.

Production concerns: latency, cost, and monitoring the monitor

Scoring architecture follows from the response time the mission actually needs. Fraud interdiction at authorization time needs synchronous scoring inside a tens-of-milliseconds budget, which rules out anything that requires a wide feature join at request time and pushes you toward precomputed features and a small model. Most federal monitoring work does not need that. Nightly or five-minute micro-batch covers payment integrity, log analytics, and condition monitoring, and micro-batch is dramatically cheaper to build, test, and re-run after a bug.

Cost tracks the same decision. A 100-tree isolation forest over thirty features scores in microseconds per row on a commodity CPU, so a full daily pass over tens of millions of rows is a rounding error on an existing cluster. A windowed autoencoder over dense telemetry may want a GPU node that costs more per month than the analyst hours it saves. Price the whole path (storage for the feature history, the scoring compute, the case-management surface, and the review labor) before choosing the method, because in low-volume programs the review labor dominates everything else and the right optimization is fewer, better alerts rather than faster scoring.

Then monitor the monitor. The failure we see most often is a detector that quietly stopped producing anything for eleven days because an upstream credential expired, discovered only when someone asked why the queue was calm. Alert on alert volume itself against an expected band, on input record counts, on feature null rates, on score distribution shift, and on time since last successful run. A dead detector and a healthy system look identical from the outside, which is exactly why they need to be told apart mechanically.

The federal overlay

In federal work the detector is rarely the whole deliverable. It sits inside a control framework that has opinions about logging, review, and evidence. NIST SP 800-53 Rev. 5 gives you SI-4 for system monitoring, AU-6 for audit record review and analysis, and IR-4 for incident handling, and an anomaly component is usually implementing part of all three. NIST SP 800-92 covers log management practice, SP 800-137 defines information security continuous monitoring, and OMB M-21-31 sets the event-logging maturity tiers that many agencies are still working through. CISA's Continuous Diagnostics and Mitigation program is where a lot of this becomes procurement rather than architecture.

Reporting clocks change the latency requirement. DFARS 252.204-7012 requires a contractor to report a cyber incident to the Department within 72 hours of discovery, and the Cyber Incident Reporting for Critical Infrastructure Act (Pub. L. 117-103) directs covered entities toward a 72-hour report for substantial incidents. A detector whose median time-to-alert is four days is not compatible with either. Mapping detections to MITRE ATT&CK techniques is the common way to show coverage rather than assert it.

On the financial side, the Payment Integrity Information Act of 2019 (Pub. L. 116-117) and OMB Circular A-123 Appendix C set the regime: a program is treated as susceptible to significant improper payments when estimates exceed both 1.5% of outlays and $10 million, or exceed $100 million outright. GAO has reported cumulative government-wide improper payment estimates of roughly $2.8 trillion since fiscal 2003. That is the size of the problem anomaly work is asked to bite into, and it is also why sampling design and statistical defensibility matter more than model novelty in that domain. For sustainment, DoD Instruction 4151.22 establishes condition-based maintenance plus, which is the policy hook for vibration, oil, and telemetry anomaly work on materiel. And the NIST AI Risk Management Framework (AI 100-1) gives you the MEASURE and MANAGE vocabulary that program offices increasingly expect in AI documentation.

When the simpler method is the right answer

Sometimes the correct recommendation is that a machine-learned detector is the wrong instrument. Four situations where we say so plainly. When the process has one governing metric and a physically meaningful specification limit, a control chart is better: it is auditable, an engineer can recompute it, and the false alarm rate is derivable rather than empirical. When the anomaly is defined by policy rather than by statistics (a payment above an authority ceiling, a login from a country the program does not operate in), it is a rule, and dressing a rule as a model only makes it harder to change when the policy changes.

When there are fewer than a few hundred historical events of the type you want to find, no learned model will estimate the tail reliably, and the honest path is a rule set plus a structured random review that builds the label set for a future model. And when nobody can act on an alert within a useful window, the correct deliverable is a report and a process change, not a real-time detector. Building the detector anyway produces the specific outcome we are hired to prevent: a system that is technically live, statistically fine, and completely ignored.

The upgrade path from simple to learned should stay open in every one of those cases. Log the scores from the simple method, capture dispositions from day one, and the labeled corpus that makes a learned model viable accumulates as a by-product of running the boring version well.

Bottom line

Anomaly detection rewards discipline in the places that are easy to skip: naming the receiver, sizing the queue, stating the cost of a miss, splitting time honestly, and measuring at the operating point the customer actually lives at. Our engineers build these systems for federal, state, and commercial customers, and the pattern that holds across all of them is that the winning system is usually the least clever one that fits the budget and explains itself. Get the operating point right and a modest model outperforms an excellent one aimed at the wrong target.

Common objections we hear

Frequently asked questions