The arithmetic that decides everything
Start with a detector that is 90% sensitive and 99% specific. Those are respectable numbers and most teams would ship them. Now run it over 50,000 entities a day — accounts, devices, endpoints, SKUs, whatever your units are — where roughly fifty of them are genuinely worth someone's attention. The detector finds 45 of the 50. It also fires on about one percent of the other 49,950, which is 500 alerts. The queue that lands in front of your team is 545 items long and 92% of it is noise. Nobody is going to work that queue in week six. They will filter it into a folder, and the 45 real findings go into the folder with it.
This is the whole problem, and it is arithmetic rather than modelling. Precision — the share of alerts that turn out to be real — is governed mostly by how rare the event is. Sensitivity and specificity are properties of the model; precision is a property of the model and the world together. A detector can be excellent and useless at the same time, and the number that tells you which is not on the model card.
The second thing this arithmetic tells you is where the leverage is. Raising sensitivity from 90% to 95% adds two or three true findings. Raising specificity from 99% to 99.9% removes 450 false ones. When events are rare, essentially all the value is in the false-positive side, and essentially all the effort in most projects goes into the other one.
You are probably here because
- The detector works and the team has quietly stopped reading its output
- Every alert on the first Monday of the month is the month-end batch, again
- One underlying event produced forty alerts and nobody could tell it was one event
- The offline evaluation says F1 of 0.91 and the on-call engineer says it is unusable
Each of these is a design problem with a known fix, and none of them is fixed by a better model.
Design backwards from the alert budget
Before choosing a method, get one number: how many alerts can this team work per day, at the quality of investigation you actually want? Ask what a single review involves and time it. Fifteen minutes of pulling context, checking two systems and writing a note is common. At that rate one person contributes maybe twenty-five reviews in a working day once meetings and everything else are subtracted, and a three-person rotation gives a budget somewhere near seventy-five.
That number is the design constraint. If the budget is seventy-five alerts a day and you expect fifty real events, the detector must operate at roughly 65% precision to be worth having, and at that point you are asking for a specificity around 99.9% rather than 99%. If the model cannot reach it, the answer is not to ship anyway and hope. The answer is to change what you are detecting: narrow the population, raise the severity floor, group alerts into incidents, or add a cheap second stage that filters the first stage's output.
| Event rate | At 99% specificity | At 99.9% specificity | What that means for the queue |
|---|---|---|---|
| 1 in 100 | Precision about 48% | Precision about 90% | Workable either way; tuning is a comfort question |
| 1 in 1,000 | Precision about 8% | Precision about 47% | The 99% version is twelve dead ends per finding |
| 1 in 10,000 | Precision about 1% | Precision about 8% | Neither is shippable as a single stage |
| 1 in 100,000 | Precision under 0.1% | Precision about 1% | Needs staged filtering or a much narrower population |
All four rows assume 90% sensitivity. The table is worth putting in front of whoever asked for the system, early, because it reframes the conversation from “how accurate is the model” to “how rare is the thing and how many people do we have,” which are the two questions that determine the outcome.
Three different problems share one name
“Anomaly detection” covers at least three tasks that want different methods, and conflating them is a reliable source of disappointment.
Point anomalies. A single observation is implausible on its own: a payment of a size never seen on this account, a sensor reading outside physical limits. Robust univariate statistics handle most of these, and a rule usually beats a model because the rule is explainable and does not drift.
Contextual anomalies. The value is ordinary in general and wrong for its context. Traffic of 400 requests a second is normal at noon and alarming at four in the morning; a temperature is fine in July and not in January. These need the context in the model — time of day, day of week, entity, segment — and they are where most real detection value sits.
Collective anomalies and changepoints. No single point is unusual, and the sequence is: a slow ramp, a level shift, a variance change, a correlation that broke. Changepoint methods and forecast-residual monitoring find these; point detectors never will, because there is no point to flag.
Ask which of the three the requester means. Frequently they mean the third one and the team builds the first, then everyone is puzzled that a gradual doubling over three weeks produced no alert.
The calendar is most of your false positives
In the detectors we are asked to repair, a large share of false alarms are ordinary, knowable calendar events. Month-end close. Quarter-end. Public holidays and the shifted business days around them. Daylight-saving transitions, which give a duplicated or missing hour and reliably produce a spike or a hole once a year. Marketing sends. Deploy windows. Payroll dates. Batch jobs whose schedule nobody documented.
The remedy is unglamorous and effective: put the calendar in the model. Include holiday and month-end indicators as features, or maintain an explicit suppression window with an owner and an expiry. Keep the two clearly separated, because a suppression window is a decision to be blind for a period and it should be visible, time-boxed and reviewed, not a permanent silence someone added in a hurry.
Where false positives come from — typical mix on a detector we are asked to fix
Our rough working split, shown to set expectations. Measure it on your own alerts before acting — it is a two-day exercise and it usually reorders the roadmap.
The point of that split is the last row. Modelling is generally the smallest contributor to a noisy queue, and it is where teams instinctively spend. Sampling two hundred of your own false positives and labelling their cause is a two-day exercise that redirects a quarter of work.
Per-entity baselines, and the cold-start problem they create
A single global threshold across heterogeneous entities is the fastest way to a queue dominated by your largest accounts. Normal for one customer is an outrage for another. Per-entity baselines fix this, and they introduce two new problems worth planning for.
The first is thin history. An entity with three weeks of data has no reliable seasonal profile, and an entity with a week has nothing. Pooling helps: estimate a group-level profile for similar entities and shrink each entity's estimate toward it in proportion to how little data it has. This is standard hierarchical modelling and it is worth the modest complexity, because the alternative is a wave of false alarms on everything new.
The second is the state explosion. One baseline per entity per metric per hour-of-week becomes millions of parameters that must be stored, refreshed, versioned and explained. Decide early how often baselines are recomputed and how much history they use, and write the answer down where the on-call engineer can find it, because “why did this fire” is unanswerable without it.
Set a minimum-history rule as policy: below some number of observations, an entity is monitored by the pooled profile and not by its own, and it is exempt from paging. That single rule removes a category of alert that is never actionable.
An incident is not an alert
One underlying cause usually shows up in many places. A bad deploy moves error rate, latency and throughput on nine services at once. A broken upstream feed makes forty downstream metrics look strange simultaneously. If each of those is an independent alert, the team's experience is forty notifications for one problem, and their conclusion is that the system is noisy even though it was correct forty times.
Grouping is therefore part of the detector, not a nicety on top of it. Three rules go a long way. Cluster alerts that fire within a short window on entities that are related — by dependency graph, by shared upstream, by ownership. Deduplicate repeats of the same condition on the same entity, and update the existing item rather than creating a new one. Suppress downstream alerts when an upstream condition is already open, which requires a dependency map you probably have somewhere in your service catalogue.
Where a dependency graph does not exist, correlation over history is a workable substitute: entities whose alerts historically co-occur get grouped. It is cruder and it removes most of the volume, which is the goal.
Keep a small random audit sample outside the detector
You only learn the truth about cases you alerted on, so the labelled data grows only where the model already looks. Retraining on it makes the model more confident about the region it already covers and blind everywhere else. The fix is a small random sample — a few dozen cases a week — reviewed regardless of score. It is the only source of unbiased information about the events you are missing, it gives you an honest denominator for recall, and it is the first thing cut when the team gets busy. Protect it.
Send us a week of your alerts and we will tell you where the noise is.
Email a week of alert records, the disposition of each one where you have it, and how many your team can work per day to contact@precisionfederal.com. You get back a short written note with the measured precision, the cause split of the false positives, and the three changes we would make first. One business day. No charge, no meeting, no deck.
contact@precisionfederal.comScoring the detector honestly
Offline metrics on this problem are unusually easy to fool, and a good score is worth very little unless you know how it was computed.
Score alerts, not timesteps. The team experiences alerts, so evaluate alerts: of the items delivered to a person, what share were real? Point-level accuracy over a time series has almost no relationship to that experience, and a metric averaged over millions of normal timestamps is dominated by the easy majority.
Beware the segment-credit convention. A widespread evaluation practice in time-series work is to count an entire true anomalous segment as detected if any single point inside it was flagged. It is intuitive and it inflates scores dramatically, to the point that near-random scoring can look strong on long segments. If a reported F1 was computed that way, it is not comparable to anything and it is not evidence the detector works.
Report event recall and alert precision separately. Two numbers with plain meanings: of real events, how many produced an alert; of alerts, how many were real. Anyone can act on that pair. A single combined score hides which side of the trade the system is failing on.
Measure time to detect. For most operational problems, an alert twenty minutes in is worth far more than one four hours in, and detectors with identical precision and recall can differ by an order of magnitude on this axis. It rarely appears in evaluations and it is often the number the business cares about most.
Evaluate on a forward window. Fit on the past, score on the future, always. Random splits leak in time series through overlapping windows and shared normalisation, and they produce numbers that never survive contact with production.
The escalation ladder
Not every finding deserves the same delivery. Four tiers cover almost everything, and putting each detector explicitly in one of them is a ten-minute exercise that changes how the system feels.
Page. Someone is woken. Reserved for conditions that are both urgent and actionable at that hour, with a runbook. If a page has no action, it is not a page.
Ticket. Enters a work queue with an owner and a due date. This is where most real detections belong, and where the alert-budget arithmetic applies.
Digest. A daily or weekly summary, grouped and ranked. Ideal for slow drifts and for anything where the right response is a pattern rather than a case.
Dashboard only. Recorded, visible when someone looks, notifying nobody. A perfectly respectable destination for a detector you are still calibrating.
Start new detectors at dashboard, promote them to digest when precision holds for a fortnight, and only then to ticket. Very few things should ever be promoted to page. The instinct to launch a new detector straight into someone's phone is how trust is spent before the calibration is finished.
When a threshold beats a model
A large fraction of production detection is a well-chosen rule over a well-chosen quantity, and this is not a failure of ambition. A rule is inspectable, its behaviour on the day it fires is explainable in one sentence, it does not need retraining, and it does not silently change when an upstream distribution shifts. When the underlying quantity is understood — a rate that should never exceed a physical limit, a ratio with a known operating band — a rule is simply the better engineering choice.
Models earn their place when the normal region genuinely depends on many interacting variables, when there are far too many entities to hand-tune, or when the seasonal structure is strong enough that a static band is wrong most of the day. Even then, the winning architecture is often a cheap model that proposes and a rule set that disposes: a first stage with high recall, a second stage of deterministic checks that removes the known-benign patterns. The second stage is where the calendar, the maintenance windows and the known-noisy entities live, and it is easy to reason about and easy to change.
A four-week build
Detection Build
Step three placed before model tuning is deliberate and it is the ordering teams argue with. Grouping typically removes more queue volume than any modelling change available in the same time, and it does so without touching sensitivity, which means nothing real is lost.
Common objections
Can we not just raise the threshold until the queue is manageable?
You can, and it is sometimes right, but understand what it costs. Raising a threshold trades recall for precision along a fixed curve, so a queue reduced tenfold has usually given up a meaningful share of true findings. The changes described here — calendar features, per-entity baselines, incident grouping — move the curve rather than sliding along it. Do those first, then set the threshold.
We have almost no labels. Is supervised detection out of reach?
Not permanently. Start unsupervised or rule-based, then treat every reviewed alert as a label and keep the review capture structured from day one — a disposition field and a cause field are enough. Within a couple of months most teams have several thousand labelled cases, which is plenty for a second-stage classifier that filters the first stage. The random audit sample is what keeps that dataset from being biased toward what you already catch.
How often should baselines be recomputed?
Slowly enough that a genuine problem cannot be absorbed into normal. A baseline refreshed nightly on a trailing 28-day window will quietly learn a degradation that developed over three weeks and stop alerting on it. Common practice is a longer window for the seasonal shape and a shorter one for the level, with a rule that excludes periods flagged as incidents from the training window entirely.
Should the same detector serve on-call and the analyst queue?
Usually not. They have different budgets and different costs of being wrong. A single detector run at two thresholds and delivered through two tiers is the simplest arrangement: the high-precision cut pages, the lower cut becomes tickets or a digest. What does not work is one threshold chosen as a compromise, which is too noisy for on-call and too conservative for review.
The mistakes we are called in to fix
- A threshold chosen from a curve, never from the number of alerts a team can actually work
- No calendar in the model, so month-end and holidays generate the same alerts every cycle
- One global threshold across entities that differ in volume by three orders of magnitude
- Forty alerts for one incident, with no grouping, dedup or upstream suppression
- Retraining only on reviewed alerts, which teaches the model the region it already covers
- An F1 computed with segment credit, reported as if it were comparable to anything
- A random train/test split on a time series, leaking the future into the fit
- Every detector wired to a page, spending the team's trust in the first fortnight
Before you turn it on
- The event is defined in one sentence and its base rate is measured, not guessed
- The alert budget came from the people who will work the queue
- Required precision is derived from that budget and written down
- Calendar effects are modelled or explicitly suppressed with an owner and an expiry
- Entities below a minimum history are pooled and exempt from paging
- Alerts group into incidents, deduplicate, and suppress under an open upstream
- Evaluation is alert-level and forward in time, with event recall reported separately
- A random audit sample runs outside the detector and is protected from cuts
- Every tier has a runbook, and anything that pages has an action at that hour
- Precision is re-measured on a schedule, with a stated level at which the detector is demoted
Bottom line
A detector is a system for spending human attention, and attention is the scarce input. Work out how much of it you have, compute what precision that implies at your event's base rate, and build backwards from there. Most of the distance is covered by unglamorous things: model the calendar, give each entity its own baseline with pooling for the new ones, and collapse related alerts into one incident before anybody sees them. Save the modelling for the residue that survives. Then measure precision the way the team experiences it, keep a small unbiased audit sample so you know what you are missing, and start every new detector on a dashboard rather than on someone's phone. A detector that is trusted at 60% precision beats one that is ignored at 85%.
Frequently asked questions
Because the events are rare. One percent of a large normal population is a much bigger number than all of the real events combined. At an event rate of one in a thousand, a detector with 90% sensitivity and 99% specificity runs at roughly 8% precision, so twelve of every thirteen alerts are dead ends. Precision depends on the base rate, which is why it must be computed before the system is designed rather than discovered after launch.
Usually both, in stages. An unsupervised or rule-based first stage gives high recall without labels; a supervised second stage, trained on the dispositions your reviewers produce, removes the known-benign patterns and is where most of the precision comes from. The requirement is that the review loop captures structured labels from day one, and that a random audit sample keeps the training data from drifting toward what you already detect.
Separate the shape from the level. Estimate the recurring weekly and daily profile over a long window, estimate the current level over a short one, and hold an explicit calendar of known events instead of hoping the model learns them from a single occurrence. Annual events like holidays have very few historical instances, so an explicit indicator is more reliable than anything estimated from the data.
It follows from the budget rather than from a benchmark. If the team can work seventy-five items a day and there are fifty real events, you need roughly two thirds of alerts to be real. Where the investigation is cheap and the miss is expensive, teams run happily at 20 to 30%. Where each review takes an hour, anything under 50% will be abandoned. Set the number with the people doing the work, and re-measure it monthly.
Only from cases you review without regard to the score. Reviewed alerts tell you about precision and say nothing about recall, because they are selected by the model. A small random audit sample — a few dozen a week — gives an honest denominator, surfaces failure modes the detector has never flagged, and is the first thing to protect when the team is under pressure.