Skip to main content
Product Engineering

Notifications people do not turn off

Every notification channel has one life. It gets attention until the day it interrupts someone for nothing too many times, and after that it is furniture. The engineering that prevents this is not about delivery. It is about precision, budget and restraint.

A channel dies once

The interesting property of notifications is that trust in them is not continuous. A person does not gradually value your alerts less. They read them, read them, read them, and then one week they stop — and after that no improvement in content brings them back, because they are no longer looking. We have watched teams fix the underlying precision problem, ship a genuinely better alert, and see no change in response time for months, because the humans had already rewired around it. The channel had died and nobody had recorded a death.

That asymmetry is the reason this is an engineering problem worth real attention rather than a feature you add in a sprint. You are spending a finite, non-renewable resource — a specific person's willingness to look — and almost every default in every notification framework spends it faster than necessary.

Two populations are in scope here and they have different economics. Operators receive alerts about systems: on-call engineers, analysts working a review queue, support staff. Their attention is expensive and measurable, and a bad alert costs a quantifiable amount of it. End users receive product notifications: something happened, someone mentioned you, an approval is waiting. Their attention is cheaper per unit but they have an exit you cannot appeal, which is the mute button and the unsubscribe link. The mechanics below apply to both. The thresholds differ.

You are probably here because

  • A channel gets hundreds of messages a day and nobody has opened it in a month
  • A retry somewhere sent the same message four times and a customer noticed
  • An incident produced 900 alerts for one root cause
  • Mute and unsubscribe rates are climbing and nobody can say which notification is responsible

The first and third are precision and grouping problems. The second is a delivery-idempotency problem. The fourth is a measurement gap, and it is the one that makes the other three invisible until they are severe.

Measure action, not delivery

Notification systems ship with delivery metrics because delivery is what the sender controls: sent, delivered, bounced, opened. All four can be excellent while the channel is dead, and none of them will tell you.

The metric that predicts survival is the share of notifications that lead to an action within a reasonable window. For an operator alert, action means acknowledging, investigating, or resolving — anything that shows a human engaged with this specific item rather than skimming past it. For a product notification, it means clicking through and doing the thing the notification was about.

Our working rule, from watching a fair number of these decay: an alert stream where fewer than roughly one in five items produces an action is on a path to being ignored, and one below one in ten is already being ignored whether or not anyone admits it. Those are not laws of nature and your context can move them — a stream where the rare true positive is catastrophic can sustain a much lower rate as long as the humans believe it. But if you have no number at all, you have no way to know which side of the line you are on, and the first evidence will be a missed incident.

Three companions to that metric. Mute rate, which is the honest one, because muting is a considered act. Time to first action, whose slow drift upward is the earliest visible sign of decay. And action rate by notification type, which is what turns “alerts are noisy” into “these three types are noisy and the other eleven are fine.”

You are spending a specific person’s willingness to look. It does not replenish, and almost every framework default spends it faster than necessary.

The alert budget

Here is a framing that changes design conversations, particularly where a model or a rules engine is generating the alerts. Start from capacity rather than from the detector.

Count the reviewers. Multiply by the minutes per day they can genuinely spend on this, which is much less than their working hours — one to two hours is realistic for people who also have other duties. Divide by the average handling time per item. That number is your alert budget, and it is fixed by staffing, not by how many things the detector considers interesting. Six reviewers, ninety minutes each, twelve minutes per case, gives about forty-five cases a day. If the detector produces four hundred, you have not built an alerting system. You have built a queue that will be worked by whoever is least busy, in whatever order the interface happens to present, which is a bad way to allocate scarce expert attention.

What follows from that is the part teams resist. The threshold is set by the budget, not by the model's optimum. Tuning for the best balanced accuracy score and then shipping whatever volume that implies is exactly backwards: you have optimised a number nobody is accountable for and handed the capacity problem to the people receiving the output. Rank by expected value, take the top N where N is the budget, and be honest in the documentation about what is not being reviewed. Everyone already knows there is a cut. Writing it down converts an unmanaged risk into a managed one.

What most affects whether a channel keeps its audience — our ranking

Share of items that lead to an action
95
Volume matched to a real review budget
88
Grouping and deduplication of related events
80
The message says what to do next
72
Per-recipient controls and quiet hours
58
Speed of delivery
26

Our judgment, not a study. Delivery speed ranks last because it is the thing most often optimised and the thing least often responsible.

Deduplicate and group before you send

One root cause produces many events. A dependency goes down and every service that calls it complains. A batch job fails and every one of its four hundred items reports a failure. A configuration error affects nine thousand accounts and each one generates a notification.

Grouping is not cosmetic. It is the difference between a message that says “this dependency is down, 900 downstream failures, first at 14:02” and 900 messages that a human has to reconstruct that sentence from. Three mechanisms cover almost everything.

A grouping key on every event. Derived from what a human would consider the same problem — usually the failing component plus the error class, not the individual item. Everything sharing a key within a window becomes one notification with a count.

A short delay before sending. Ten to sixty seconds is enough for the related events to arrive so the first notification carries the full picture. Almost nothing genuinely loses value in a minute, and the exceptions are few enough to name explicitly and exempt.

Update rather than resend. If the situation changes, edit the existing notification where the channel supports it, or send a clearly marked update. Sending a fresh message for each state change of one incident is how a single outage becomes forty interruptions.

Then add the digest. A large share of what teams currently send in real time is genuinely a daily-summary item, and moving it there is usually the single largest reduction in volume available. The test is direct: if nobody would act differently by knowing this within the hour, it belongs in a digest.

Route by what the recipient should do

Severity levels drift into meaninglessness because they are assigned by the person writing the check, who is naturally biased toward their own thing mattering. Define the ladder by consequence instead, and make the definition the thing that must be argued.

TierDefinitionChannelVolume expectation
Wake someoneOngoing harm that a person can stop right nowPhone, pageA handful a month, at most
Interrupt in hoursNeeds a human today; waiting until tomorrow costs somethingChat, pushA few per day per team
Queue itNeeds a human this week, order matters, timing does notA work queueWhatever the review budget allows
DigestWorth knowing, nobody acts differently within a dayDaily emailUnbounded, it is a summary
Log onlyWorth having when investigating something elseNo notification at allUnbounded

The bottom row is the one to fight for. A large fraction of what is sent as a notification is a log line that somebody wanted to feel confident about. Making “this is recorded and searchable, and nobody is interrupted” a legitimate, respectable outcome is most of the volume reduction in a typical system.

Delivery mechanics, where the embarrassing failures live

Everything above is about what to send. This is about not humiliating yourself in the sending.

Idempotency per recipient, per channel. Delivery is at-least-once by construction: a worker sends the email, dies before recording that it sent, and the retry sends it again. The fix is a uniqueness constraint on the tuple of event identifier, recipient and channel, written before the send attempt and checked on retry. Without it, every deploy during a backlog drain is a chance to mail someone four times.

A hard cap per recipient per hour. Not a nicety, a circuit breaker. The classic incident is a loop — a webhook that triggers a state change that triggers the webhook — and it is discovered by a customer receiving thousands of emails overnight. A cap of a few dozen per recipient per hour, with the overflow collapsed into a single summary and an internal alarm, converts a reputational event into a graph.

Quiet hours in the recipient's zone. Store the zone, not an offset. Suppress non-urgent notifications overnight and deliver them in the morning. The urgent tier is exempt by definition, which is what the tier is for.

Deliverability is a real discipline. If you send email at any volume, you need authentication records correctly published and aligned, a dedicated sending domain or subdomain, gradual warm-up on new sending infrastructure, and automated handling of bounces and complaints. Suppress hard bounces permanently and complaints immediately and forever, including for transactional mail — continuing to send to someone who marked you as spam is how a sending reputation is lost for everyone. Complaint rates are watched by the large mailbox providers at a threshold in the region of one in a thousand, and the consequence of crossing it is not a warning, it is your mail quietly going to spam. Watch your own rate as a first-class metric.

Mobile push has silent failure modes. Tokens rotate and expire. A push accepted by the platform is not a push shown to a person. Reconcile against in-app receipt rather than trusting the send response, and prune dead tokens or your delivery numbers will drift steadily away from reality.

Send us a week of your alert stream.

Email the volume by type, how many led to an action, and how many people are meant to read them to contact@precisionfederal.com. You get back a short written note on where the budget actually sits, which types to collapse and which to stop sending. One business day. No charge, no meeting, no deck.

contact@precisionfederal.com

Write the message for someone who is doing something else

Your notification arrives in the middle of another task. It has a few seconds to convey what happened, whether it matters, and what to do. Most notifications spend those seconds on the sender's context instead of the reader's.

Four things belong in every operational notification, in this order: what happened in one line, the specific entity it happened to, why it needs a person, and the action with a link straight to the place that action happens. Everything else is detail that belongs behind that link.

The failure worth naming is the notification that reports a state without an implication. “Job 44812 finished with warnings” makes the reader do the work of finding out whether that matters, and if the answer is usually no, they will stop checking within two weeks. Either say what the warning means and what to do, or move it to the digest.

And do not send a notification that cannot be acted on. “An error occurred processing your file” with no error, no file identifier and no next step is worse than silence, because it converts a solvable problem into anxiety plus a support ticket.

Treat unsubscribe as data

Every mute and every unsubscribe is a person telling you something specific, and most systems record only that it happened. Capture which notification type triggered it and offer granular options at the moment of the decision — this type, this project, all of it — because the person who wanted to stop one thing will turn off everything when that is the only button available.

Then review the mute reasons monthly alongside the action rates by type. Between the two you can usually name the two or three notification types responsible for most of the damage, and they are frequently not the ones anyone suspected. In our experience the offender is rarely the loudest type; it is a moderate-volume one that is almost never actionable.

If nobody would act differently by knowing this within the hour, it belongs in a digest. That single test removes most of the volume in a typical system.

The mistakes we get called in to fix

  • Delivery rate reported as health, while nobody has opened the channel in weeks
  • Threshold set by model accuracy rather than by how many cases can be reviewed
  • One notification per event, so a single root cause produces hundreds
  • No idempotency on event, recipient and channel, so retries send duplicates
  • No per-recipient rate cap, so a feedback loop mails someone all night
  • Severity assigned by the author of the check, with no consequence-based definition
  • Unsubscribe as a single global switch, so one annoyance turns off everything
  • Complaints and hard bounces not suppressed, taking the sending reputation down with them

A three-week reset for a channel already in trouble

Notification Reset

1
Instrument action rate, mute rate and time-to-first-action by notification type
Days 1–3
2
Compute the real review budget from headcount, minutes and handling time
Day 4
3
Retire or demote every type whose action rate cannot be defended
Days 5–8
4
Add grouping keys, a short send delay, and a digest tier
Days 9–12
5
Per-recipient idempotency, hourly caps, quiet hours, bounce and complaint suppression
Days 13–16
6
Rewrite the surviving templates around the action, then re-announce the channel
Days 17–21

The last half of step six is not decoration. If a channel has already lost its audience, quietly improving it does not bring anyone back. Tell the recipients what changed, what you stopped sending, and what the remaining messages now mean. That is the only mechanism we have seen actually restore attention, and it works exactly once, so do it after the fix rather than before.

Before you turn a new channel on

  • Action rate is instrumented per notification type from the first day
  • The expected volume is compared against a computed review budget
  • Every event carries a grouping key, and related events collapse into one message
  • A digest tier exists and log-only is a respectable outcome
  • Severity is defined by consequence, not by the author of the check
  • Idempotency covers event, recipient and channel
  • A per-recipient hourly cap exists with an internal alarm on the overflow
  • Quiet hours use the recipient’s stored time zone
  • Bounces and complaints are suppressed automatically and permanently
  • Every message names the action and links straight to where it is taken

Bottom line

Notifications fail on precision and volume, not on delivery. Measure whether people act, size the stream to the attention that actually exists, group aggressively, and make silence a legitimate outcome for anything that is not actionable. Then do the unglamorous delivery work — idempotency, caps, suppression, quiet hours — because that is where the incidents that cost trust in a single night come from. A channel that respects its readers keeps them, and the only reliable way to demonstrate respect is to send less.

Frequently asked questions

What is the right metric for a notification system?

The share of notifications that lead to an action within a sensible window, broken down by notification type. Delivery, open and bounce rates can all look healthy while nobody reads the channel. Track mute rate and time-to-first-action alongside it, since both drift before anyone reports a problem.

How many alerts is too many?

More than the people receiving them can review. Compute it: reviewers, multiplied by the minutes per day they can realistically spend, divided by the handling time per item. That capacity is fixed by staffing, so it should set the alerting threshold rather than the other way round.

Why do the same notifications get sent twice?

Because delivery is at-least-once. A worker sends and then dies before recording the send, and the retry sends again. Enforce a uniqueness constraint on event identifier plus recipient plus channel, written before the attempt, and duplicates stop being possible rather than becoming unlikely.

How do you reduce alert volume without missing things?

Group related events under a shared key so one root cause produces one notification with a count, delay sending by a short window so the first message is complete, and move anything nobody would act on within the hour to a daily digest. Nothing stops being recorded; it stops interrupting.

Can a channel people have stopped reading be recovered?

Only with an explicit relaunch. Fix the precision problem first, then tell the recipients what you stopped sending and what the remaining messages now mean. Quietly improving a dead channel does not work, because the people who left are no longer looking at it to notice.

1 business day response

Alert channel nobody reads any more?

Send a week of volume by type, the action rate if you have it, and how many people are meant to be reading. Our engineers will come back with the real budget, the types worth keeping and the delivery gaps that will bite — or take the rebuild as a scoped piece of work. Email bo@precisionfederal.com.

Email an engineerCapabilitiesMore insights →
Product EngineeringAlertingEvent SystemsOperations