Guardrails that do not break the product

Every guardrail has two error rates, and teams measure one

A control that blocks bad output also blocks some good output, and the second number is almost never on the dashboard. It is invisible in a way the first is not: a harmful response that escapes becomes an incident with a name and a review, while a legitimate question wrongly refused becomes a user who quietly stops using the feature. One generates a meeting. The other generates churn nobody attributes to anything. So the incentives push in exactly one direction, and six months later the product has a reputation for being unhelpful and no one can point to the change that did it.

The arithmetic is worth doing once, because it is more brutal than intuition suggests. Suppose half a percent of your traffic is genuinely something you want to block. Your filter catches ninety-five percent of it, which sounds excellent, and it flags two percent of everything else, which sounds negligible. At a hundred thousand requests a day: five hundred are genuinely bad and you catch four hundred and seventy-five. The other ninety-nine thousand five hundred are fine, and you block one thousand nine hundred and ninety of them.

So of roughly two thousand four hundred and sixty-five blocked requests, fewer than one in five deserved it. Four out of five blocked users did nothing wrong, and every one of them saw a refusal. That is not a bad filter. Two percent false positives is a good filter. It is what a low base rate does to any classifier, and it is the reason a guardrail conversation that never mentions base rates is going to produce something users hate.

False positive rate	Legitimate requests blocked per day	Share of blocks that were correct	What users experience
2.0%	1,990	19%	Four in five refusals are wrong. The feature feels broken
1.0%	995	32%	Two in three refusals are wrong. Complaints are constant
0.5%	498	49%	A coin flip. Tolerable only with an easy appeal path
0.1%	100	83%	Mostly correct. This is the target for anything auto-blocking
0.02%	20	96%	Achievable only for narrow, well-specified categories

Assumes 100,000 requests a day, a 0.5% genuine rate, and 95% recall throughout. Change the base rate and every number moves, which is the point: you cannot reason about a threshold without knowing how rare the thing you are catching actually is. Measure it first.

Name the risk before choosing a mechanism

"Guardrails" covers at least seven distinct concerns that get one budget line and one implementation. They have different frequencies, different consequences and different correct mechanisms, and merging them produces a filter that is simultaneously too strict for one and useless for another.

Risk	Mechanism that actually works	Where it belongs	What it costs
Harmful content generated	Provider-level safety plus a narrow output classifier for your domain	Output, before display	Latency; a real false positive rate
Sensitive data in the output	Never retrieve it: filter at retrieval by the caller’s permissions	Retrieval, before scoring	Almost none. Structural, not probabilistic
Injected instructions in content	Capability limits and validated destinations, not detection	Tool design	Design effort once
Actions with consequences	Prepare-and-commit with a human-readable preview	Tool design	One extra call; a real UI
Claims not supported by sources	Programmatic check that every cited passage exists and contains the figure	Output, deterministic	Low, and no false-positive problem
Out-of-scope questions	A cheap classifier with a soft response, not a block	Input	Tens of milliseconds
Statements you are not licensed to make	Template-constrained output for those categories	Generation	Less fluency in a narrow set of answers

Look at the second column and notice how few of the good answers are classifiers. That is the central point of this piece.

Put the control where it is structural, not probabilistic

A model cannot leak a document that was never retrieved. It cannot email an address that is not on the validated list. It cannot delete a record when the deletion tool was not in the list assembled for this caller. It cannot issue a refund without a prepared token that a person approved. None of those require a classifier, a threshold, or a review of borderline cases, and none of them fail on an input someone phrased cleverly.

Compare that to detecting the same risks in text. An output filter looking for leaked account numbers has to know every format an account number takes, including the ones in an attachment nobody thought about. A prompt injection detector is in an open-ended contest with an adversary who can rewrite the input indefinitely. These controls are worth having as a second layer and they are a poor first layer.

So the design order is: remove the capability, then constrain the interface, then check deterministically, then classify, and only then ask a model to judge. Every step down that list adds latency, adds a false positive rate, and adds a threshold someone has to own. Most systems we review have started at the bottom of the list because a classifier is the thing you can add in a day without touching the architecture.

The latency budget nobody wrote down

Controls are additive and they are usually added one at a time by different people. An input classifier, an output classifier, a moderation call, a citation checker, a second model reviewing the first. Each addition is defensible; the sum is a product that takes four seconds to answer a question it used to answer in one.

Rough orders of magnitude, so a design conversation can happen before the code exists. A small local classifier costs tens of milliseconds. A hosted moderation endpoint costs a network round trip, typically well under a couple of hundred milliseconds. A model-based check is a full generation — hundreds of milliseconds to seconds depending on the size of what it reads. A second model reviewing the whole output roughly doubles your total cost and latency, which is a real decision rather than a detail.

Run independent checks in parallel rather than in sequence, and run input checks concurrently with the beginning of generation rather than before it, cancelling generation if a check trips. That one change often recovers most of the added latency for a modest amount of wasted compute.

Streaming makes output checks genuinely hard, because a token you have sent cannot be recalled. Three workable options, each with an honest cost. Buffer a window of output and check on sentence boundaries, which adds a visible stutter but keeps the stream. Stream freely and retract in the interface if a check trips afterwards, which is fast and means a user may briefly see something you intended to block. Or do not stream the categories where the risk is real. Pick deliberately; the common outcome is that nobody picks and output checks are silently skipped in the streaming path while the dashboard still shows them enabled.

A refusal is a product surface, so design it

Most refusals are a dead end: a flat sentence, no explanation, no way forward. For the four in five users who were blocked wrongly, that is the whole experience of your product that day. It also generates support load, because the only remaining option is to ask a person.

Four things make a refusal survivable. Say what category of thing was declined, in plain words, without revealing the rule that fired — enough for a user to understand, not enough for someone to tune against it. Offer the nearest thing you can do, because a large share of refusals are near misses on scope. Give a route to a human, and make it one click. And keep a record the user can reference, so the conversation with support starts from a fact instead of a description.

Never expose the raw classifier output or the matched rule. It teaches evasion and it reads as an accusation. "I cannot help with account changes here — here is how to reach the team that can" is a better outcome on every axis than a score and a category name.

Thresholds are a product decision, and they are not one number

The same model serving an internal tool used by twelve trained analysts and a public feature open to anyone does not warrant the same threshold. Internally, the cost of a miss is low and the cost of a block is high, because the analyst is on deadline and will route around the tool permanently. Publicly, the calculation inverts.

So make thresholds configurable per surface, and where you serve multiple organizations, per tenant. Some customers will want stricter controls than your default and will say so in procurement; others will be furious at anything that slows their staff down. One global constant guarantees you are wrong for most of them.

Consider a middle state as well. Not everything has to be allow or block. A flagged response can be shown with a caution, routed for review before it is sent, or delivered with the sources placed more prominently. Binary controls on a continuous score throw away most of the information the score contains.

Test the controls like a feature, including what must not be blocked

Two sets, and the second is almost always missing. A set of inputs that must be blocked, which everybody builds. And a set of inputs that must not be blocked — ordinary, legitimate, slightly awkward requests drawn from real traffic, including the ones that sit near a boundary. That second set is what stops the ratchet, because without it every incident tightens the threshold and nothing ever loosens it.

Populate the must-not-block set from your review queue: every wrongly blocked request becomes a permanent test case. Report both numbers on every change, and treat a rise in false positives as a regression with the same weight as a miss. Then watch the behavioural signal in production, which is more honest than either: what share of users abandon the session after a refusal, and does that share move when a threshold moves? Users do not file tickets about over-blocking. They leave.

The failure that is invisible by construction

Worth stating on its own because it survives every review. Over-blocking produces no artifact. There is no error in the log, no alert, no incident report, no unhappy message in the channel. There is a user who asked a reasonable question, got told no, and formed a view about whether this tool is worth using. Multiply by months.

The only defenses are measurement and a standing counterweight. Measure the false positive rate on sampled real traffic and publish it beside the miss rate on the same dashboard, at the same size. Give someone explicit ownership of the cost of over-blocking, so the tightening conversation has two sides. Without that, the ratchet only turns one way, and every individual turn is justified.

A two-week pass over controls already in production

Days three to five are the ones that change minds. A room that has been arguing about thresholds from instinct behaves very differently once it has read two hundred actual blocked requests, and the discovery is nearly always the same: the majority of them were fine.

The mistakes we are called in to fix

• A threshold set during an incident and never revisited once the pressure lifted
• No measured false positive rate, so every tightening is free and every loosening is a fight
• Classifiers doing work a permission check should do, probabilistically and slowly
• Five checks in sequence where four could run in parallel
• Output checks silently skipped on the streaming path while the dashboard shows them on
• Refusals with no reason, no alternative and no route to a person
• The raw rule or score shown to the user, teaching evasion and reading as an accusation
• One global threshold across an internal tool and a public product
• No must-not-block test set, so the ratchet only turns one way

Before you ship a control

• The risk is named specifically, not as "safety"
• You checked whether it can be structural before reaching for a classifier
• The base rate of the thing being caught has been measured, not assumed
• Both error rates are on the dashboard, at the same size
• Every block is logged and a sample is labelled by a person on a schedule
• Independent checks run in parallel and the latency cost is written down
• The streaming policy is an explicit decision, not an omission
• Refusals name a category, offer an alternative and route to a human
• Thresholds are configurable per surface and per tenant
• A must-not-block set exists and gates releases alongside the must-block set
• Someone owns the cost of over-blocking by name

Bottom line

Guardrails are an engineering trade-off, not a virtue, and the trade is invisible unless you measure both sides. Start by naming the specific risk. Prefer controls that make the bad outcome impossible over controls that try to recognize it, because structural limits have no threshold and no false positives. Measure the base rate before choosing a cut point, or the arithmetic will surprise you in a direction your users feel. Design the refusal, since for most of the people who see one it is wrong. Put both error rates on the same dashboard and give the second one an owner. A product that is safe and unusable has not managed risk; it has moved it somewhere nobody is counting.

Frequently asked questions