Webhooks that do not lose events

The integration that worked until it did not

Webhook integrations almost never fail on the first day. They fail on a Tuesday four months in, when a payment shows as unpaid, an account stays provisioned after cancellation, or a shipment never leaves the warehouse because the order event that would have released it went into a process that restarted mid-request. Nothing in the logs says lost. The sender's dashboard says delivered. Your dashboard says the endpoint is healthy. Somewhere between those two green lights there is a customer with a problem, and the only person who noticed was the customer.

The reason this happens is structural, not sloppy. A webhook consumer written the obvious way is a request handler that parses a payload, does the work, and returns 200. That shape puts a database write, a third-party API call, an email send, and a queue publish inside the window the sender is holding open, and it puts all of them on the wrong side of the acknowledgement. Any failure after the first side effect and before the response leaves the sender believing something your system does not believe. That gap is where events live when they are lost.

The fix is not more retries or a bigger timeout. It is a different division of labor between the part of your system that receives and the part that acts. That division is the whole article, and it takes about a day to implement in most codebases.

A webhook is not a data feed

The most useful reframe we give teams is this one. Webhooks are not a data feed. They are a notification that your copy of somebody else's state is now stale. Treat the payload as authoritative and you inherit every problem the sender has: retries that arrive out of order, payloads serialized from a snapshot taken before a later write, redeliveries of an old version of an object, and truncated fields that the sender's API would have returned in full.

Treat the payload as a hint and most of those problems become irrelevant. The event tells you which object changed. You go read that object from the sender's API and act on what you read. It costs one extra round trip and it makes the ordering problem disappear for a large class of integrations, because whatever order the notifications arrive in, the state you fetch is the current one.

That trade is not free. Rate limits are real, high-volume event streams can generate more fetches than the API allows, and some events describe a transition rather than a state, so there is nothing to re-read. Payment intent transitions and one-time delivery receipts fall in that category. But the default should be hint-plus-fetch, with payload-as-truth as the deliberate exception, and most teams have it backwards.

The eight places an event actually goes missing

Loss has a small number of causes, and naming which one you are looking at saves days. In rough order of how often we find them:

One: the response beat the durability. The handler did the work in-process, returned 200, and then the pod was killed, the connection dropped, or the transaction rolled back on an error the framework swallowed. The sender recorded a success. Nothing else did.

Two: the deploy ate it. A rolling restart terminated a process holding in-flight requests. The sender saw a connection reset or a 502 from the load balancer, retried a few times inside the same deploy window, hit the same wall, and gave up. Deploys are a delivery failure mode and almost no runbook treats them as one.

Three: the endpoint was slow, not down. The handler did four seconds of work under normal load and eleven seconds under a traffic spike. Timeouts across common senders run from a couple of seconds to about half a minute, so a handler with variable latency is a handler that silently drops its tail. Worse, several platforms treat sustained timeouts as failure and disable the endpoint entirely.

Four: the proxy rejected it before your code ran. nginx ships with client_max_body_size at one megabyte and returns 413 for anything larger. A serverless function behind an API gateway carries a payload ceiling of its own. Batch-style events with large arrays cross those lines eventually, and the request never reaches application logs, so the investigation starts in the wrong place.

Five: signature verification failed for a boring reason. A framework parsed and re-serialized the JSON body before the verification code saw it, so the bytes no longer matched. Or clock skew pushed the timestamp outside the tolerance window. Or a key rotation went out with a single active key and every delivery in the cutover window was rejected as forged.

Six: the queue lost it after the ack. The receiver did its job and published to a broker, but the publish was fire-and-forget with no confirm, or the consumer acknowledged on dequeue rather than on completion, or the dead-letter queue exists and nobody reads it. A dead-letter queue nobody monitors is a place events go to be forgotten politely.

Seven: deduplication discarded a real event. The dedup key was derived from something insufficiently unique, such as an object id plus an event type, so a second legitimate change to the same object inside the window looked like a duplicate and was thrown away. This one is nasty because it looks like correct idempotency behavior in every log line.

Eight: the sender never sent. Sometimes it really is the other side. Endpoints get auto-disabled after failure streaks, subscriptions get scoped to event types someone changed, and an expired or misconfigured TLS certificate on your side quietly turns every delivery into a connection error. Every one of those shows up as absence, and absence is invisible unless something is watching for it.

Where it dies	What you see	What actually fixes it
Work done before the ack	Sender says delivered, you have no record	Durable write first, process after
Rolling deploy	Bursts of 502s clustered at release time	Draining, preStop delay, in-flight completion
Slow handler	Sender-side timeouts, endpoint auto-disabled	Move all work off the request path
Proxy or gateway limit	413 or 502 with no application log	Raise body limits, check gateway ceilings
Signature mismatch	401 spikes, often after a framework upgrade	Verify raw bytes, dual-key rotation
Broker or consumer	Inbox rows stuck in received	Ack on completion, alert on oldest-unprocessed age
Over-eager dedup	Nothing at all, until a customer complains	Key on the sender's event id, never on content
Never sent	Silence	Reconciliation sweep against the source of truth

The receiver does exactly two things

Here is the rule that removes most of the list above. The HTTP handler verifies the signature and durably writes the raw event. Then it returns 200. It does nothing else. No business logic, no outbound calls, no email, no cache warming, no analytics, and no clever inline processing for the fast path.

A 200 means you have written the event down somewhere it will survive a power loss. If it means anything else, you are lying to the sender, and the sender believes you. Everything the event triggers happens afterward, in a worker, under your own retry policy, with your own failure handling, on your own clock.

Once the receipt path is that thin, it is fast and predictable, which matters because sender timeouts are unforgiving and variance is what kills you. We budget the receipt path in the low hundreds of milliseconds at p99 and treat anything above it as a defect worth a ticket.

The inbox table, concretely

The durable write goes to an inbox table, and it is deliberately dumb. Columns: a surrogate primary key, the sender name, the sender's event id, the event type, the raw body as bytes rather than parsed JSON, the headers you care about, a received timestamp, a status, an attempt count, and a last-error field. A unique index on the pair of sender and event id. That is the entire schema and it has survived every integration we have put it behind.

Storing the raw bytes matters more than it looks. When something goes wrong six weeks later, the argument is always about what the sender actually sent, and a parsed-and-normalized copy cannot settle it. Raw bytes also let you replay through a fixed parser after you find a bug in the old one, which is the cheapest recovery path there is.

A worker then claims rows with SELECT ... FOR UPDATE SKIP LOCKED, processes them, and marks them done in the same transaction as the side effect where the side effect is in the same database. Where it is not, use an outbox row committed with the business write and a separate publisher, which is the standard way to avoid the dual-write problem. If your queue infrastructure is already solid, publish from the worker instead of from the handler. The point is that the broker sits behind the durability boundary, not in front of it.

You do not always need a separate broker. A Postgres table with SKIP LOCKED handles a surprising amount of volume, keeps the transaction boundary in one place, and removes an entire piece of infrastructure from the on-call surface. Reach for a dedicated broker when fan-out, ordering guarantees, or cross-service consumption justify it, not by reflex.

Idempotency, and why the key matters more than the mechanism

At-least-once delivery means duplicates are normal traffic, not an incident. Any sender worth integrating with will redeliver after a timeout, and your 200 can be lost on the way back even when processing succeeded. So the receiver must be idempotent, and the interesting question is what you key on.

Key on the sender's event id. Nearly every serious webhook provider puts a unique id in the payload or a header for exactly this purpose. It is stable across redeliveries of the same event and different across distinct events, which is precisely the property you need.

Do not key on a hash of the body. It looks equivalent and it is not. Senders re-serialize payloads between attempts, embed a fresh timestamp or attempt counter, and change field ordering. Two hashes for one event means a duplicate slips through. Worse, in systems where the payload genuinely repeats, such as a heartbeat or a status that flaps to the same value twice, one hash for two events means a real event is dropped, which is failure mode seven above.

Do not key on business identifiers alone. An order id plus an event type feels natural and quietly collapses two legitimate changes to the same order into one. If you have no choice because the sender provides no event id, include the sender's event timestamp and a sequence field, and set the dedup window as narrow as correctness allows.

Mechanically, the insert is the lock. INSERT ... ON CONFLICT (source, event_id) DO NOTHING tells you in one round trip whether you are first. If zero rows were affected, the event is already in the inbox and you return 200 immediately. There is no check-then-act race, no distributed lock, and no advisory lock to leak.

Idempotency has to reach the side effect too, not just the inbox. If processing an event charges a card, calls a partner API, or sends an email, that downstream action needs its own idempotency key derived from the event id. Most payment and messaging APIs accept one; use it. The failure you are protecting against is a worker that crashes after the external call and before the local commit, and no amount of receiver-side deduplication helps with that.

Ordering you did not design is ordering you do not have

HTTP delivery over the public internet gives you no ordering. Two events emitted a millisecond apart can arrive in either order, and a retried event from four minutes ago can land after a fresher one. Teams discover this when a record flips back to a previous state and stays there.

Three strategies work, and the third is the one to reach for first.

Version guards. If the sender includes a monotonic version or updated-at value on the object, store it and reject any update whose version is not greater than what you already hold. This is a single comparison in the update statement and it makes out-of-order delivery harmless for state-carrying events.

Per-entity serialization. Route all events for one entity through a single ordered lane, keyed on the entity id, so processing for that object is sequential even when receipt was not. This is what partitioned brokers give you, and it is real work to build without one.

Re-fetch on receipt. Ignore the payload's state and read the object from the sender's API. Whatever order events arrived in, you converge on the current value. This is the hint-not-feed idea from earlier, and for most integrations it costs one API call and removes the entire ordering problem.

Retry behavior differs wildly, and it is on you to read it

Senders do not agree on what a failed delivery deserves. Some retry with exponential backoff for days. Some give you one short window and a manual redelivery button in their dashboard. Some expect an answer in a couple of seconds because a human is waiting on the other end of it. Some disable your endpoint after a failure streak and send an email you will not see for a week.

Read the sender's published policy before you write the consumer, because it decides how much you have to build. A sender with a multi-day retry schedule gives you room to be down for an hour. A sender with a ten-second timeout and no automatic retries means the receipt path has to be near-perfect and you need reconciliation from day one.

Sender behavior	What it gives you	What you must build
Long backoff, days of retries	Real tolerance for an outage on your side	Strict idempotency; late duplicates are guaranteed
Short window, manual redelivery	A recovery path, but only if a human drives it	Gap detection and a documented replay runbook
Fast synchronous expectation	Very little; a human is waiting	Sub-second ack, all work deferred, no exceptions
Auto-disable on failure streak	Protection for the sender, a cliff for you	Alert on delivery-failure rate before the streak ends
No retries at all	One attempt, take it or lose it	Reconciliation as the primary correctness mechanism

Verify the signature on the bytes that arrived

Signature verification is straightforward and the bugs are always in the same two places. First, verify against the exact bytes the sender transmitted, before any middleware parses the body. Frameworks that helpfully deserialize JSON and hand you an object have already destroyed the whitespace and key ordering the signature covers. In most web frameworks this means capturing the raw body in a hook that runs before the JSON parser, or configuring a raw body reader for that route specifically.

Second, compare in constant time. A naive string equality check on an HMAC leaks timing information; use the constant-time comparison your standard library provides. Then check the signed timestamp against your clock with a tolerance of a few minutes to blunt replay, and remember that a tolerance window makes your NTP configuration part of your integration's correctness.

Key rotation deserves a sentence of its own because it is the most common self-inflicted outage in this whole area. Support two active keys at once. Accept a delivery if it validates against either. Roll the new key out, wait a full delivery cycle, then retire the old one. A single-key rotation rejects every event in flight during the cutover, and those events are gone unless the sender retries past the window.

Reconciliation is the only thing that catches silent loss

Everything above reduces loss. None of it detects loss. If a sender never delivered an event, or delivered it during a two-minute window when your certificate had expired, no retry, no dedup, and no queue depth metric will ever tell you. Absence produces no signal.

So run a reconciliation job. On a schedule, list objects changed in the source system over a trailing window using the sender's API, compare them against your copy, and repair or alert on the differences. Start with an hourly sweep over the last twenty-four hours plus a nightly sweep over the last seven days, and tune from what it finds. The first run on an established integration almost always finds something, which is the point.

Where the sender emits sequence numbers or you can order event ids, gap detection is cheaper and faster. Watch for holes in the sequence per stream and alert on them. It will not catch an event the sender never generated, but it catches the delivered-and-lost class in minutes rather than at the next sweep.

Deploys are a delivery failure mode

A rolling restart terminates processes that are holding open requests. If the process exits on SIGTERM without finishing what it has, every in-flight delivery becomes a connection reset, and a deploy that takes four minutes across a fleet can produce four minutes of failures the sender may or may not retry.

Three things fix it, and they have to be done together. Handle SIGTERM by refusing new work and letting in-flight requests finish, with a drain timeout shorter than the platform's grace period. On Kubernetes the default termination grace period is thirty seconds, so a fifteen or twenty second drain leaves headroom. Add a preStop delay of a few seconds so the load balancer stops routing before the process starts refusing, because endpoint propagation and pod termination are not synchronized. And make sure the readiness probe fails immediately on shutdown while the liveness probe does not, so the pod leaves rotation without being killed outright.

The receipt path being thin is what makes this achievable. A handler that only writes a row finishes in milliseconds and drains cleanly. A handler that calls three external services cannot drain inside any reasonable grace period, which is another reason the two-job rule pays for itself.

What goes on the dashboard

Most webhook dashboards show request rate and error rate, both of which look fine while events are being lost. These are the signals that actually surface a problem, ranked by how early they catch it.

The silence alarm is worth building even though it is fiddly. For any sender with predictable volume, define a period in which zero events is abnormal, and page on it. That is the alert that catches an auto-disabled endpoint, an expired certificate, a subscription somebody edited, and a DNS change that pointed the hostname somewhere polite and wrong.

If you are the one sending webhooks

The obligations invert and get harder, because now other people's reliability is your support burden. Sign every payload with a shared secret per endpoint and publish exactly how to verify it, including which bytes are covered. Support two active signing keys so customers can rotate without an outage. Put a unique, stable event id in the payload and document that consumers should key idempotency on it.

Retry with exponential backoff and jitter, publish the schedule, and stop at a bound you have written down. Isolate endpoints from each other with per-endpoint concurrency limits and a circuit breaker, because one customer's ten-second endpoint should not create head-of-line blocking for everyone else on the same queue. Keep a delivery log with request, response, status, and timing, and expose it to the customer with a redelivery button. That single feature removes most of your webhook support tickets.

Two more that get skipped. Publish a stable set of egress IP addresses so customers can allowlist you, and give a heads-up before you change them. And treat the disable-after-failures policy carefully: notify aggressively and through more than one channel before you switch an endpoint off, because a silently disabled endpoint is a customer data loss event that you caused.

Mistakes we see most

• Business logic inside the request handler, so the sender's timeout is now your processing budget
• Returning 200 before anything durable happened, which converts every crash into silent loss
• Deduplication keyed on a hash of the body, which both admits duplicates and discards real events
• Signature verified after the framework parsed and re-serialized the body, then "fixed" by disabling verification
• A single signing key, making every rotation an outage with no recovery path
• Assuming ordering because events arrived in order during three weeks of testing
• A dead-letter queue with no alert, which is a place events go to be forgotten politely
• No reconciliation, so the only loss detector in the system is a customer email
• Deploys with no draining, producing a burst of dropped deliveries at every release
• Trusting payload contents as current state when a re-fetch would have cost one API call

A two-week hardening plan

Two weeks is enough because none of this is research. The inbox table is one migration. The handler change is usually under a hundred lines. Draining is configuration plus a signal handler. Reconciliation is a scheduled job that calls a list endpoint you already have credentials for. The reason it does not get done is that it never competes well against feature work until the first incident, and after the first incident it competes too well and gets built in a hurry by someone tired.

Before you turn one on

• The handler verifies the signature and writes one row, nothing else
• Raw request bytes are stored, not a parsed and re-serialized copy
• A unique index on sender plus event id, with insert-on-conflict-do-nothing
• Downstream side effects carry their own idempotency keys
• Two signing keys accepted, with a documented rotation procedure
• Body-size limits raised at every proxy and gateway in the path
• Draining verified by watching a real deploy under load
• A replay command that has been run at least once outside an incident
• Reconciliation scheduled, with its findings going somewhere a human reads
• Alerts on oldest-unprocessed age, signature failures, and silence per sender

Bottom line

Webhook reliability is not a hard problem, but it is an easy problem to get wrong in a way that stays invisible for months. Split receipt from processing. Make the 200 mean durability and nothing else. Deduplicate on the sender's event id. Assume no ordering and design for it explicitly. Then build the reconciliation sweep, because it is the only mechanism in the system that can detect the events you never got. Everything else on the list reduces loss; that one is the only thing that tells you the truth about it.

Frequently asked questions