Background jobs and idempotency

The second run is not a bug, it is the contract

A support ticket says a customer was charged twice. Or two identical shipping labels print. Or the monthly usage rollup runs and every invoice comes out double. The engineer on call reads the worker logs, finds the job, sees it succeeded, then sees it succeed again eleven seconds later with the same payload. Nothing crashed. The system did what it was built to do, which was run the job until it was sure the job ran.

Duplicate execution is not a failure mode of your queue. It is the guarantee your queue offers. SQS standard queues, Redis-backed workers, Kafka consumers, Postgres job tables and workflow engines converge on one promise: a message will be processed at least once. Where a product advertises better, the better guarantee is scoped to a boundary that stops at the edge of your process.

So the goal is not preventing duplicates. It is making one cost nothing. That property has a name, idempotency, and it is not one technique but about eight, applied where the job's effects require them.

At-least-once is an acknowledgement problem, not a queue problem

The mechanism explains why no vendor can sell you out of it. A worker pulls a message, does the work, and acknowledges it so the broker can delete it. Those are two events separated by a network. If the process dies after the work and before the ack, or the connection resets before the broker records the ack, the broker holds a message it has no evidence was handled. Two options, both wrong some of the time: delete it and lose work, or redeliver it and do the work twice.

Every serious system picks redelivery, because losing an order is worse than sending two emails. The ambiguity is structural: nothing can distinguish "the job did not run" from "the job ran and the confirmation was lost."

Kafka's exactly-once semantics are real inside Kafka: an idempotent producer plus a transaction committing offsets and output records atomically to Kafka topics. The moment your consumer calls a payment API, that transaction has no reach. Temporal gives you durable retries and deterministic replay, and its activity execution is still at-least-once by design. Exactly-once delivery is unavailable. Exactly-once effect is available to anyone willing to build for it.

Where duplicates come from, in the order worth checking

Handed a system that produced a double effect, we walk this list top down.

Classify the side effect before you write the handler

Idempotency work goes wrong when it is applied uniformly. Some jobs are already safe. Some need a guard around the one line that matters. The classification takes two minutes and decides everything downstream.

Effect class	Example	What a duplicate does	How to make it safe
Absolute write	UPDATE user SET status='active', an upsert keyed by primary key, a rendered file written to a fixed object key	Nothing. Second run computes the same value and writes it again	Already safe. Do not add machinery
Relative write	balance = balance + 50, appending a row, incrementing a counter, pushing to a list	Doubles the number silently, and the error compounds	Unique constraint on the source event, or convert to an absolute write derived from a ledger
External create	Charging a card, creating a subscription, provisioning a tenant, buying a shipping label	Real money and a support ticket	The provider's idempotency key, plus your own claim record
Irreversible send	Email, SMS, push notification, an outbound webhook to a customer's system	Cannot be undone. The cost is trust, not data	Claim before send, mark after, plus a collapse window keyed to the recipient and event
Cross-system sequence	Charge, provision and email the receipt in one handler	A retry repeats the part that already completed	Separate jobs with their own keys, or guard each step

Two rules fall out of it. A job doing only absolute writes needs no machinery at all. And a handler mixing an external create with a relative write is the worst shape there is, because a retry is safe for one half and catastrophic for the other. Split it first.

The key is the hard part. The store is trivial.

Almost everyone gets the storage right and the key wrong. The key identifies the business event and must be identical on every attempt: across restarts, across a replay six days later, across a producer that sent twice.

Derive it from the domain, not the transport. Good keys look like invoice:inv_8841:capture, order:9f3a:fulfil, usage_rollup:acct_221:2026-06-14. Bad keys are the message ID, the delivery attempt ID, or a UUID generated inside the handler. The message ID changes when the producer resends. The attempt ID changes on every retry by definition, which is the exact case you are trying to protect against.

Hashing the payload is a trap. It breaks in two directions. Two attempts at the same event hash differently once the payload carries an enqueue timestamp, a trace ID, or a schema version. And a genuinely distinct second event with a byte-identical payload, a customer buying the same item twice in a minute, gets silently suppressed. Hash only after stripping every non-deterministic field on purpose.

Scope the key to the effect, not the job. A handler that charges a card and then sends a receipt has two effects with two lifetimes. Give them ...:charge and ...:receipt, so a retry after a successful charge and a failed email re-sends only the email.

The dedupe record, concretely

The store is a table with a unique index on the key, a state column, a lease expiry and somewhere to keep the result. The race is the interesting part: two workers can arrive at the same key in the same millisecond.

The claim is a single statement, not a read followed by a write. In Postgres, INSERT INTO idempotency(key, state, lease_until) VALUES ($1,'claimed',now()+interval '5 minutes') ON CONFLICT (key) DO NOTHING RETURNING id. A row back means you own the work. Nothing back means somebody else does, so read the existing row and branch on its state: succeeded returns the stored result, claimed with a live lease exits quietly, claimed with an expired lease means the previous owner died and you take over, failed falls through to a retry.

Storing the result matters more than it looks. When an endpoint accepts a key from a caller, the second identical request should return the response the first one produced, not a bare 409. That is what makes the caller's retry loop safe. Keep records long enough to cover your longest retry window and a manual replay; Stripe, for comparison, keeps a key's result for 24 hours.

Whenever you can, let the database be the guard

An application-level check is a read and a write with a gap in the middle, and the gap is where the duplicate lands. A unique constraint has no gap, and it holds under code paths you have not written yet.

For money or counts, the shape that works is an append-only ledger with a unique index on the originating event and a balance derived by summing rather than mutated in place. A double delivery attempts a second insert, the index rejects it, the handler treats the conflict as success, and the balance is right because nothing was incremented. That is a migration and a view, not an architecture.

The same idea covers state machines. A transition row with a unique constraint on (entity_id, from_state, to_state, event_id) makes a second attempt to move an order from paid to fulfilled a conflict, not a second fulfilment.

The enqueue is where duplicates are born

Here is the bug in most codebases that have otherwise thought about this carefully. A request handler writes a row and enqueues a job.

Commit the transaction, then publish. If the process dies in between, the row exists and the job never runs, so the order sits paid and unfulfilled with nothing to alert on. Publish first, then commit. If the transaction rolls back, the job runs against a row that does not exist and fills the dead-letter queue with a message about an order nobody can find. Neither ordering is correct, because they are two systems and one has no transaction.

The fix is the transactional outbox, and it is old, boring and correct. Insert the job into an outbox table in the same transaction as the business write. A relay reads unpublished rows, pushes them to the queue and marks them sent. If it crashes between those two it publishes again, which is fine, because the consumer is idempotent. An unsolvable two-system problem becomes an ordinary at-least-once one.

If throughput allows, skip the broker and run the queue in Postgres. SELECT ... FOR UPDATE SKIP LOCKED gives safe concurrent claiming, the enqueue joins the same transaction as the business write, and the outbox and the queue collapse into one table. It carries far more load than people expect, and the dual-write problem leaves with the broker.

Visibility timeout is a claim, not a lock

This is the highest-yield thing to check when duplicates appear suddenly in a system that was fine last month. When a worker receives a message the broker hides it for a fixed period and expects an ack inside it. SQS defaults to 30 seconds and allows up to 12 hours. Redis-based runtimes do the same under a different name, reclaiming work whose lock was not renewed.

If the job runs longer than that window the message reappears, and a second worker starts it while the first is still running. Not sequentially, concurrently: two workers, same payload, same row. Then the first acks a message the broker already redelivered.

The sizing rule: measure p99 duration in production, not the average, and set the timeout to three times it. Above a minute, extend the lease on a heartbeat from inside the handler and treat a failed heartbeat as a signal to stop working. Duration grows with the data and the timeout does not, so alert on the ratio between them.

Serialize on the entity, not on the queue

When two runs must not overlap, the instinct is to set worker concurrency to one. That fixes the race, destroys throughput, and is rarely necessary. The overlap you care about is per entity. Two jobs touching different orders can run together all day.

Three mechanisms, in the order we reach for them. A transaction-scoped advisory lock, pg_advisory_xact_lock(hashtext($entity_id)), which releases on commit or crash and needs no cleanup. A claim row with a lease, when you want to see who holds what. Or a FIFO queue with the message group set to the entity ID, which serializes per group and keeps parallelism across them.

What we avoid is a cache entry used as a mutex. A lock with an expiry in a non-consensus store depends on clock skew and on outliving the work, and both fail under the load that made you want it.

Retries help only if you classify the error first

An unclassified retry policy is a delay generator. A validation error, a 404 on a resource that will never exist, a permission denial: none improve on the fifth attempt. Retrying them burns the budget and turns a page that could have fired at 9:04 into one that fires at 9:47 with less context.

Split errors into terminal and transient where you raise them, and default to terminal so an unclassified error surfaces instead of looping. Transient means a timeout, a connection reset, a 429, a 503, or a deadlock, and those get exponential backoff with full jitter, because synchronized retries from a hundred workers are a denial of service against your own database.

Cap attempts per queue. Sidekiq's default of 25 attempts across roughly three weeks suits a job whose upstream might be down for a day and is far too patient for one a person is waiting on. Give the dead-letter queue an owner and a replay path through the production handler, because a replay script written at 11pm is how the duplicate arrives.

The external call, and the ambiguous timeout

The hardest case in this article is one line: you called a payment API and it timed out. The work may have happened. Retrying blind is how the double charge occurs, and giving up is how the order never ships.

Two mechanics resolve it. First, use the provider's idempotency support, offered by most serious APIs as a request header carrying a key you generate and reuse on every retry. Generate it before the first attempt and persist it with the job. A key generated inside the retry loop is not a key.

Second, when a provider offers no such support, look before you create. Query by your own reference, attached on the way in, and act on what you find. That read-then-write is racy in general, which is why it belongs inside your claim record, where one worker holds the window. Then reconcile against the provider daily.

Three more places duplicates hide

Sends you cannot take back. Email, SMS and outbound webhooks have no undo, and recording the send before or after it happens is wrong in one direction each: before, a crash means a notification nobody received; after, a crash means a second one. Choose per message type, then add a collapse window keyed on recipient, template and entity. Ten minutes covers nearly every duplicate-execution case, and routing sends through one module keeps the window enforced.

Schedules that fire on every replica. In-process cron on a service running three replicas fires three times, an autoscaler adding an instance at 02:59 gives you a fourth, and a rolling deploy across the top of the hour can produce a run from the old pods and the new ones. Give every scheduled execution a key naming the interval it covers, daily_rollup:2026-06-14, with a unique constraint, so three replicas racing produce one insert and two conflicts. Compute the window from a payload parameter rather than now(), because a run at 03:00 and a re-run at 07:40 must cover the same interval.

Payloads that outlive the code. A queue is a time machine: a message enqueued Friday afternoon may run Monday morning against code that shipped over the weekend, and SQS retains messages for four days by default. Enqueue identifiers and a version, {"v":2,"order_id":"9f3a"}, so the handler loads current state whenever it runs. A serialized order carries Friday's prices into Monday's execution, and when the class shape changes the backlog turns poison.

Test the second run, once, per handler

This is the cheapest reliability work in a backend codebase, and it is missing almost everywhere we look. One property: invoking the handler twice with the same payload leaves the system as it was after invoking it once.

In an integration test against a real database: run the handler, snapshot the affected tables, run it again with the same payload, assert the snapshot is unchanged, and assert against a fake gateway that the external create and the email each happened once. Then the harder variant, where the external call succeeds and the process dies before the ack. That one finds the real bugs, because it reproduces the interleaving production will produce on its own.

Write it once as a shared helper and every new job costs four lines to cover. When a team tells us their jobs are idempotent, this is what we ask, and the answer is usually that they believe it rather than test it.

What goes on the dashboard

Queue depth is the metric everyone graphs and the least useful of the set. A depth of 40,000 draining in ninety seconds is healthy. A depth of 12 that has not moved in an hour is an outage.

The suppression counter is the one nobody thinks to add. It counts how often the dedupe record prevented work. A low steady rate means the system is working. A climb means a producer started double-sending, or a visibility timeout is now too short for a handler that got slower. It turns silent corruption into a graph that moves before anybody files a ticket.

Patterns that guarantee a duplicate charge

• Using the message ID or the delivery attempt ID as the idempotency key.
• Generating the provider's idempotency key inside the retry loop instead of before the first attempt.
• Checking whether a record exists, then inserting it, with no unique constraint underneath.
• Committing the business transaction and publishing the job as two separate operations.
• In-process cron on a service that runs more than one replica.
• A dead-letter replay script that is not the production handler.

Before a job touches money or a customer's inbox

• Its effects are classified in the handler's docstring
• The idempotency key comes from the business event and is stable across attempts
• A unique constraint enforces the invariant in the schema, not only in code
• The enqueue happens in the same transaction as the write that justifies it
• Visibility timeout is at least three times the measured p99 duration
• Errors are classified terminal or transient, with terminal as the default
• A test runs the handler twice and asserts the state is unchanged
• The dead-letter queue has an owner, an alert and a replay path through production code

A two-week hardening pass

Two weeks is enough because none of it is research. The slow part is the inventory, because it forces somebody to read jobs nobody has opened in a year, and that reading is where the surprises live.

Bottom line

Background jobs fail in a small number of shapes, all downstream of one fact: the broker cannot tell whether your work happened. Accept that and make the second run boring. Classify what the job does to the world, key it on the business event, push the invariant into a unique constraint, write the enqueue inside the transaction, size the timeout against measurements, and test the second run. That is a week or two on an existing system, and it converts the incident that costs a weekend into a counter that ticks quietly on a dashboard.

Frequently asked questions