The pull request is small. The commitment is not.
A queue is one of the few architectural changes a team can ship in an afternoon and then live with for years. The library is mature, the diagram gets cleaner, and the slow endpoint returns in 40 milliseconds instead of four seconds. Six weeks later somebody asks why a customer's export from Tuesday never arrived, and answering takes three engineers most of a day. The queue was not the mistake. Shipping it without the operational surface a queue requires was the mistake, and those two pieces of work are almost always shipped separately.
We review a lot of systems where a queue was added as a performance fix, and in most of them it is neither the problem nor the solution. What follows is the decision we run first, the arithmetic that settles the argument, and the list of things that have to ship in the same pull request as the queue.
You are probably here because
- Someone wants to put a queue in front of a four-second endpoint, and nobody has profiled it yet.
- The queue shipped, and now a customer’s export from Tuesday is missing and no one can say where it stopped.
- The same job ran twice and charged or emailed somebody twice, and it only happens under load.
- There is a queue depth on a dashboard and nobody can say whether 40,000 is normal or an outage.
The symptom table below sorts the first of these from the rest, and the rest usually share one root cause: the queue shipped without the idempotency key, the age alarm, the dead-letter path and the replay command, which are listed under Before it goes to production.
What a queue actually does
A queue does one thing. It decouples the rate at which work arrives from the rate at which work gets done. Everything else attributed to a queue is either a consequence of that decoupling or a property of one specific broker, and keeping those apart makes the conversation shorter.
Four consequences are worth having.
Burst absorption. Traffic arrives in spikes and capacity is sized closer to the average. The queue holds the spike while workers catch up. This is where a queue genuinely rescues you, and it works only if the burst ends.
Durability across process death. Work in an in-process thread pool dies with the process: a deploy, an out-of-memory kill, a reclaimed spot instance. Work in a durable queue survives all three. If losing that work is unacceptable and your design holds it in memory, you have a real reason.
Fan-out. One event, several independent consumers, each with its own retry behavior and its own failure. With direct calls the producer knows about every consumer, and each new consumer is a change to the producer and a new way for it to fail.
Serialization per key. A queue with per-key ordering is a cheap mutex over an entity: every update to account 4471 handled by one consumer, in order, with no distributed lock service.
And one thing a queue never does: add capacity. If work arrives faster than workers finish it, the queue hides that for a while, then hands it back as a backlog with a long tail of stale results and a customer who noticed before your monitoring did.
The arithmetic, and why "we will add a queue" is not a capacity plan
Two formulas settle most of this argument.
Little's Law states that the average number of items in a system equals the arrival rate multiplied by the average time each item spends there: L equals lambda times W. It holds for any stable system and assumes nothing about the distribution of arrivals. Rearranged, it is the most useful sentence in capacity planning: if you know your arrival rate and your target wait, you know the queue depth you are allowed to have, and a depth above that number means your wait is already worse than the dashboard implies.
The second is utilization: arrival rate divided by the throughput of the worker pool. For a single-server queue with random arrivals, average time in the system is the service time multiplied by one over one minus utilization. That factor is what surprises people, and it is worth having the numbers in front of you before the design review.
Wait Time as a Multiple of Service Time, by Utilization
Single-server queue with random arrivals. Real systems with batched or bursty arrivals are worse than this, not better.
Two things follow. A system at 90 percent utilization is not almost full, it is already ten times slower than an idle one, and the next five points of traffic double that again. That is why targeting high utilization produces latency graphs with a cliff rather than a slope.
The second matters more here. All of it assumes utilization below one. Above one there is no steady state at all. The queue grows for as long as the overload lasts and then drains at the difference between service rate and arrival rate, which is why a five-minute overload routinely produces a fifty-minute backlog. Adding a queue in front of an undersized worker pool does not change the overload. It converts a fast, loud failure into a slow, quiet one.
Six signals that justify a queue
These are the reasons we accept. Score each from 0 to 10 on your own system, multiply by the weight, and total it. Adjust the weights before you score anything, not after, for the same reason you write acceptance criteria before running the test.
Signals That Justify Moving Work Off the Request Path
Weights sum to 100. Score 0 to 10 per signal. Above 55, add the queue. Below 35, fix the thing that is actually slow.
Notice what is not on the list. That the endpoint is slow is not a signal. That the team already knows Kafka is not a signal. That it is more scalable is not a signal either, because scalable is not a property, it is a claim about a rate somebody has to write down. If the only true statement is that a request takes four seconds, the next step is a profile.
What is actually wrong: the symptom table
Most requests for a queue arrive as a symptom rather than a design. This is the mapping we use to get from one to the other.
| Symptom | What is usually actually wrong | Does a queue help |
|---|---|---|
| This endpoint takes four seconds | An N+1 query, a missing index, or a synchronous third-party call in the path | No. Profile it. The work is unchanged after the queue |
| We lose work every time we deploy | Background work held in an in-process thread pool | Yes. Durability is exactly what a queue provides |
| A vendor API rate-limits us | Arrival rate above a service rate you cannot raise | Yes. The textbook case, with a token bucket at the consumer |
| Traffic is 20x at 9am and idle overnight | Capacity sized for peak, or for average and failing at peak | Yes, if the peak is bounded and the trough is long enough to drain |
| Two services need the same event | Producer coupled to every consumer and its failures | Yes, fan-out. An outbox table plus a poller also works |
| The database falls over under load | Unbounded concurrency against a fixed connection pool | No. Unbounded consumers deliver the same load. Bound the concurrency |
What gets worse the day the queue ships
Every one of these is survivable, all of them cost engineering time nobody budgeted, and they arrive together.
Read-your-writes breaks. The user clicks Save, the write goes on the queue, the page reloads and shows the old value. You now owe the product three things: an optimistic update in the interface, a status the user can poll or subscribe to, and an answer for what the interface shows when the job fails 90 seconds later and the user has navigated away. Teams ship the first and skip the other two, and the result is an interface that lies confidently.
At-least-once is the default, everywhere. SQS, Cloud Tasks, Pub/Sub, Kafka across a consumer restart, RabbitMQ after a channel drop: every one can deliver the same message twice, and at volume every one will. Where a broker offers exactly-once it means a transactional read-process-write inside that one system, and it does not survive a call to a payment provider. Every consumer is idempotent or it is a bug waiting on a network partition. Idempotency is not a broker setting. It is a unique constraint, an upsert, and a decision about what counts as the same request.
Ordering is narrower than people assume. Kafka orders within a partition. SQS FIFO orders within a message group. RabbitMQ orders within a queue, and only with one consumer. Add a second consumer to go faster and you have chosen throughput over order, and code that assumed created-before-updated starts losing races. Pick the key that needs ordering, make it the partition key, and accept that ordering caps parallelism at the number of distinct keys.
The backlog is invisible. Before the queue, a slow dependency produced errors and a graph anyone would notice. After, it produces a queue depth of 200,000 that nobody has an alarm on. Depth alone is the wrong metric anyway: 5,000 is fine at 2,000 messages a second and a disaster at three a second. Alarm on the age of the oldest unprocessed message, which is already in the unit the business cares about.
One bad message can take the fleet. A message that reliably crashes the handler gets redelivered forever, and on a partitioned log it blocks everything behind it. Without a maximum receive count and a dead-letter queue, one malformed payload consumes a worker fleet's capacity. With a dead-letter queue and no drain procedure, it consumes an afternoon four months later when somebody opens it and finds 9,000 messages.
Debugging gets expensive. A stack trace used to contain the whole story: the request, the code path, the failure. Now it is spread across a producer log, a broker, a consumer log on another host, and possibly a dead-letter queue. Without a correlation identifier attached at the producer and carried through every hop, nobody can answer what happened to one customer's export without a search across time ranges and a lot of guessing.
Writing to the database and publishing to the queue are two writes
Commit the row, then publish the message, and if the process dies in between, the row exists and nothing downstream knows. Publish first and the message can reference a row that never committed. Neither ordering is safe. The window is small enough that it never appears in testing and wide enough to happen weekly at volume. The fix is the transactional outbox: insert the event into an outbox table in the same transaction as the business write, then have a separate process read that table, publish, and mark rows sent. You get at-least-once delivery with the database as the arbiter, which is the same guarantee the broker was giving you, and one fewer place for the truth to split.
Backpressure is the part that gets skipped
An unbounded queue is a memory leak with a nice interface. Every queue needs a policy for what happens when it is full, and never-full is an assumption about traffic that will be wrong on the day it matters. Three answers are usable. Reject at the producer with a 429 and a Retry-After header, which is honest and lets the caller decide. Shed low-value work while protecting high-value work, which requires that somebody classified the work in advance. Or block the producer, which pushes pressure upstream, and is reasonable inside your own system and rude at a public interface.
The related mistake is unbounded consumer concurrency. Autoscaling a worker fleet on queue depth feels obviously correct, and it is how teams take down their own database. The queue absorbed the burst, the workers hand it to the database at once, the connection pool is exhausted, and the outage you avoided at the front door arrives at the back. Bound worker concurrency to a number the slowest dependency survives, and treat it as design rather than a tuning knob raised during an incident.
Send it over and we will tell you what we would change.
Email the endpoint you want to move off the request path and the handler that would do the work — or, if the queue already shipped, the consumer and a week of depth and age graphs — to contact@precisionfederal.com. You get back a short written note naming the three things we would change and why. One business day. No charge, no meeting, no deck.
contact@precisionfederal.comThe cheaper fixes to rule out first
Most queues we are asked to review were installed to fix something a queue does not fix. Rule these out in order. Each is an afternoon, and none adds a broker to your on-call rotation.
Get a profile before you get an architecture. A four-second endpoint is usually 380 queries, one query with a sequential scan, or a synchronous call to a vendor having a bad day. Instrument the path and look at the top ten calls by total time. It is the highest-yield hour in the process and it gets skipped constantly.
Move the work to a database-backed job table. In Postgres, SELECT ... FOR UPDATE SKIP LOCKED has been available since 9.5 and turns an ordinary table into a work queue that is transactional with your business data. The job insert and the business write are one transaction, which removes the dual-write problem rather than papering over it. Every SQL tool you own can inspect it, and retry logic is a column. Under a few thousand jobs a minute this is the right answer more often than teams expect.
Batch. If the reason you want a queue is that a request makes 500 individual calls to a downstream service, the fix is one call carrying 500 items.
Cache the read, and never queue one. A queue on a read path is a decision to show the user something stale while calling it fresh. If the read is expensive, cache or precompute it.
Stream the response. Chunked transfer or server-sent events keep the connection honest for work that takes ten seconds. Progress is visible and the failure stays where it happened.
Every one of these keeps failure synchronous and observable. That is a feature you give up when you add the queue, and it is worth naming out loud at the point where you give it up.
Picking the technology, once you have decided
| Option | Delivery and ordering | Fits when | What bites |
|---|---|---|---|
| Postgres job table FOR UPDATE SKIP LOCKED | At-least-once; ordering is whatever you index | You already run Postgres and volume is modest | Poll interval adds latency; churn needs vacuum attention |
| SQS or Cloud Tasks | At-least-once; FIFO variants order per group | You want a managed queue with no cluster to run | A visibility timeout shorter than the handler duration silently duplicates work |
| Redis Streams | At-least-once with consumer groups and claim | Redis is in the stack and latency matters | Durability is only as good as your persistence and failover setup |
| RabbitMQ | At-least-once; order per queue with one consumer | You need routing rules, priorities or per-message TTL | Unbounded queues under memory pressure; flow control in a backlog |
| Kafka or Kinesis | At-least-once, ordered per partition, replayable | A replayable log with independent consumers | Partitions cap parallelism; rebalances duplicate work; it is a system to operate |
The default answer for a small team is the database table, and it stays right longer than most architecture discussions allow. Move to a managed queue when the work has to leave the database transaction anyway, or when polling latency becomes the constraint. Move to a log when you need to replay history, when consumers belong to teams you do not control, or when one relational instance will not carry the write rate.
One operational detail causes more incidents than anything in that table. The visibility timeout or acknowledgement deadline must exceed the p99 duration of the handler, checked again after any change to it, or the broker hands the same message to a second worker while the first is still working. Size limits are the other trap: SQS caps a message at 256 KB, so put a pointer in the message and the payload in object storage.
Before it goes to production
These belong in the same pull request as the queue. Anything deferred gets built later, under time pressure, by whoever is on call.
- An idempotency key on every consumer, enforced by a unique constraint in the database
- A dead-letter queue, a maximum receive count, and a written procedure for draining it
- An alarm on the age of the oldest unprocessed message, in business units
- A visibility timeout or ack deadline longer than the p99 handler duration
- Bounded consumer concurrency, sized to the slowest downstream dependency
- A correlation identifier attached at the producer and logged at every hop
- A replay command that reprocesses a time range, tested at production data volume
- A producer-side backpressure policy: reject, shed or block, chosen deliberately
- One dashboard showing depth, age, throughput and error rate on a single screen
- A drain drill: stop the consumers for ten minutes under production-shaped load and watch recovery
Where the Hours Go on a Queue That Reaches Production
Our planning split. Publishing and consuming a message is roughly a day of it.
The message passing itself is a day. Everything else is why a queue is a larger commitment than the diff suggests. If a plan has no room for the bottom four rows, the queue ships without them and the first incident pays for them at a worse rate.
Where these projects go wrong
- Adding a queue to fix a slow query. The query is still slow and now nobody can see it from the outside.
- Treating exactly-once as a broker feature. It is a property of your consumer and your database, and only for writes you control.
- Alarming on depth instead of age. Depth without throughput is a number with no meaning attached.
- Autoscaling consumers on depth with no upper bound. The burst you absorbed gets delivered to the database all at once.
- Queueing a read. Then adding a poll loop, a spinner and a cache to hide the latency the queue introduced.
- Shipping with no replay path. Every consumer bug is then answered with a script written under pressure at 11pm.
- Publishing after the commit and calling it good enough. A fraction of a percent of events vanish, and reconciliation becomes somebody's permanent job.
How to take a queue back out
Removing a queue is harder than adding one and it is occasionally the right call, usually when the original reason turned out to be a slow query. The path that works: make the consumer callable as a plain function with no broker in the signature, call it synchronously behind a flag for a small share of traffic and compare results and latency, then move callers over in batches. Leave the queue running and empty for a full retention period so anything in flight drains. Then delete the producer, the consumer entry point, and the alarms, in that order.
The step teams skip is the last one. A deleted queue with live alarms trains the on-call rotation to ignore alarms, and that habit outlives the queue by years.
A two-week way to decide
Decision Sequence
Two weeks is enough because every question here is measurable inside it. Whether the endpoint is slow because of the database has an answer in a profile. Whether arrival rate exceeds service rate has an answer in two counters. Whether a job table carries your volume has an answer from a load test. Everything else is opinion.
The order matters more than the duration. Profiling first means that when the answer turns out to be a missing index, you find it before there is a broker in the diagram, a consumer service in the deployment pipeline, and a runbook page somebody maintains for four years.
Bottom line
A queue is the right tool when arrival rate and service rate are genuinely different and you can name why: a bursty producer, a rate-limited downstream, work that has to survive a restart, several consumers for one event, or a per-entity ordering requirement. It is the wrong tool for a slow query, an unindexed table, a chatty API call, or a database that cannot absorb the load. In all four the work is identical after the change and only the visibility is worse.
If you do add one, build the idempotency, the dead-letter path, the age alarm, the replay command and the backpressure policy in the same pull request. They will not get built later. The queue works fine for six weeks either way, which is what makes this decision easy to get wrong.
Frequently asked questions
When work has to survive a process restart, when a downstream dependency has a rate you cannot control, when arrival is bursty and the trough is long enough to drain, when several consumers need the same event, or when updates to one entity must be serialized. If the only reason is that a request is slow, profile it first.
It makes the request return faster and the work finish later. Throughput is set by the worker pool, not the queue. If work arrives faster than workers complete it, the queue grows for as long as the overload lasts and drains at the difference between the two rates.
A job table lives inside your database, so enqueueing is part of the same transaction as the business write and the dual-write problem disappears. A broker is a separate system with its own durability and failure modes, and it wins when you need fan-out to consumers you do not control, a replayable log, or throughput one relational instance will not carry. Under a few thousand jobs a minute, the table is usually the better call.
Assume at-least-once delivery and make the consumer idempotent. Derive a stable key from the work itself, enforce it with a unique constraint, and write the result as an upsert so a repeat is a no-op. Also set the visibility timeout above the p99 handler duration, since a timeout shorter than the handler produces duplicates on every slow message.
The age of the oldest unprocessed message comes first, because it is already in the unit the business cares about. Then throughput in and out on one graph so divergence is visible, consumer error rate, dead-letter arrivals as a page rather than a chart, and p99 handler duration against the visibility timeout.