Workflow engines and when you need one

What you are actually buying

Strip away the marketing and a workflow engine sells exactly one thing: your program's position survives a crash. A normal process holds its state in the stack and the heap, so when the pod is evicted at step four of nine, everything about where it was is gone. A durable engine writes each completed step to storage, and when a new worker picks the execution up it replays the record and continues from step five. Everything else — the visual designer, the retry policy, the dashboard, the timers — is built on top of that one property, and if you do not need that property you are buying a lot of operational surface for a diagram.

The second thing you get, which people undervalue until they have it, is that the process becomes an object you can query. How many onboardings are stuck at the identity check right now, and for how long? Before, that answer lived in whichever engineer could write the right join across three tables. After, it is a list. For anything with a support team attached, that visibility often turns out to be worth more than the durability that justified the purchase.

What you give up is the subject of most of this article, because it is almost never in the evaluation. You give up writing ordinary code — the execution path has to be deterministic and replayable. You take on a versioning problem with no clean solution. And you add a stateful service to your operational load, with a database, an upgrade path and a failure mode your on-call rotation has never seen.

Three products, one word

Before evaluating anything, work out which of these three you are shopping for. Teams routinely trial a tool from the wrong category, conclude that workflow engines are heavy and awkward, and go back to the status column.

Data orchestrators. Airflow, Dagster, Prefect and their kin. The unit of work is a scheduled graph over datasets: run these transformations nightly, in this order, and tell me which one broke. They are built for a modest number of large, scheduled runs. Using one to drive per-customer business processes means one graph instance per customer, and they fall over at that shape, usually somewhere between a few thousand and a few tens of thousands of concurrent runs depending on how the scheduler is configured.

Durable execution engines. Temporal, Restate, DBOS, and the managed state-machine services from the major clouds. The unit of work is a long-running instance of a process, one per entity, and the design point is millions of them, mostly idle, mostly waiting. This is the category that fits per-order, per-customer, per-document processes. It is also the category with the determinism constraint.

Human-centric process engines. Camunda, Flowable and the BPMN family. The unit of work is a case that people act on: assignment, queues, forms, escalation, delegation, a visual model the operations team edits without a deploy. If the hard part of your problem is who approves what and what happens when they are on holiday, this category has solved things the other two have not, and reimplementing them in a durable-execution engine is a year of work.

Category	Unit of work	Fits when	Breaks when
Data orchestrator	A scheduled graph over datasets	Nightly and hourly batch, dozens to hundreds of runs	You need one instance per customer, or sub-second reaction
Durable execution	A long-lived instance per entity	Per-order or per-user processes, waits measured in days	The team wants to write ordinary non-deterministic code
Process / BPM engine	A case a person works	Approvals, assignment, escalation, operations-editable flows	High-throughput machine-to-machine work with no humans
A queue plus a state column	A row and a message	Under roughly five steps, no waits longer than an hour	Steps grow past a handful and transitions become implicit
A cron job over a table	A scan and an update	Genuinely simple, low volume, latency-tolerant	Two schedules overlap, or the scan cannot keep up

Rule out the cheaper answers first, honestly

Most systems that reach for an engine could have been fixed with less. The two cheaper designs are worth a genuine attempt, not a paragraph of dismissal, because both are things your team can operate on day one.

A state column plus a queue plus an idempotent worker. Each step reads the current state, does its work, and writes the next state in the same transaction that acknowledges the message. Crashes redeliver, the worker sees the state has already advanced, and it skips. This handles perhaps three to six steps cleanly. Beyond that the transition rules become implicit in the code and nobody can answer what happens from state seven, which is the actual failure — not a limitation of scale but of comprehension.

An explicit state machine in your own code. One table of transitions, one function per state, a scheduled sweeper for timeouts. A capable engineer builds this in a week and it is the right answer more often than the industry admits. It stops being the right answer at roughly the point where you want retries with per-step backoff, waits longer than a few hours, compensating actions on failure, and a way for support to see where things are — because at that point you are writing an engine with a smaller test suite than the one you could have adopted.

The four signals

The work outlives the process. A step waits on something you do not control — a partner API that takes six hours, a payment that settles overnight, a batch job that runs at 2am. Holding this in memory means every deploy loses work, and deploys happen weekly.

A human is in the middle. An approval, a review, a signature. Waits measured in days, with reminders, escalation and reassignment. This is where hand-rolled designs decay fastest, because the sad paths — the approver left the company, the request was superseded, two people acted at once — outnumber the happy path four to one.

Failure requires undoing. Step six fails and steps two through five must be reversed: release the inventory hold, void the authorization, cancel the shipment. Compensation logic scattered through exception handlers is unreviewable. An engine with an explicit saga or compensation model makes it something you can read.

You cannot answer where things are. If “how many are stuck and for how long” requires an engineer and a query, and it is asked more than once a week, the visibility alone can justify the adoption.

One signal is usually not enough. Two is a real conversation. Three or four and the hand-rolled version is going to be rebuilt as an engine anyway, just slower and by you.

The determinism constraint, which surprises everyone

Durable execution works by replay. When a worker resumes an execution it re-runs your workflow function from the top, feeding recorded results for steps that already completed, until it reaches the first step with no recorded result. For that to produce the same path, the function must be deterministic.

Which means: no random(), no reading the system clock, no direct network calls, no iterating a hash map whose order is not stable, no reading a config file, no uuid4(). Every one of those has a sanctioned replacement supplied by the engine, and every one of them is a habit your team has. Expect two to six weeks before the code review comments stop, and expect the first production replay bug to be an iteration-order difference that never appeared in any test.

The mental model that makes it click: the workflow function is not code that does things. It is code that decides what should be done next. All doing lives in activities, which run once, are recorded, and are allowed to be as messy and non-deterministic as reality requires. Teams that learn this early write thin workflows and fat activities and have a good time. Teams that learn it late have business logic inside the deterministic layer and rewrite it.

Versioning, which is the actual hard problem

You have 40,000 live executions. Eleven thousand are somewhere in the middle. You need to change step four. The new code will be replayed against histories produced by the old code, and if the sequence of steps no longer matches the recorded history, the replay is non-deterministic and the execution fails.

Every engine gives you a way out, and none of the ways out are pleasant. Version gates in the code — ask the engine which version this execution started under and branch — keep everything running but leave permanent forks in your source until the last old execution drains. Draining, where you stop starting new executions on the old definition and wait, is clean but takes as long as your longest-running instance, which for anything with a human step can be weeks. Deploying the new definition under a new name and routing new work to it is often the most honest choice, at the cost of two definitions in production.

Three practices make this bearable. Keep workflow definitions short, because a nine-step workflow has far fewer version-sensitive edit sites than a forty-step one. Push everything you expect to change into activities, where a change is just a deploy. And instrument how many live executions exist per definition version, so “can we delete the old branch yet” is a query and not a guess. Teams that skip the third one accumulate version gates for years.

Idempotency does not come for free

Activities are executed at least once. A worker can complete the side effect, die before recording the result, and have the activity re-run. So every activity that touches the outside world needs its own idempotency: a key derived from the execution id and the step, passed to the downstream system, with the downstream system honouring it.

The engine guarantees the workflow's progress, not your payment provider's. That distinction is the most common production surprise we see after adoption. A team adopts durable execution, correctly concludes that it makes the process resilient, and does not notice that resilience is expressed as re-running things. The first duplicate charge or duplicate email is the moment the distinction lands.

The operational cost, stated plainly

Self-hosting a durable execution engine means running a stateful cluster with a database behind it, and the database is the part that will page you. Retention policy matters: full execution histories accumulate quickly, and a workflow with a few hundred steps and large payloads can run to megabytes of history each. Our default is to keep payloads out of the history — pass identifiers, store blobs in object storage — which sounds fussy until the first time a history exceeds the engine's size limit mid-execution and the instance cannot proceed.

Budget honestly. One to two weeks for a first working workflow. Two to six weeks before the determinism mistakes stop appearing in review. A quarter before the team's instinct for what belongs in a workflow versus an activity is reliable. Managed offerings remove the cluster but not the determinism constraint, not the versioning problem and not the learning curve, which are the expensive parts. If the pitch you heard was that adoption is a library import, the pitch was about the first afternoon.

Agent systems change the calculation

A model-driven agent that plans, calls tools, waits on results and loops is structurally a long-running workflow: minutes to hours of wall time, many external calls, each of which can fail, and a partial result that is expensive to throw away. Durable execution is a genuinely good fit, and it is why the pattern has moved from back-office processing into AI infrastructure over the past two years.

There is a real friction, though, and it is worth understanding before you commit. Model calls are non-deterministic, which is exactly what replay cannot tolerate. The resolution is straightforward once stated: every model call is an activity, so its output is recorded on first execution and replayed thereafter. The workflow never calls the model directly; it calls an activity that calls the model. Then the loop that decides whether to continue is deterministic given recorded outputs, and replay works.

Two consequences follow. Histories get large, because model outputs are text and there can be many of them — keep them in object storage and record references. And a step-level retry re-invokes the model, so an activity that failed after a successful generation and before recording will generate again, at full cost. Cache on a hash of the resolved prompt if that cost matters, which at scale it does.

The other thing durable execution buys agent systems is the human interrupt. Pausing an execution for a person to approve a tool call, for as long as that takes, and resuming exactly where it stopped, is the same primitive as an approval workflow. Teams building this by hand end up with a job table, a poller and a partial reimplementation of timers. That is the strongest argument in this whole article, and it is the newest.

When it makes things worse

Adopting an engine and then routing everything through it. Two-step processes become workflows because the pattern is now available, and a straightforward request path acquires two extra hops, a serialization boundary and a new failure mode. The rule we use: if it completes in one process, in under a second, and a retry from the caller is acceptable, it does not belong in an engine.

The other failure is organisational. The engine becomes the architecture, business logic migrates into workflow definitions, and now the hardest-to-change layer in the system holds the most-frequently-changing rules. Keep pricing, eligibility and policy in ordinary services the workflow calls. The workflow should read like a table of contents.

The mistakes we get called in to fix

• A data orchestrator running one graph per customer, with a scheduler at its knees
• Business logic inside the deterministic layer, so every rule change is a versioning exercise
• Activities with no idempotency key, discovered via a duplicate charge
• Large payloads passed through the workflow, until a history hits the size limit mid-run
• No telemetry on live executions per version, so version gates are never removed
• Timers used as the only escalation path, with nobody watching what timed out
• Every operation wrapped in a workflow, including ones that finish in 40 milliseconds
• An engine chosen for the visual designer, then driven entirely from code

A four-week evaluation that produces a real answer

Step five is the one nobody does and the one that decides whether you will still like this tool in a year. Change a workflow definition while a thousand instances are mid-flight and see what the migration actually costs. If that rehearsal is uncomfortable in a sandbox, it will be worse with real customers waiting.

Before you commit

• You can name at least two of the four signals in your own system
• The state-machine version was attempted and its specific limits are written down
• The category is chosen before any vendor is
• Every activity that touches the world has an idempotency key
• Payloads are references; blobs live in object storage
• A versioning strategy exists and has been rehearsed against in-flight executions
• Live-execution counts per definition version are on a dashboard
• Business rules live in services, not in workflow definitions
• Somebody owns the cluster, the database and the retention policy
• The team has killed a worker mid-execution on purpose and watched the replay

Bottom line

Buy a workflow engine when the work outlives the process, when a person sits in the middle of it, when failure means undoing, or when you cannot answer where things are stuck. Buy from the right category, which is usually durable execution for per-entity processes, a data orchestrator for scheduled batch, and a process engine when approvals are the hard part. Then plan for the two costs nobody quotes: the determinism constraint that changes how your team writes code, and the versioning problem that never fully goes away. Both are survivable. Neither is a surprise you want in month three.

Frequently asked questions