Streaming architecture in practice: when real-time earns its complexity

What a streaming architecture actually buys

Streaming buys one thing a batch system cannot: a short interval between an event happening and something acting on it. Scale, decoupling, and replay get cited as reasons to stream, and all three are available from batch designs too, usually for less money and with fewer parts to operate. So the opening question on a streaming design is never which broker. It is how small that interval has to be, who is waiting on it, and what specifically goes wrong when it is larger.

That question has a testable answer, and the answer is usually written down somewhere already. If a person reads a dashboard once each morning, the requirement is hours and a scheduled job is the correct engineering. If an authorization hold has to be placed before a payment settles, the requirement is a few hundred milliseconds and no batch design meets it. Most real requirements land between those poles, and the middle is exactly where teams overbuild.

Our engineers build both shapes across federal, state, and commercial work, and the split in our own delivery runs closer to batch than most teams expect. Streaming is right often enough that we keep it as a core skill, and rare enough that reaching for it by default is a reliable way to spend a program's budget on operations instead of outcomes.

The cadence test, run before any tool selection

Name the consumer of the output. Name the decision that consumer makes. Then name the cost of that decision being made on data that is five minutes stale, then an hour stale, then a day stale. Three numbers, one page. If the cost curve is flat between five minutes and an hour, streaming is not the requirement and a short-interval scheduled job will serve the same decision at a fraction of the operating load.

The curve bends sharply in a few identifiable places. It bends when a human or machine action is irreversible once taken, which is why payment holds, safety interlocks, and dispatch routing stream. It bends when the data has a short useful life, as with equipment vibration signatures that only matter before a bearing fails. It bends when a downstream index has to agree with a source system continuously, which is the change-data-capture case. Outside those, the bend is usually imagined.

Three shapes, and what each one costs to own

The choice is not binary. Micro-batch sits between the poles and covers more requirements than either neighbor, at meaningfully lower operating cost than continuous streaming.

Property	Scheduled batch	Micro-batch (1–15 min)	Continuous streaming
End-to-end interval	Hours to a day	Single-digit minutes	Milliseconds to seconds
Cost when idle	Near zero; compute bills only while the job runs	Low; short bursts on a schedule	Continuous; brokers and consumers bill around the clock
Fixing a logic bug	Re-run the job for the affected window	Re-run the affected intervals	Replay from the retained log, if retention was long enough
State handling	Stateless by construction; each run reads the source	Mostly stateless with a high-water mark	Distributed keyed state that must be checkpointed and bounded
Failure blast radius	One late report	One late interval	Growing consumer lag that compounds until drained
Team skill required	SQL and a scheduler	SQL, a scheduler, watermark discipline	Partitioning, delivery semantics, state backends, on-call

The five decisions that are expensive to reverse

Everything in a streaming system is cheap to change except five things. Getting these right at design time is most of the value an experienced team brings, because each one is embedded in every producer and consumer written afterward.

• The partition key. It fixes both ordering and parallelism. Ordering in Apache Kafka is guaranteed within a partition, never across them, so the key decides which events are ordered relative to each other. Changing it later re-shuffles history and breaks every stateful consumer keyed on the old scheme.
• The event schema and its compatibility mode. Backward, forward, or full compatibility in the schema registry governs which future changes are legal. A schema published without required-field discipline will be produced against by a dozen services within a quarter.
• Delivery semantics. At-most-once, at-least-once, or effectively-once. This choice propagates into every sink, because idempotency has to be implemented at the write, not promised by the framework.
• Whether the log is the system of record. If the log is authoritative, retention becomes a data-durability question rather than a buffer-sizing question, and the compliance posture changes with it.
• Event time versus processing time. The semantics of every window, every aggregate, and every late-arrival rule follow from this. Restating a year of aggregates because the original design windowed on arrival time is a project, not a patch.

Delivery semantics, stated plainly

Exactly-once is the most oversold phrase in this field. What Kafka and Apache Flink implement is effectively-once processing: an idempotent producer plus a transactional protocol that makes the visible result equivalent to a single application of each record, within the boundary of the systems participating in the transaction. Kafka has supported idempotent producers and transactions since the 0.11 release in 2017, and Flink pairs its checkpoint barriers with a two-phase commit on the sink.

The boundary is the part that gets skipped. If the sink is an external database, a search index, or a REST endpoint that does not participate in the transaction, the guarantee stops at the edge of the framework. The practical rule our team applies: build for at-least-once delivery and make every write idempotent by natural key, then add transactional sinks only where duplicate application is genuinely unrecoverable. That design survives replay, survives operator restarts, and survives the day someone reprocesses a topic by hand.

Event time, watermarks, and the late data problem

Events arrive out of order. A mobile device buffers offline and uploads an hour of readings at once. A partition lags and its records land after a peer's. Any system that aggregates over time has to decide when a window is complete, and a watermark is the mechanism: an assertion that no event older than timestamp T will still arrive.

That assertion is always wrong sometimes. The design question is what happens when it is. Flink lets a window keep an allowed-lateness period after the watermark passes, and route anything later than that to a side output. Teams that skip the side output are silently dropping data, and they usually find out during an audit rather than on a dashboard. Every windowed pipeline we build emits a late-record counter and a watermark-skew gauge, because a watermark drifting away from wall-clock time is the earliest warning that a source has stalled.

What good means numerically

Streaming systems fail quietly, so the acceptance criteria have to be numbers rather than adjectives. Six that we hold ourselves to on delivery:

Consumer lag measured in seconds, not records. Record counts are meaningless across topics with different rates. Lag in time is comparable, alertable, and understandable by a program manager. A healthy pipeline holds p99 lag under a stated ceiling, commonly 30 to 60 seconds for operational displays.

End-to-end latency measured by a canary. Inject a synthetic heartbeat event at the producer, timestamp it at the sink, and publish the difference. Broker-side metrics measure the broker. Only a canary measures what the customer experiences.

Sustained headroom of 40 percent or better. Run steady-state at no more than 60 percent of measured per-partition capacity. The remaining margin is not waste, it is recovery budget.

A stated drain time after an outage. This is arithmetic, not a guess. If arrivals continue at rate λ during an outage of duration T, the backlog is λT. If the pipeline can process only 1.2λ once it recovers, the surplus draining that backlog is 0.2λ, so drain time is five times the outage. A one-hour outage becomes a five-hour recovery. Sizing for 2λ turns the same outage into a one-hour recovery.

Checkpoint duration below checkpoint interval. When a stateful job's checkpoint takes longer than the gap between checkpoints, the job is already in trouble and the next failure will be expensive.

A bounded and observed dead-letter queue. Depth, age of oldest message, and a named owner. A dead-letter queue nobody reads is a data-loss mechanism with extra steps.

The failure modes teams actually hit

Key skew. One tenant, one device, or one null-valued key routes a disproportionate share of traffic to a single partition. Parallelism stops helping because Kafka consumers in a group cannot exceed the partition count, so extra consumers sit idle while the hot partition falls behind. The fix is a composite or salted key, chosen before launch.

Unbounded state. A keyed aggregation with no time-to-live grows until the state backend exhausts disk. This surfaces months after go-live, which is why state TTL belongs in the first version rather than the hardening sprint.

Rebalance storms. A consumer that pauses longer than max.poll.interval.ms, which defaults to 300000 milliseconds, gets evicted from the group. Its partitions redistribute, the replacement inherits the same slow work, and the group thrashes. Long per-record processing belongs behind an async boundary, not inside the poll loop.

Poison records with no exit. One malformed message at the head of a partition blocks everything behind it forever. Deserialization failures need a route to the dead-letter queue on the first attempt, with the raw bytes preserved for diagnosis.

Replay that double-counts. The team reprocesses a topic to fix a bug and the downstream totals double, because the sink was append-only rather than keyed upsert. This is the idempotency rule failing in production instead of design review.

Monitoring the wrong layer. Broker CPU can be flat while lag climbs steadily. Alert on lag, watermark skew, checkpoint failures, and dead-letter depth. Infrastructure metrics are for diagnosis after the alert, not for the alert itself.

Where the money goes

Streaming cost is dominated by capacity that runs whether or not data flows. An Amazon Kinesis Data Streams shard in provisioned mode bills per shard-hour continuously, so a stream sized for peak carries that cost overnight and on weekends alike. A production Kafka cluster typically runs three brokers with replication factor 3 and min.insync.replicas set to 2, which means three copies of every byte and three sets of instance hours before a single consumer is deployed.

The second line item surprises people: cross-availability-zone network transfer. Replication across zones is what makes the cluster survive a zone failure, and on the major clouds inter-zone traffic is billed. Rack-aware replica placement plus fetch-from-closest-replica reduces the read side of that bill. Compression on the producer, zstd or lz4, reduces both network and storage at a modest CPU cost and is close to free money on text-heavy payloads.

Retention is the third lever, and the one that moves both cost and capability the most. Kafka's default retention is 168 hours. Kinesis Data Streams defaults to 24 hours and extends to 365 days at additional cost. Retention is what makes replay possible, so cutting it to save storage quietly removes the ability to reprocess a bug. Tiered storage, which moves older log segments to object storage, is the way to keep long retention without keeping it on broker disk.

Change data capture is the case that usually justifies itself

Of all the reasons teams adopt streaming, keeping a secondary store in agreement with a primary database is the one that most often holds up. Reading the write-ahead log directly, through a MySQL binlog reader, PostgreSQL logical replication, or Oracle LogMiner, avoids the two classic sins of polling: the load a repeated full-table scan puts on a transactional system, and the deletes that a high-water-mark query can never see.

The design still has to be earned. Snapshot-then-stream transitions are where implementations break, since the initial consistent snapshot and the ongoing log tail have to hand off without gaps or duplicates. Schema changes on the source propagate into the stream and need a compatibility policy waiting for them. Our engineers treat the initial snapshot as a first-class part of the build rather than an afterthought, because a change-data-capture pipeline that cannot re-snapshot cleanly cannot recover from its own worst day.

When a simpler method is the right answer

An honest list, because the strongest recommendation we give some customers is to not build the thing they asked us to price.

The output has a human cadence. A daily leadership report, a weekly reconciliation, a monthly regulatory extract. A scheduled job with an incremental high-water mark does this correctly, cheaply, and can be operated by a team without streaming on-call.

The source is a vendor API with a rate limit. No streaming architecture makes a partner endpoint answer faster than the contract allows. The interval is set upstream and the broker adds only cost.

Volume is modest and the window is long. A few million rows a day into a warehouse is a scheduled load, not a stream. Modern warehouses ingest that in minutes, and incremental models on a schedule cover the requirement with tooling the team already runs.

Correctness matters more than freshness. Financial close, actuarial runs, and anything a reviewer will trace line by line benefit from a batch's clean boundaries. A deterministic re-run over a fixed window is easier to defend to an auditor than a replayed stream.

There is also an honorable middle: incremental micro-batch every one to five minutes. It covers a surprising share of what gets specified as real-time, costs a fraction of continuous streaming to operate, and stays legible to whoever inherits it. When a requirement reads "near real-time" without a number attached, micro-batch is usually what the requester meant.

A build sequence that keeps options open

• Write the interval requirement as a number with a named consumer before selecting any tool.
• Fix the partition key and the event schema first, and review both with the teams who will produce into them.
• Make every sink write idempotent by natural key on day one, not after the first duplicate incident.
• Set state TTL, allowed lateness, and a late-record side output in the first version of every windowed job.
• Ship the canary heartbeat and the lag-in-seconds alert before the first production producer connects.
• Load-test at two times projected peak and record the measured drain time after a simulated outage.
• Set retention from the governing policy floor, then design tiered storage to hold it affordably.
• Document the replay procedure and rehearse it once before go-live, with the runbook the on-call engineer will actually open.

Common objections

Bottom line

Streaming is a legitimate and sometimes mandatory shape, and the teams that succeed with it treat it as a distributed-systems commitment rather than a faster pipeline. The commitment is real: partition design, delivery semantics, state discipline, replay rehearsal, and an on-call rotation that understands lag. Where the requirement earns that commitment, the result is worth it. Where it does not, an incremental job on a schedule delivers the same decision to the same person at a small fraction of the cost, and stays running after the people who built it move on.

Frequently asked questions