Observability for a small engineering team

A small team does not have a data problem

It has an attention problem. Teams this size usually collect more telemetry than anyone reads, pay for all of it, and still spend forty minutes at 2am rebuilding the story from Slack scrollback. The gap is never volume. It is that nothing in the stack was built to answer a question a specific person asks under pressure. So we size the stack backward from those questions and cut whatever does not shorten the interval between a user noticing a problem and an engineer knowing which change caused it.

There are three questions worth building for. Is it broken right now, for whom, and how badly? What changed? Where is the time or the money going? Larger organizations add capacity forecasting, chargeback and a compliance evidence trail, and those are real needs at real scale. At eight engineers they are quarterly spreadsheet exercises, and treating them as live systems is how a team ends up maintaining a platform instead of a product.

Start with one wide event per unit of work

At the end of every request, job run, consumer message or scheduled task, emit one structured event carrying everything you know about that unit of work. Not a line when it starts, another when it hits the database, another when it calls the payment provider. One row, thirty to sixty fields.

What belongs on it: request ID and trace ID, normalized route, method, status code, total duration in milliseconds, database time and query count, cache hit or miss, each outbound dependency with its call count and elapsed time, the tenant identifier, the build SHA, region and instance, queue depth at entry, and the values of any feature flags evaluated. If a flag decided which code path ran, that flag belongs on the event, because "it only breaks for accounts with the new checkout flag on" is a five-second query when the flag is a field and a two-hour investigation when it is not.

This beats a stream of log lines because it turns debugging into filtering and grouping instead of reading. Ask for every event where status is 500, grouped by build SHA, and the errors line up against one deploy. Ask for the ninety-fifth percentile of duration for the tenant that complained, split by route. Neither question is answerable from prose logs without writing a parser first, and nobody writes a parser during an incident.

Keep debug logging, but make it cost nothing when it is off. The pattern that works is a per-request override: a header or internal flag that raises the log level for one trace, so verbose output exists when an engineer is reproducing something and not for the other 17 million requests that day. Static DEBUG in production is how a telemetry bill becomes an escalation.

The arithmetic is easy. A service holding a steady 200 requests per second emits 17.28 million events per day. At roughly a kilobyte per structured event that is about 17 GB per day, or half a terabyte a month, manageable in any pricing model. Turn on twenty debug lines per request and the same traffic produces over 300 GB a day. The decision that moves your bill twentyfold happens in the logging config, not in the vendor negotiation.

Cardinality is the whole bill

The most expensive misunderstanding in this space is that metrics and events are interchangeable. They are not, and the difference is cardinality.

A time series exists for every unique combination of label values. Take one counter with three labels: route, status class, instance. Forty normalized routes, six status classes and eight instances gives 40 × 6 × 8 = 1,920 active series, which is nothing. Add a customer_id label because somebody wanted per-customer error rates on a dashboard, with 4,000 customers, and the same counter can produce 7,680,000 series. At a few kilobytes of memory per active series in a Prometheus-style store, that one label edit is a capacity project.

The rule holds everywhere. High-cardinality identifiers belong on events and spans, never on metric labels. Metrics answer how bad, over what period, trending which way. Events answer who, which request, and why. A dashboard that needs per-customer error rate queries the event store instead of carrying a customer label on a counter.

The usual sources are unintentional. Raw URL paths with identifiers in them, so /orders/81723 becomes a label value forever. Error message strings, which explode the moment an upstream appends a timestamp. User agent strings. Pod names in an autoscaling group, which churn on every deploy and leave dead series behind. Normalize the route to /orders/{id} at the instrumentation layer, before the label is set, and most of it disappears.

Which signal answers which question

Before buying anything, be able to say what each signal is for and what it costs to keep.

Signal	Answers	Cardinality posture	What it costs
Wide request events	Who was affected, on which build, through which code path	Unbounded, that is the point	Volume scales with traffic; controlled by sampling and field discipline
Metrics (RED and USE)	How bad, trending which way, over weeks	Strictly bounded, low-cardinality labels only	Cheap per data point, ruinous if a label goes unbounded
Distributed traces	Where the latency actually went, across service boundaries	Bounded by sampling policy	Highest per-unit storage; requires context propagation work first
Error tracking	Which exception, which line, since which release, how many users	Grouped by fingerprint, naturally bounded	Low; the work is symbol and source-map upload in CI
Continuous profiling	Which function is burning CPU or allocating	Bounded by sample rate	Low overhead, high setup cost, rarely used during an incident
Synthetic probes	Is the product usable from outside your own network	Trivial	Near zero, and it is the only signal that survives your platform being down

Two well-known method acronyms stop dashboard sprawl. RED, from Tom Wilkie, covers anything that serves requests: rate, errors, duration. USE, from Brendan Gregg, covers anything with a finite resource: utilization, saturation, errors. Services get RED; databases, queues, connection pools and disks get USE. Google's four golden signals are the same idea from another angle. Apply one framing uniformly, so every dashboard has the same panels in the same positions and an engineer under pressure does not have to read the titles.

Traces: sample hard, keep what matters

Tracing is where small teams overspend first, because auto-instrumentation defaults to emitting a span for everything and sorting it out later. Sort it out first.

Head sampling decides keep-or-drop at the start of a trace, before anything interesting has happened, so a 1% head sample discards 99% of your errors. Tail sampling decides after the trace completes, which is what you want: keep every trace containing an error, keep every trace above a per-route latency threshold, and keep 1% to 2% of the rest as a baseline.

Know the operational catch before committing. Tail sampling requires every span of a trace to reach the same decision point, which in an OpenTelemetry Collector deployment means a load-balancing exporter keyed on trace ID in front of the sampling tier. Skip it and you get partial traces that look like data loss, because they are. It is half a day of configuration that people routinely discover three weeks after go-live.

Link the two directions. Exemplars attach a trace ID to a histogram bucket, so a spike on a latency graph becomes a click through to an actual slow request instead of the start of a manual search. It is usually off by default and it changes how people use a dashboard.

None of it works without context propagation, which is the part that is genuinely work. The W3C traceparent header has to survive every hop: load balancer, service call, background job enqueue, message consumer, retry, and the scheduled task that picks the row up three minutes later. Async boundaries are where propagation quietly dies.

SLOs sized for a team that also ships features

Most small teams either have no service level objectives or have twenty of them, and both produce the same result, which is that nobody looks. Start with one per user-facing service and add a second only when the first one has changed a decision.

Define the indicator as a ratio of events, not an average. Availability is good events over valid events. Latency is the fraction of requests under a threshold, so 99% of checkout requests under 400 milliseconds rather than "average latency under 400ms," because an average hides the tail users complain about. Write down what counts as a valid event, including whether load-balancer 502s with no application involvement count against you. Have that argument once, in daylight.

Then look at what the target buys in minutes, because the jump from three nines to four is a different engineering organization, not a different config value.

Four minutes and nineteen seconds a month does not survive one bad rolling deploy, a certificate rotation, or a managed database failover. Promising four nines without a redundant serving path promises something the arithmetic already refuses. Three nines is reachable for most product teams and leaves 43 minutes, roughly one bad deploy caught quickly.

The budget is only useful if it drives alerting. The multi-window multi-burn-rate approach from Google's SRE workbook is the standard and worth copying exactly: page at 14.4 times the sustainable burn rate over one hour, which consumes 2% of a 30-day budget; page at 6 times over six hours, which consumes 5%; open a ticket at 1 times over three days, which consumes 10%. Pair each long window with a short one at about a twelfth of its length so the alert clears promptly instead of hanging on after recovery.

Alerting: page on symptoms, ticket on causes

Alert design is where a small team keeps its people or loses them. The bar for a page is not that something is unusual. It is that a human is having a bad experience now, or will within the hour, and the person waking up can act on it.

Tier	Trigger	Route	Required attachments
Page	Fast error-budget burn, checkout or login failing, synthetic probe down from two regions	Phone, 24 hours	Runbook link, dashboard filtered to the failing dimension, named rollback target
Ticket	Slow budget burn, saturation trending to a limit within a week, certificate expiring in 14 days, disk at 80% and climbing	Queue, next business day	Owner, the measurement that would close it
Dashboard only	Everything informational: deploy markers, cache hit rate, queue depth under threshold	Nobody is notified	A panel someone actually opens

Every page carries four things or it is not ready to exist: the user-visible symptom in plain words, a link to the dashboard already filtered to the failing service and region, a runbook with the first three commands to run, and the name of the artifact to roll back. Assemble those when you write the alert, not at 2am with one hand.

Then enforce a deletion rule. Any alert that fired in the last ninety days without producing an action gets deleted or demoted to a ticket. The resistance this generates is informative on its own.

Alert patterns that produce nothing but fatigue

• Paging on CPU above 80% with no statement of what a user would notice.
• One notification channel carrying both pages and informational chatter, so the pages stop registering.
• Alerts defined by clicking through a vendor UI, so no change is reviewed and nobody can diff what shifted last quarter.
• Thresholds frozen at whatever the value happened to be on the afternoon the alert was written.
• Per-host alerts on an autoscaling group, which fire every time the group scales in normally.
• No inhibition rules, so one dependency outage pages six services simultaneously and the real signal is buried.
• Firing on a single failed scrape rather than a sustained for duration, which turns a network blip into a page.
• A page with no runbook, which converts every incident into an archaeology exercise through old Slack threads.

Retention, and cutting the bill at the collector

Tier retention against how people actually work. Incident response happens within hours, so keep events hot and instantly queryable for 7 to 14 days. Postmortem work happens within a couple of weeks, so keep a cheaper object-storage tier for 30 to 90 days. Capacity work happens across a year, so keep downsampled metrics for 13 months and nothing else. Very few organizations need full-fidelity spans older than a month.

Cut volume at the collector rather than at the source, since the source is application code and every change there needs a deploy. An OpenTelemetry Collector with filter and transform processors, or a pipeline tool like Vector or Fluent Bit, gives one reviewable place to drop, redact, aggregate and route. Health-check traffic goes first and it is larger than people expect: a liveness probe every 5 seconds across 8 instances produces 138,240 spans a day before a single user arrives. Multiply by the number of probes your orchestrator runs and health checks can outnumber real requests.

The other lever is field discipline. Every attribute on a wide event costs storage on every event, forever. Strip the ones nobody has queried in a quarter; most vendors will show attribute-level query frequency if you ask.

Self-host or buy, at this size

The scarce resource on a team of eight is engineer-hours, and an observability stack is a distributed system with its own storage, retention behavior, upgrade path and failure modes. Running Prometheus, Grafana, Loki and Tempo yourself is doable and people do it well. It also needs someone on call for it, and it is the system you least want unavailable during an incident, which is exactly when correlated failures make it most likely to be.

Our default split: run the collector yourself, because portability, redaction and cost control live there, and buy the storage and query layer until the bill approaches the loaded cost of the part-time engineer the alternative needs. That is the comparison worth doing, and it is rarely the one in the spreadsheet, which usually compares a vendor invoice against zero.

If you do self-host, put the stack in a different failure domain from the thing it monitors: separate account, separate region, separate credentials. A dashboard hosted inside the cluster that just went down is decoration. Keep one external synthetic probe on an independent notification path, so the answer to "is the site up" never depends on the system that is down.

OpenTelemetry is the portability decision

Instrument with OpenTelemetry SDKs, export OTLP to a collector you control, and configure the backend as one destination among several. The reason is not standards enthusiasm. It is that changing vendors becomes an exporter change instead of a re-instrumentation project across every service, and that you can dual-write to two backends during an evaluation and compare them on your own traffic rather than on a demo dataset.

Use the semantic conventions for attribute names, so http.request.method and server.address and db.system rather than whatever your first service called them. Shared names let a prebuilt dashboard work on day one and let a new engineer read a trace without a translation table. Tracing and metrics specifications have been stable for a while; logs arrived later and the tooling is still catching up, which is one more argument for spending your effort on wide events and spans.

One caution from doing this repeatedly: automatic instrumentation gets you most of the way and produces span volume nobody asked for. Budget a pass to disable the noisy packages, drop health-check and static-asset spans, and add the ten attributes specific to your domain. Tenant, plan tier, job type, feature flag, queue name. Those fields are the difference between generic telemetry and telemetry about your product.

On-call when there are five of you

Rotation arithmetic is unforgiving. Three people means one week in three, forever, which is how good engineers start reading recruiter email. Four is the practical floor, and below that the honest move is to narrow what pages rather than pretend the rotation is sustainable.

Two structures work at this size. A weekday-hours primary with a written after-hours bar, so the on-call person knows which two or three conditions justify a call at night and what waits. Or a ship-and-hold rotation, where whoever deployed is first responder for 48 hours, which puts the feedback where the change was made and improves deploy hygiene faster than any policy document.

Write a handoff note at the end of every rotation. Five lines: what fired, what is silenced and until when, what is still degraded, what changed in the alert set, what the next person should watch. Skip it and the incoming engineer rediscovers a known-degraded dependency at midnight.

The first thirty days

Thirty days works because every step is usable on its own. After week one you can answer which build started this. After week two you know before your customers do. After week three the pager means something. Nothing depends on a platform migration finishing, which is why these projects usually stall at 60% and stay there.

Own these regardless of which vendor you pick

• Instrumentation through OpenTelemetry SDKs, using semantic-convention attribute names
• The collector configuration in your repository, reviewed like application code
• Alert and SLO definitions as code, never clicked into a vendor console
• Dashboards exported to JSON and version controlled, with a restore that has been tested
• A runbook per page, carrying the first three commands and the rollback target
• Redaction and retention applied at the collector, before anything leaves your network
• An automated test proving trace context survives every async boundary
• The on-call handoff note, and a quarterly pass that deletes alerts nobody acted on

Bottom line

Observability for a small team is a subtraction problem. Every tool in this category ships defaults built for organizations with a platform group, and adopting them gives a team of eight a large bill, a noisy pager and no faster path from symptom to cause. Emit one rich event per unit of work. Keep identifiers off metric labels. Sample traces on outcome instead of on arrival. Run one SLO driving two burn-rate alerts, and delete anything that pages without an action attached. That is a stack four engineers can operate and a new hire can read in an afternoon.

Frequently asked questions