The dashboard was never the problem, and it was never the answer
Every team that ships a model builds the same wall of charts. Requests per minute, p95 latency, mean predicted score, a drift number, a confusion matrix from last week's labels. The wall is useful. It is also a set of precomputed answers to questions written down before the system met its users, and the failures that hurt are the ones nobody wrote down. A support lead says approvals feel wrong for one partner, and forwards a screenshot. You open the dashboard, everything is green, and now you have to go find out why.
That gap has a precise definition, worth being pedantic about because the tooling market blurs it. Monitoring watches known quantities against known thresholds. Observability is the property of a system that lets you answer questions you did not instrument for in advance. A dashboard is monitoring. Filtering on a field you have never filtered on, over a population you have never grouped by, and getting an answer in seconds is observability. One is a display layer. The other is a data model.
So buying a monitoring product does not give you observability, and neither does adding fifteen more charts. What you can ask later is decided by what you write down at inference time and whether it joins to what happened afterward. Get the record right and a mediocre query tool is enough.
You are probably here because
- A customer tells you the model got worse for them, every chart is green, and you cannot confirm it or rule it out.
- The first question on the incident call is whether it started when you shipped last week, and nobody can answer that from the data you have.
- You are fairly sure the damage is concentrated in one account, one region or one language, and the overall accuracy number will never show it.
- Investigating a prediction from last month means rerunning a pipeline that has changed since, so you never actually see what the model saw.
These are one problem wearing four faces: the row that would answer them was never written at serve time, or it does not join to what happened next. The next two sections — the prediction record and the outcome join — are where that gets fixed.
The unit of ML observability is one prediction, fully described
Web observability converged on the wide event: one structured record per unit of work carrying every attribute that might matter, instead of counters and log lines reassembled by timestamp. ML needs the same shape, and the unit of work is one inference. A single row, written at serve time, describing the prediction well enough for a stranger to reconstruct it a month later.
Logging the inputs and the output feels sufficient. That row cannot answer a real question, because a real question is comparative: was this prediction unusual for this segment, under this model version, against the same segment last month? Comparison needs dimensions, and dimensions have to be on the row.
| Field group | What goes in it | The question it unlocks |
|---|---|---|
| Identity | prediction_id, request_id, trace_id, timestamp in UTC, tenant or account id | Joining this row to the trace, the outcome, and the customer ticket that started the investigation |
| Version | model version, feature pipeline version, prompt or template version, config hash, deployed image digest | "Did this start when we shipped something?" — the first question in most incidents and the one most often unanswerable |
| Inputs | the feature vector as served, or a reference to it, plus null and default flags per feature | Training-serving skew, silent nulls, a default value quietly standing in for a broken upstream |
| Output | raw score, calibrated score, decided action, threshold applied, any override or business rule that fired | Separating a model change from a threshold change from a rules change, which look identical downstream |
| Context | channel, geography, device, product surface, experiment arm, cohort | Slicing. Nearly every real regression lives in a slice and is invisible in the mean |
| Cost and timing | per-stage latency, retries, cache hit or miss, token counts, compute unit consumed | Whether quality moved because a fallback path started firing under load |
| Uncertainty | confidence, entropy or margin, out-of-distribution score if you compute one | Finding the population the model was already unsure about before anyone complained |
Two of those groups are worth more than the rest combined. The version bundle answers "what changed," and an incident review without it burns days on hypotheses while the answer sits in a deployment log nobody joined to the predictions. The input flags are the other: a boolean per feature saying whether the value was present, imputed, defaulted or stale. Upstream pipelines fail quietly far more often than models fail loudly, and a default that was rare and is now 40 percent of traffic is the most common root cause we find in systems the dashboard called healthy.
The join everyone defers, and it is the whole system
Predictions are cheap to log. Outcomes are the hard half, and skipping them is the most consequential shortcut in this discipline. Without an outcome you can measure distributions, latency and volume, but not whether the model is right, and every claim about quality stays a proxy argument.
The outcome record is a separate table with its own arrival time. It keys on prediction_id, carries the observed result, and is deliberately late. A fraud label arrives when a chargeback posts, typically weeks after the transaction. A churn label arrives at the end of a billing period. A lead-quality label arrives when a sales cycle closes, possibly two quarters out.
Design for the delay instead of pretending it away. Write the prediction row immediately, write outcome rows as they arrive rather than mutating it, and compute quality as of a maturity horizon: accuracy for predictions made at least N days ago, where N is where the arrival curve flattens. Skip that and the metric always looks better than reality, because fast-arriving labels differ systematically from slow ones.
Plot the label arrival curve before you pick a monitoring cadence
Take a month of predictions, plot the cumulative share of labels received against days since the prediction, and read the shape. If 80 percent land within four days, a weekly accuracy report is honest. If the curve is still climbing at day 45, that report is fiction: you need proxy signals for the near term and a maturity-horizon report for the truth. It takes an afternoon.
While labels mature, proxy signals carry the near-term read, and they are usually already in the product. Did the user accept the suggestion, edit it, or discard it? Did they retry or abandon? Did a downstream rule override the model? How often does a reviewer disagree on the queue you already route to review? None is ground truth, and all move within hours, which makes them the right instrument for the first 48 hours.
Instrumentation Priority — Where We Wire First
Our default build order, weighted to 100. Drift comes last on purpose: teams reach for it first, and alone it answers least.
Drift is a symptom, and a weak one
Distribution drift gets disproportionate attention because it is computable without labels. Population Stability Index, Kolmogorov–Smirnov, chi-square and Jensen–Shannon distance each reduce a two-sample comparison to one chartable number. The credit-risk convention treats a PSI under 0.1 as stable, 0.1 to 0.25 as worth a look, and above 0.25 as material. Reasonable as a start, with one limitation: input drift is neither necessary nor sufficient for a quality problem. A campaign can shift your input mix hard while the model performs identically on both populations, and the relationship between features and outcome can invert with no movement in either marginal distribution.
Treat a drift alarm as a prompt to investigate, never as a finding. Its value is as a pointer into a population, which is why the wide record matters more: given a signal, you filter to the rows that drove it. The predictable failure is computing a statistic over 60 features, daily, on high-volume traffic. Significance scales with sample size, so at millions of rows an immaterial shift is significant by construction. Alert on effect size, require persistence across windows, and route drift to a queue, not a pager.
A prediction is a pipeline now, so trace it
Ten years ago inference was one function call. Today a single answer passes through query rewriting, embedding, vector retrieval, a reranker, prompt assembly, a model call, a tool call or two, output parsing, a validator and a fallback. Nine places to fail, and the aggregate latency chart says only that the total moved.
The fix is standard distributed tracing applied to inference. One trace per request, one span per stage, attributes carrying what matters for that stage: retrieved document identifiers and scores, reranker cut-off, prompt version, token counts, tool name and status, retry count, cache hit, whether a fallback fired. Now the p95 chart decomposes. Retrieval flat, generation flat, the reranker went from 40 milliseconds to 900 because an index rebuilt and its cache is cold. Five minutes with traces, days without.
OpenTelemetry is the right substrate here, for the same reason it is elsewhere: it decouples what you emit from who stores it, and vendor migrations in this category are common. The GenAI semantic conventions for model calls are real and still moving, so pin the version you emit and treat attribute renames as a query-compatibility concern. The discipline that matters is not which names you use. It is that the trace and the prediction record share an identifier, so you can pivot between them without a heuristic timestamp join.
Cardinality is the bill, and sampling is the lever
The first serious ML observability invoice surprises people. The arithmetic is volume times bytes per record times retention, plus what the engine charges for indexing high-cardinality fields. A service handling 5,000 predictions per second, writing a 2 KB record, produces on the order of 850 GB per day before compression. Keep the raw feature vector on every row for 90 days and you have built a data warehouse by accident.
Three levers control it, in order. Tier by stream rather than deleting uniformly, since metrics are tiny and should live for years while traces are the largest thing you emit. Sample the expensive streams, with a bias. And push feature vectors into columnar files in object storage keyed by prediction_id, so the hot store carries dimensions while the bulky payload sits somewhere cheap.
Default Retention by Stream (Days)
A starting policy, not a law. The long lines exist so year-over-year comparison is possible. Set them before launch; retention cannot be applied retroactively.
The bias is what people get wrong. Uniform sampling at one percent discards precisely the rows an investigation needs, because interesting events are rare by definition. Sample uniformly for the baseline, then keep everything unusual: every error, timeout, fallback, low-confidence prediction, every request from a tenant in an incident, every row in a slice small enough that one percent leaves nothing. Tail-based sampling in a collector does this cleanly, since the decision happens after the request completes. Record the retained rate on the row so aggregate math weights correctly.
Slice first, aggregate second
Aggregate metrics hide the failures that matter, structurally rather than occasionally. Overall accuracy is a traffic-weighted average, so a model that collapses on 3 percent of requests moves the headline by a fraction of a point. If those 3 percent are your enterprise accounts, the headline is worse than useless: it is actively reassuring while the problem grows.
Define slices deliberately and monitor them as first-class objects. The productive dimensions are tenant or account tier, geography and language, input complexity, channel and device, time of day, model version, and cohort by account age. Track per-slice volume, quality and latency, and rank slices by size times degradation so attention lands where the damage is largest. Simpson's paradox is the default outcome here: overall accuracy can rise while every segment falls, purely because traffic mix moved toward an easy segment.
Where the records should live
| Store | Good for | Where it hurts |
|---|---|---|
| Time-series database Prometheus and compatible engines | Counters, rates, latency histograms, SLO burn, cheap long retention | Cardinality. Account id or prediction id as a label takes the server down, and the failure is sudden rather than gradual |
| Columnar analytical store ClickHouse, BigQuery, Snowflake and peers | Wide prediction records, arbitrary group-by, high-cardinality filters, joins to outcomes | Not free. Needs partitioning and TTL discipline or the bill grows with no change in usage |
| Object storage plus open table format Parquet on S3 or equivalent, with Iceberg or Delta | Feature payloads, raw inputs, long-horizon reprocessing, training-set reconstruction | Query latency in seconds to minutes. Fine for investigation, wrong for interactive use |
| Tracing backend OpenTelemetry-compatible | Per-stage decomposition, tail latency, dependency and fallback behavior | Volume and cost. Needs a sampling policy on day one, not after the first invoice |
Most teams end up with all four, and that combination is fine. The decision that matters is not the vendor. It is that prediction_id is a real key with identical meaning in every store, so an investigation can start in a chart, filter to a slice, join the outcomes and open a trace without a reconciliation script. Where that key is ambiguous, every investigation opens with an hour of data archaeology, and people stop investigating.
Send it over and we will tell you what we would change.
Email the field list you write for each prediction — the CREATE TABLE, a sample JSON row, or the log line your serving code emits — along with how outcomes get joined back to it, to contact@precisionfederal.com. You get back a short written note naming the three things we would change and why. One business day. No charge, no meeting, no deck.
contact@precisionfederal.comWhat changes when the model is a language model
Everything above holds, and three things get harder. There is usually no label, so proxy signals become the primary quality instrument: edit distance between generated text and what the user shipped, regeneration rate, copy events, escalation to a human, task completion. Output is open-ended, so quality needs a scored rubric rather than a comparison to a correct string, which makes the grader an instrument needing calibration against human judgment.
The pipeline is longer, so per-stage attribution matters more, and cost becomes a quality signal. A retrieval-augmented answer can be wrong because retrieval missed the document, the reranker buried it, the context window truncated it, or generation ignored it, and those are indistinguishable from the final text alone. Record retrieved document identifiers, scores, ranks and whether each survived into the final prompt, and they separate cleanly. Put token counts per stage, cache hit rates, the model tier chosen under dynamic routing, and retries on the same row. A quality regression coinciding with a cost drop is usually a routing change. A cost spike with flat quality usually means retries firing on a path nobody watches.
How ML observability projects fail
- Logging inputs and outputs with no version fields. "What changed" becomes unanswerable, and it is the first question asked.
- Mutating the prediction row when the label arrives. You lose the arrival curve, and any honest read on today's numbers.
- Alerting on drift with no written response. An alert nobody can act on trains people to ignore the channel that later carries a real one.
- High-cardinality identifiers as time-series labels. This does not degrade gracefully. It takes the metrics backend down.
- Uniform sampling. Rare events are what you need and what uniform sampling throws away.
- Deferring the outcome join to phase two. Phase two does not arrive.
Privacy and audit obligations shape the schema
The prediction record is often the most sensitive table a company holds, because it carries the features describing a person next to the decision made about them. Design for that in the schema, not in access control. Hash or tokenize direct identifiers, keeping the mapping elsewhere. Store feature payloads keyed by prediction_id in a partitioned layout that supports deletion by subject, because GDPR erasure requests will arrive and a hand-rolled scan across a year of Parquet is not a workable answer. Set retention per stream in code so the policy is inspectable during a SOC 2 or ISO 27001 audit rather than living in a runbook. The same schema shortens that audit: an assessor asking how you know the model behaves consistently across customer segments wants what a per-slice quality table already holds.
The four questions the stack has to answer without new code
A useful acceptance test a team can run today. Pick a real prediction from last month and answer these four without shipping instrumentation. Anything needing a code change is a gap.
One: what did the system see, and what did it do? Feature vector as served, model version, raw and calibrated scores, threshold, any rule that overrode it. If features must be reconstructed by rerunning a pipeline, the answer is no: the pipeline has changed since.
Two: was this normal for its population? The same slice over preceding weeks, with volume, score distribution and quality. Requires slices as objects, not ad hoc SQL written mid-incident.
Three: what changed, and when? A timeline of model, pipeline, prompt and config versions overlaid on the metric that moved. Requires deployment events in the same store as predictions.
Four: what happened afterward? The outcome and its arrival time. Requires the outcome table and the join.
Observability Maturity — Score Your Own Stack
Most teams shipping models sit at Level 1 or 2. The jump from 2 to 3 is a schema change, and it holds nearly all the debugging value.
A thirty-day build that gets you to Level 3
Build Sequence
Thirty days is realistic because none of it is research. The schema is a design conversation, emitting the record is a serving change measured in hundreds of lines, the outcome table is a pipeline, spans are a library. Step three takes longest, because it forces a conversation with whoever owns the system where outcomes live, and that conversation has usually been postponed for a year. Start it on day one.
Own these regardless of what you buy
- The prediction record schema, versioned alongside the serving code
- prediction_id as a real key, identical across every store you use
- The outcome table and the label arrival curve, refreshed quarterly
- The slice definitions, reviewed when the customer mix changes
- The sampling policy, written down, with the retained rate recorded on each row
- Retention per stream set in configuration, not in a runbook
- A raw export of prediction and outcome records, tested once before you need it
- The four-question test, run after every significant change
Common objections we hear
Our model is small and low-traffic. Is this overkill?
Lower traffic makes it cheaper, not less necessary. At low volume the storage argument disappears and you can keep everything, which is the easy version of this problem. What does not change is the outcome join and the version fields, schema decisions that cost nothing now and hurt to retrofit. Skip the tracing backend and the sampling policy at that scale, not the record.
We bought a monitoring platform. Does that cover it?
Partly, and the boundary is worth testing before you rely on it. Send a synthetic incident through and answer the four questions inside the vendor's interface. Most platforms are strong on drift views and weaker on joins to your outcome data. Ask what raw export looks like, because the prediction history is the asset and you want it queryable without them.
Can we reconstruct features later instead of logging them?
Only with point-in-time correct storage and an unchanged pipeline, and in practice the pipeline has changed. Reconstruction yields the value computed today rather than the value served, and that gap is the bug class you are hunting. Log what was served, and if the payload is large keep it in object storage keyed by prediction_id.
Bottom line
Dashboards are worth having and they are not observability. Observability in a machine learning system is a data model: one wide record per prediction carrying versions and dimensions, an outcome table joined by a stable key, spans across each stage of inference, sampling biased toward the unusual, and slices defined before anyone needs them. Build that and most investigations end in an afternoon of querying. Skip it and every investigation opens with the discovery that the evidence was discarded at request time.
The version of this work we are most often brought in for is not greenfield. It is a system already in production, with charts that look fine and a quality complaint nobody can trace, where the fix is a schema change on the serving path and a join deferred at launch. Scoped work with a clear finish line, worth doing before the next incident rather than during it.
Frequently asked questions
Monitoring watches quantities you chose in advance against thresholds you set in advance. Observability is answering a question you did not anticipate, such as why one customer segment degraded after a release. Monitoring is a display layer over aggregates; observability is a data model that keeps per-prediction records with enough dimensions to slice arbitrarily after the fact.
A prediction identifier and trace identifier, the timestamp, the model and feature pipeline versions, feature values as served with null and default flags, raw and calibrated output, the threshold and any rule that overrode it, business dimensions for slicing, per-stage latency and cost, and a confidence measure. Version fields and input health flags carry the most diagnostic weight.
Plot the label arrival curve first, then report accuracy only for predictions old enough that most labels have landed. For the near term use proxy signals already in the product: edits, overrides, retries, escalations, abandonment, reviewer disagreement. Not ground truth, but they move within hours and correlate well enough to drive an investigation.
It scales with prediction volume times record size times retention. Keep counters and aggregates for years since they are small, keep dimension-only prediction rows for about a year, push bulky payloads to object storage on a shorter clock, and sample traces hard while keeping errors whole. Set retention per stream before launch, because it cannot be applied retroactively.
Same foundation, three additions. There is rarely a label, so proxy signals become primary. Quality needs a scored rubric and a grader calibrated against human judgment. And the pipeline has more stages, so retrieved document identifiers, ranks, prompt versions and token costs belong on the record.