Feature stores, and whether you need one

The two problems it exists to solve

A feature store is not a database for features. It is a specific answer to two failures that are hard to see and expensive to find late. The first is training-serving skew: the same feature computed twice, in two languages, by two people, and the two computations disagree in a way nobody notices until the model performs worse in production than it did offline. The second is point-in-time correctness: assembling a training set by joining features as they are now onto labels from six months ago, which quietly hands the model information from the future. If you have both problems, a feature store is a reasonable purchase. If you have neither, you are about to add a distributed system to enforce a naming convention.

Both failures share a signature that makes them worth naming precisely: the offline numbers look fine. Nothing errors. Validation accuracy is strong, the notebook runs, review passes. The damage appears weeks later as a model that underperforms its own evaluation, and by then the trail is cold and three other things have changed. That delay is why the tooling conversation gets emotional, and why it is worth separating the two problems and dealing with each on its own terms.

Point-in-time correctness, concretely

Suppose you are predicting whether an account will cancel in the next thirty days, and one feature is support_tickets_last_90d. You have labels for accounts as of the first of March. You build the training set by joining today's value of that feature onto each March label. The model learns that a high ticket count predicts cancellation, and it learns it far too well, because many of those tickets were filed during the cancellation, after the label date. In production the feature is computed before the cancellation, so the signal it saw in training is not there. Accuracy drops and the residual looks like drift.

The correct join is by entity and timestamp together: for each row, the value of the feature as it was known at the label time. This is an as-of join, and the awkward part is that it needs the history of every feature, not just its current value. Warehouses that overwrite a customer row on every update have thrown that history away, which is why the fix is often an ingestion change rather than a query change.

Two further details cause most of the remaining leakage. Availability lag: a feature computed by a nightly job that finishes at 04:00 was not available at 23:00 the previous evening, so a strictly correct join uses the value the serving system would actually have had, not the value the batch eventually produced. Late-arriving data: records backfilled after the fact change history retroactively, so a training set built in June and rebuilt in August will differ. Both are ordinary and both should be decided explicitly.

Training-serving skew, and the test that catches it

The classic version is two implementations of one definition. Training features come from a warehouse query written in SQL; serving features come from application code written in Python or Java, reimplemented from the same description. They agree on the happy path and disagree on the edges, and the edges are where models fail. In our experience the recurring culprits are few and specific: null handling, where SQL and application code make opposite default choices; time zones, where one path uses local time and the other runs in UTC; window boundaries, inclusive on one side and exclusive on the other; unit and rounding differences; and category vocabularies that drift after one path is updated and the other is not.

There is a test that finds all of these and it does not require a platform. Log the exact feature vector the serving path used, keyed by request id. On a schedule, recompute those same features from the offline definition for the same entities at the same timestamps, and compare. Report the share of values that differ beyond a tolerance, per feature. Alert when it crosses a threshold.

Run it nightly over a sample — ten thousand requests is plenty — and treat any feature with more than a fraction of a percent mismatch as a defect. Teams that add this find real bugs in the first week roughly as often as not, and it is a few days of work. If you build only one thing from this article, build this.

What is actually inside the product

Feature stores bundle four components, and most teams need one or two rather than all four. Buying the bundle to get the one is a common and expensive mistake, and separating them makes the decision much easier.

Component	What it does	What it replaces	Do you need it?
Transformation layer	Defines features once, in one place, with lineage	Scattered SQL and notebook code	Often already handled by your warehouse transformation tool
Offline store	Historical values with as-of join semantics	Hand-written point-in-time joins	Yes if you build training sets from time-varying data
Online store	Low-latency lookup of current values at serving time	A key-value store you would otherwise run	Only if you serve in real time under a tight latency budget
Registry and metadata	Discovery, ownership, versions, freshness, consumers	A spreadsheet and institutional memory	Value scales with the number of teams sharing features

Read that table as a checklist rather than a package. A team doing batch scoring with a single model needs the offline store's semantics and nothing else, and it can get them from a dated table and a careful join. A team serving three models at 20ms with forty features per request needs the online store and will build one whether or not it is called a feature store. A company with six teams reusing features across models gets most of its value from the registry, which is the component nobody gets excited about and the one that pays back at scale.

The cheaper design that covers most teams

For a single team with a handful of models and batch or near-real-time scoring, this arrangement gets most of the benefit for a fraction of the operational cost, and it is worth trying before buying anything.

One definition, in one language, in version control. Every feature is defined once as a transformation in whatever tool already owns your warehouse models. No feature has a second implementation anywhere.

Materialise history, not just current state. Write to an append-only table keyed by entity and effective timestamp. This is the change that makes point-in-time joins possible at all, and it is usually the only structural change required.

One shared as-of join helper. A single reviewed function that takes an entity-and-timestamp frame and returns features as known at that time, with the availability lag applied. Nobody writes their own; that is where leakage enters.

Serve from a materialised snapshot. Publish the latest row per entity to a key-value store on a schedule. The serving path reads values, it does not compute them. This is an online store; it is simply one you already know how to operate.

The nightly skew test described above, treated as a build-breaking check rather than a report.

That is a few weeks of work for one engineer and it covers the two real problems. It does not give you discovery across teams, governed sharing, or streaming aggregations with sub-second freshness. Those are the things you buy a platform for, and they are worth naming out loud so the decision is made on what is missing rather than on what is fashionable.

Online serving: latency and cost

If you do need real-time serving, the online path is where the engineering is, and it is mostly about the shape of the reads rather than the store you pick.

Budget the whole request, then subtract. If the product allows 100ms end to end and the model needs 25ms, network and serialisation take some, and what remains for feature retrieval is often 20 to 40ms. That number, not a vendor benchmark, decides the design.

Fan-out is the usual killer. Forty features fetched as forty round trips is forty times the tail latency risk, and at p99 the slowest of forty calls dominates. Group features by entity into a small number of multi-key reads — ideally one per entity type — and pack them so a single lookup returns everything for that entity.

Freshness is a per-feature decision, not a global one. Account tenure can be a day stale with no consequence. A session counter cannot. Set a time-to-live per feature group and, more importantly, decide what happens when a value is missing or expired: a documented default, a fallback to a coarser aggregate, or a refusal to score. Undefined behaviour here is how a partial outage becomes a silent accuracy collapse.

Cost tracks writes far more than reads. Refreshing ten million entities hourly is ten million writes an hour whether or not anyone reads them. Materialising only the entities that are actually served, or refreshing on access with a fallback, frequently cuts the bill by an order of magnitude. Work out the write volume before choosing a store; it is the number that determines the invoice.

What still breaks after you adopt one

A feature store solves two problems and inherits several others. Knowing them in advance keeps the adoption from being sold as a fix for things it does not fix.

Upstream schema changes. A source column renamed by another team breaks the feature regardless of where it is defined. You still need contracts and tests at the ingestion boundary.

Backfill correctness. Adding a feature and backfilling its history is the single most error-prone operation in this stack, because the backfill must reproduce what would have been known at each historical moment. Most leakage we find post-adoption is introduced by a backfill, not by a live pipeline.

Ownership. A shared feature with no owner rots. Every feature needs a named owner, a stated freshness expectation and a list of consumers, or deprecating anything becomes impossible and the registry fills with things nobody dares delete.

Cardinality. Entity counts grow faster than anyone plans. A per-user-per-item feature over a million users and ten thousand items is not a table, it is a decision to compute on demand instead.

Cost visibility. A platform makes it easy to add features and easy to lose track of what each costs to keep fresh. Attribute storage and refresh cost per feature group and review it quarterly, or the bill grows without anyone able to say what to cut.

Streaming features, honestly

Streaming aggregations — counts and sums over the last few minutes, updated continuously — are where feature platforms are most useful and most expensive. They genuinely matter in a narrow set of problems: fraud and abuse, real-time personalisation, dynamic pricing, anything where the behaviour of the last ninety seconds is the signal.

They also bring a full streaming stack into your operational surface: windowing semantics, watermarks, out-of-order arrival, exactly-once concerns, and a backfill story that must reproduce streaming state from historical logs so that training matches serving. That last item is the one that surprises teams. A streaming feature you cannot recompute historically cannot be used in a training set correctly, and rebuilding it from an event log is a project of its own.

The honest test: can you name the decision that changes when a feature is ninety seconds fresh rather than an hour fresh, and quantify what that decision is worth? If yes, build it and budget for the operational weight. If the answer is that fresher seems better, use hourly batch and revisit when a specific decision needs the speed.

A two-week evaluation

Steps two and three usually settle the argument on their own. A team that measures a 0.02% skew rate and a clean point-in-time join does not have the problem the platform solves, and the honest recommendation is to spend the quarter elsewhere. A team that finds three features disagreeing on more than one percent of requests has just found the reason its production numbers never matched the offline ones, and the business case writes itself.

Common objections

The mistakes we are called in to fix

• Training features joined at current time, leaking post-label information into every row
• Two implementations of one definition, disagreeing on nulls, time zones or window edges
• Availability lag ignored, so training uses values that serving could not have had
• A backfill that reproduced today's logic rather than what was knowable at each past moment
• Forty separate lookups per request, with a p99 governed by the slowest of forty
• No defined behaviour for a missing or stale feature, turning a partial outage into silent damage
• A platform bought for the registry, then operated by nobody in particular
• Served feature vectors never logged, making every production question unanswerable after the fact

Before you commit either way

• The problem being solved is written down: skew, point-in-time, latency or discovery
• Actual skew has been measured on at least one model, not assumed
• One training set has been audited for point-in-time correctness end to end
• Feature history is materialised, not overwritten in place
• Availability lag and late-arriving data have explicit, documented policies
• The serving latency budget for feature retrieval is a measured number
• Write volume per hour is known, because it determines the cost
• Every feature has an owner, a freshness expectation and a consumer list
• Missing and stale values have defined behaviour at serving time
• Someone is named to operate whatever is chosen, twelve months from now

Bottom line

Decide on the two problems, not on the category. Measure your skew rate and audit one training set for point-in-time correctness; those two exercises take a week and they answer the question with evidence rather than architecture preference. Most single-team, batch-scoring organisations are better served by one definition in version control, an append-only history table, a shared as-of join helper, a published snapshot for serving and a nightly skew test. Real-time serving under a tight budget, or several teams sharing features across many models, is where a platform starts to pay for its operational weight. Whichever way it goes, log the served feature vector from the first day, because that single decision is what makes every future question about production answerable.

Frequently asked questions