The literature is written for a catalog you do not have
The scale assumption in recommender research is load-bearing. It is why the standard architecture has a candidate-generation stage, why it needs an approximate nearest-neighbour index, why the models are large enough to want their own serving tier, and why the evaluation methodology samples negatives instead of scoring everything. Every one of those choices is a workaround for a catalog you cannot enumerate at request time. If your catalog is 800 products, 4,000 courses, or 12,000 parts, you can enumerate it, and the workarounds become the main source of cost and bugs in the system.
You are probably here because
- The recommendations look plausible in a notebook and nobody can say whether they beat the "most popular" shelf they replaced.
- The same forty items appear on every surface and most of the catalog is never shown to anyone.
- New items take six weeks to start appearing, and the catalog turns over faster than that.
- Somebody proposed a vector database, a feature store and a two-tower model for eight thousand products, and the estimate came back at two quarters.
All four trace to the same root: an architecture designed for a catalog three orders of magnitude larger than yours. The sections on full-catalog scoring and on the popularity baseline are where that gets undone.
What follows is the way we build these. It is deliberately unfashionable in places, because the fashionable version loses to a well-tuned simple model on data this thin, and we would rather ship the thing that wins.
What "small" means, in numbers
Two numbers decide which methods are open to you, and neither is the one people quote. The first is catalog size, because it determines whether you can score everything. The second is interaction density: observed user-item interactions divided by users times items. Density is what determines whether a personalization model has anything to learn.
For calibration, the Netflix Prize dataset held 100,480,507 ratings across 480,189 users and 17,770 movies. That is a density of about 1.2 percent, and it is considered a sparse dataset. MovieLens 25M holds 25 million ratings across roughly 162,000 users and 62,000 movies, about 0.25 percent. Most operating catalogs we see are sparser than either, and the interactions are concentrated far more tightly in the head. Run the query on your own logs before you read another paper. It is five minutes of SQL and it eliminates half the candidate approaches.
The third number worth having on hand is interactions per item, at the median. Not the mean, which the head items inflate beyond usefulness. If the median item in your catalog has been purchased eleven times, no amount of model architecture is going to produce a confident personalized estimate for it, and the honest system design is one that says so and falls back to something content-based.
| Catalog size | Retrieval stage | Serving shape | Model class that fits |
|---|---|---|---|
| Under 200 items | Not needed | Precomputed table, cached in the app | Rules and curation beat a model |
| 200 to 5,000 | Not needed | Score the full catalog in process, per request | Item-item, closed-form linear, content blend |
| 5,000 to 50,000 | Optional, and usually still not needed | Full scoring with a filtered eligible set | Item-item, implicit matrix factorization |
| Above 50,000 | Required | ANN index plus a ranking tier | Two-tower retrieval, gradient-boosted or neural ranker |
Default Weights — What Decides a Small-Catalog Recommender
Our default weights, summing to 100. The model is the smallest term, and that ordering is the point.
You can score the entire catalog, and that changes everything downstream
Take a 5,000-item catalog with 256-dimensional item vectors in float32. That is 5,000 by 256 by 4 bytes, about 5.1 MB, which sits comfortably in the memory of the application process that is already handling the request. Scoring a user against the whole catalog is one matrix-vector product: 1.28 million multiply-adds, which a single BLAS call finishes in well under a millisecond. Add eligibility filtering, business rules and a sort, and the entire recommendation is a couple of milliseconds without leaving the process.
The two-stage architecture exists because you cannot do that against two million items. Retrieval narrows the field to a few hundred, ranking sorts them properly. Every part of that pipeline has a cost that nobody itemizes at design time: an index that must be rebuilt and kept fresh, a recall ceiling where anything retrieval misses can never be recovered by the ranker, a network hop in the latency budget, a second service with its own deployment and failure modes, and a whole class of debugging that starts with "the item that should have been first was never a candidate."
Full-catalog scoring also buys three things that are hard to get any other way. Exact offline metrics become free, so you never need sampled negatives. Filters can be applied before scoring rather than after, so the top slots never silently collapse. And explanation becomes tractable: when the score is a sum over a few hundred contributing terms, you can show a merchandiser which past interactions produced a recommendation, which is usually what it takes to get the system trusted internally.
The baseline that most systems never beat
The baseline is not "random." It is segmented, recency-weighted popularity with the business rules applied. Most-purchased in the last thirty days, restricted to the category the user is browsing, excluding what they already own, with a time decay so trends surface. Segment it by one or two attributes that actually matter in your business, such as region, plan tier or industry, and it gets meaningfully stronger for the cost of a GROUP BY.
The reason it is so hard to beat on a small catalog is structural. The head is short. If forty items take sixty percent of interactions, a personalized model has to find its edge in the remaining forty percent, spread across 760 items that individually carry very thin evidence. There is not much room between "show the popular thing" and "show the right thing," and the difference is exactly where the noise lives.
This is not a hunch. The reproducibility work by Dacrema, Cremonesi and Jannach presented at RecSys 2019 examined eighteen neural recommendation papers from top venues. Only seven could be reproduced with reasonable effort, and six of those seven were outperformed by well-tuned conventional baselines, including plain item-based nearest neighbours. The lesson is not that neural methods do not work. It is that the comparison baseline in most published work was weak, and if you build your system against a weak baseline you will convince yourself of a lift that does not exist in production.
So the sequencing is fixed: ship the baseline first, in production, on the real surface, with the real rules. Then measure everything afterwards against it in the same harness. A model that cannot beat segmented popularity is not ready, and knowing that in week two costs far less than discovering it in month six.
Which model class actually fits
| Approach | What it needs | Training at 5,000 items | Where it wins |
|---|---|---|---|
| Segmented popularity | Interaction counts and a timestamp | Seconds, one SQL query | Everywhere, as the floor. Strong on thin data and new users. |
| Item-item nearest neighbours | Co-occurrence counts, cosine or shrunk similarity | Seconds to a minute | "More like this" and session-based slots. Explains itself. |
| Closed-form linear item-item | One regularized inverse of the item Gram matrix | Seconds to a few minutes | Best accuracy per unit of complexity below ~20,000 items. |
| Implicit matrix factorization | Confidence-weighted interactions, enough per user | Minutes | Denser data with repeat behavior. Compact user vectors. |
| Sequence model | Ordered sessions where order carries signal | Hours, plus a serving tier | Media, curricula, replenishment cycles. Not three page views. |
| Two-tower retrieval | Large catalog, rich features, an ANN index | Hours to days, plus ops | Above roughly 100,000 items. Below that it is overhead. |
Item-item nearest neighbours. Build the co-occurrence matrix over items, then normalize. The correction that matters is shrinkage: divide the co-occurrence by the square root of the two item counts plus a shrinkage constant, typically somewhere between 10 and 100, tuned on your own data. Without it, a pair of obscure items that co-occurred twice outranks a pair that co-occurred four hundred times, and your "customers also bought" widget fills with noise. At 5,000 items the matrix is 25 million entries, which is a few hundred megabytes dense and far less sparse. It recomputes nightly in under a minute and it is trivially explainable to the merchandising team.
Closed-form linear item-item models. The EASE formulation published by Steck at WWW 2019 learns a full dense item-item weight matrix in closed form, from a single regularized inverse of the item Gram matrix with the diagonal constrained to zero. There is no gradient descent, no learning-rate schedule, no early stopping, and one hyperparameter. At 5,000 items the inverse takes seconds on a laptop. It is the highest accuracy-per-unit-of-complexity option we know of in this size range, and it is roughly forty lines of NumPy.
Its limit is memory, and the limit is sharp. The weight matrix is the square of the catalog size. At 20,000 items that is 400 million entries, about 1.6 GB in float32. At 50,000 items it is 2.5 billion entries, roughly 10 GB. That is where the closed form stops being the obvious answer and factorization or retrieval starts paying for itself. Knowing the exact number where your approach breaks is worth more than knowing which approach is fashionable.
Implicit matrix factorization. The confidence-weighted alternating least squares formulation from Hu, Koren and Volinsky at ICDM 2008 is the right version for implicit signals, where an absent interaction is not a negative rating but an unknown. It needs enough interactions per user to estimate a user vector. On data where the median user has touched four items, the factors overfit and you have arrived back at popularity with more infrastructure.
Cold start is a permanent operating condition
On a catalog of millions, cold start is a transient that any given item passes through. On a catalog of 600 items with 40 new arrivals a quarter, about seven percent of the catalog is cold at any moment, and those are exactly the items the business most wants shown: the new release, the newly launched course, the product the category manager just brought in. A model that needs fifty interactions before an item is representable will never surface them during the window when surfacing them matters.
The fix is to make the content path first-class rather than a fallback. Embed each item from its own text and attributes, meaning title, description, taxonomy path, brand, specifications, whatever structured fields you have, using a sentence embedding model. Every item then has a vector the day it is created, before anyone has touched it. Similarity in that space is not as good as behavioral similarity once behavior exists, and it is enormously better than nothing during the window when behavior does not.
Two practical notes. Attribute text is often better signal than marketing description, because marketing copy is written to be distinctive and attributes are written to be accurate. And the embedding job belongs in the item-creation path, not the nightly batch, so a product added at ten in the morning is recommendable by ten oh one.
Blending behavior and content without a second code path
Do not branch. A branch on "is this item cold" creates two code paths, two sets of bugs, and an argument about the threshold. Blend continuously instead: final score equals w times the behavioral score plus one minus w times the content score, where w equals n over n plus k, with n the number of interactions observed for that item and k a constant between 20 and 50. A brand-new item has w near zero and is scored purely on content. An item with 400 interactions has w near one. Nothing switches, nothing is forgotten, one function covers the whole lifecycle.
The step teams skip is calibration. The two scores have to be on the same scale before blending or the constant does nothing you intend. Rank-normalize or z-score each within the eligible set first. We have seen more than one system where the blend weight was tuned for a week with no effect, because one score ranged over roughly zero to one and the other over roughly zero to forty.
The same shrinkage works for cold users, which on a small catalog is most sessions. A user with no history gets scored on the current session's items, the referring category, and the segment default, weighted the same way as evidence accumulates within the session. That is usually worth more than anything you can do with a long-term user model, because on a modest catalog the session is where nearly all the intent is.
Approach Fit — 5,000-Item Catalog, Thin Interaction History
Our fit rating for this size class, weighing accuracy against build cost and operating burden. Reverse the bottom two rows above 100,000 items.
Send it over and we will tell you what we would change.
Email your catalog size, your median interactions per item, and the top ten your recommender currently returns for a dozen real users to contact@precisionfederal.com. You get back a short written note naming the three things we would change and why. One business day. No charge, no meeting, no deck.
contact@precisionfederal.comThe feedback loop closes in weeks, not months
When a catalog is small and the recommender drives one or two surfaces, the recommender is most of the exposure. Items that get shown get clicked, gain evidence, and get shown more. Items that never enter the top slots never gain evidence and never will. On a catalog of millions this distortion takes a long time to become visible. On a catalog of a few thousand you can watch it happen inside a month, and the symptom is the one everybody reports: the same forty items everywhere.
Log impressions, not just clicks. Every rendered slot needs a row with the request ID, item, position, model version and timestamp. Without an impression log you cannot separate "nobody wanted it" from "nobody saw it," and every analysis you run afterwards is guesswork dressed as data. This is the single highest-value thing to build first, and it is also the thing most often skipped because it produces nothing visible in week one.
Keep an exploration budget. Five to ten percent of slots go to items sampled outside the top-scored set, weighted toward items with low exposure. It costs a small amount of measured click-through and it is the only mechanism that generates evidence for items the current model has decided against. Without it, the model's opinion becomes self-confirming and the catalog quietly shrinks to whatever was popular the month you launched.
Put coverage on the dashboard next to click-through. Two numbers do the job: distinct items shown per week as a share of the eligible catalog, and the share of total impressions taken by the top ten percent of items. When the second climbs week over week, the system is narrowing, and you will see it long before anyone notices in revenue.
The rules are most of the system
Availability, entitlement, region, contract restriction, age rating, already purchased, already in cart, discontinued, supplier exclusion, minimum order quantity. On a small catalog these are not decoration around a model. For a specific user on a specific surface, the eligible set can be a quarter of the catalog, which means eligibility is doing more work than the scoring function.
Two consequences follow. Filter before you score, not after. Filtering after scoring means the top slots silently collapse and the surface asking for ten items renders five, which is a visible product defect that reads as a model problem. And apply exactly the same filters inside the offline harness. An offline metric computed over the unfiltered catalog measures a system nobody will ever run, and it will disagree with production in ways that consume weeks.
The "already purchased" rule deserves a real definition rather than a boolean. Consumables should reappear on a replenishment cycle estimated from the item's own repurchase interval. Durables should not reappear. Active subscriptions should not reappear while active, but should reappear at renewal. Getting this wrong is the most common production complaint about recommenders, and it has nothing to do with the model.
Measuring it when traffic is thin
Offline, score the full catalog and compute exact metrics. Do not sample negatives. Krichene and Rendle showed at KDD 2020 that sampled metrics are not consistent with their exact counterparts and can invert the relative ranking of algorithms, which means a sampled evaluation can tell you the wrong model won. On a small catalog there is no computational reason to sample at all, so the only reason to do it is that a tutorial did.
Report recall and NDCG at k as a ratio against the popularity baseline, not as absolute numbers. Recall@10 out of 800 items is a fundamentally easier task than recall@10 out of two million, so the absolute figure carries no information a reader can use, while "1.4 times the baseline on the same split with the same filters" does.
Split temporally, never randomly. A random split leaks future interactions into training and inflates every number in the report. Train on everything before a cutoff date, evaluate on what happened after, and use a holdout window long enough to include a full weekly cycle of your business.
Online, interleaving beats a split test when traffic is thin. Merge the two rankings into one list, attribute each click to whichever ranker contributed the item, and compare within the session. Because each user sees both systems, between-user variance drops out, and published search-evaluation work puts the sensitivity gain at roughly an order of magnitude over an A/B comparison on the same question. When interleaving is impractical, pick a metric close to the change, such as click-through on the recommendation surface itself, and accept that you are measuring the surface rather than the business.
An eight-week build order
Build Order
Two weeks of that plan produce a shipped system, and weeks three through eight produce the evidence to say whether anything after week two helped. That ordering is deliberate. A recommender project that spends its first two months on modelling arrives at the same place with no baseline to compare against, no impression log to analyze, and no way to answer the only question the business will ask.
Where the First Eight Weeks of Effort Go
Share of engineering effort in a typical first build. Model fitting is twelve percent, and that is not an accident.
The mistakes we find most often
- No impression log. Clicks alone cannot distinguish an item nobody wanted from an item nobody was shown, and every downstream analysis inherits that ambiguity.
- No baseline in the comparison. A lift measured against random or against nothing is not a lift. Segmented popularity is the floor and it belongs in every report.
- Sampled evaluation metrics. Free to avoid at this catalog size, and demonstrated to reorder algorithm rankings when you do not.
- Random train/test splits. Future interactions leak backwards and every number in the deck is optimistic.
- Filters applied after scoring. Slots collapse, the surface renders short, and the bug gets attributed to the model.
- Retrieval infrastructure nobody needed. A vector database and an ANN index in front of eight thousand items adds a recall ceiling and a network hop in exchange for nothing.
- Optimizing click-through alone. It reliably converges on items the user was going to find anyway, and the recommender ends up taking credit for existing demand.
- Treating cold start as a launch phase. On a catalog that turns over quarterly, the cold items are the ones with the most commercial urgency behind them.
Before the first version goes live
- Impressions, clicks and conversions logged with request ID, position and model version
- Eligibility rules implemented once, in one place, and applied before scoring
- Segmented popularity running in production as the comparison arm
- Offline harness with a temporal split, full-catalog scoring and the same filters as production
- Every item embedded from its own attributes at creation time, not on the nightly batch
- Evidence-weighted blend so a new item is recommendable the hour it is added
- Exploration budget set, and coverage plus top-decile impression share on the dashboard
- A kill switch that returns the surface to the baseline without a deploy
When not to build one at all
Under roughly 150 items with a browsable structure, good search, real facets and a curated shelf beat a model outright. Curating sixty items takes one person a few hours a month and produces something better than a model trained on nine hundred interactions, with the added benefit that a human can explain every placement.
Skip it also when there is no repeat behavior to learn from, which is common in businesses where a customer buys once in a lifetime, and when the surface has a single slot. One slot is a ranking problem with a business rule attached, not a personalization problem, and it should be solved as one.
The honest version of this decision is worth having early, because a recommender is a permanent operating commitment. Something has to retrain it, watch it, and answer for it when a merchandiser asks why their product disappeared. If the catalog does not justify that, the money is better spent on search and on the product data that search runs against.
Bottom line
Small catalogs are not a scaled-down version of the big-catalog problem. They are a different problem where the binding constraints are evidence per item, catalog turnover, and exposure concentration, not compute. The system that fits scores everything on every request, treats content as a first-class signal rather than a fallback, logs what it showed as carefully as what was clicked, spends a few percent of its impressions buying information, and measures itself against a baseline strong enough to be embarrassing. Build that, and the modelling question becomes small enough to answer in an afternoon.
Frequently asked questions
Roughly 150 to 200 items with real browsing behavior is the point where curation stops scaling. Below that, search, facets and a curated shelf produce better results for far less operating cost. Above it, the interaction data starts carrying signal a person cannot hold in their head.
Almost certainly not. A 5,000-item catalog with 256-dimensional vectors is about 5 MB in memory and can be scored exhaustively in under a millisecond. An approximate index adds a recall ceiling, a network hop and a service to operate, in exchange for solving a problem that starts around 100,000 items.
Embed every item from its own title, description and attributes at creation time, and blend the content score with the behavioral score using a weight of n over n plus k, where n is the item's interaction count. New items are scored on content, established items on behavior, and nothing has to switch over.
Compare it against segmented, recency-weighted popularity in the same offline harness with the same filters and a temporal split, then confirm online with an interleaved comparison, which needs far fewer sessions than a split test. Track catalog coverage alongside click-through so a narrowing system is visible before it costs you.
The exposure feedback loop. Items shown gain evidence and get shown more, while items never shown never gain any. On a small catalog it becomes visible within a month. The fixes are an impression log, an exploration budget of five to ten percent of slots weighted toward low-exposure items, and a coverage metric on the dashboard.