System Design: The Recommender Funnel — Candidate Generation → Ranking → Re-ranking

"Design a recommender system" is one of the most frequent ML system design questions. The classic beginner mistake is to open with whether to use a Two-Tower model or DeepFM. But what the interviewer actually wants to hear is: faced with a billion candidate items and a 100ms latency budget, why must you break the system into a funnel? What problem does each layer solve?

This post runs on the industry-standard three-layer funnel — Candidate Generation → Ranking → Re-ranking — spelling out each layer's goal, model, latency budget, and trade-offs, with the follow-ups interviewers actually ask. Like the rest of the system design series, we align on requirements first, then go down layer by layer.

⏱️ End-to-end latency budget: assume < 200ms. Candidate generation ~10–20ms, ranking ~50–80ms, re-ranking ~10ms, the rest for network and feature fetching. The latency budget is the ruler for every trade-off you make.

Step 0 — Clarify Requirements First (don't rush to draw boxes)

• Business goal: maximize what? Clicks? Watch time? Long-term retention? (Picking the last usually scores higher — it avoids short-term clickbait.)
• Surface: a home feed, related items, or post-search recs? This decides the context and candidate sources.
• Scale: e.g. 1B items, 100M DAU, 10–20 items returned per request.
• Constraints: latency < 200ms, fresh content arriving every few minutes (freshness), need for diversity and explainability.

🎤 Interviewer follow-up: "What exactly is the success metric?" "Why can't one big model just score all items?" (The answer is this whole post — you can't finish the compute.)

Why a Funnel? One Number Makes It Obvious

Suppose you score every item with a heavy ranking model at 1µs per inference. 1B items = 1000 seconds. Out instantly.

The funnel's essence is to cheaply discard 99.99% with a light model, then carefully score the survivors with an expensive one:

1B items
   │  ① Candidate Generation: cheap, high recall
   ▼
~ 1000 candidates
   │  ② Ranking: expensive, high precision
   ▼
~ 100–200 ranked
   │  ③ Re-ranking: business rules, diversity, dedup
   ▼
final 10–20

🎤 Interviewer follow-up: "How much does each layer cut, and why does candidate generation chase recall while ranking chases precision?" (Whatever candidate generation drops is gone forever, so better to over-include than to miss; ranking precisely orders what got let in.)

① Candidate Generation — Cheap, High Recall

Goal: in ~10ms, select a few hundred to a few thousand "worth a closer look" candidates from a billion. The metric is Recall@k, not precision.

Approach (usually multiple retrieval channels in parallel):

• Collaborative filtering / Two-Tower: encode user and item into vectors, precompute all item vectors offline into an ANN index (FAISS / ScaNN / HNSW); online you only compute the user vector and run approximate nearest neighbor search. This is the main workhorse.
• Popularity / trending retrieval: covers new users and exploration.
• Rule / tag-based retrieval: same author, same category, geo.
• Graph retrieval: neighbor diffusion on the user-item bipartite graph.

Union and dedup the multi-channel results, then pass them to ranking.

🎤 Interviewer follow-up: "How does Two-Tower enable real-time retrieval over a billion items?" (Item vectors are computed offline into an index; online there's just one user encoding + ANN query, turning O(N) into roughly O(log N).) "Why multi-channel instead of one?" (A single signal has blind spots; complementary channels sustain recall.)

② Ranking — Expensive, High Precision

Goal: take the ~1000 candidates and use a heavy model to precisely predict engagement probability and order them. Metrics: AUC / LogLoss (offline) and CTR / watch time (online).

Approach:

• Features: only here can you afford rich features — user-side (history, demographics), item-side (content embeddings, stats), cross features, and context (time, device, current session).
• Models: from GBDTs (XGBoost/LightGBM) to deep models (DeepFM, DCN, sequence models like DIN/DIEN). Deep models excel at learning feature crosses automatically and the temporal evolution of user interest.
• Multi-objective: real systems predict several heads at once — click, completion, like, share — then fuse them via a weighted formula into one ranking score, with weights tied to the business goal.

🎤 Interviewer follow-up: "Offline AUC went up but online CTR dropped — how do you debug it?" (Classic: check training-serving skew, online/offline feature mismatch, label definition, A/B design, feedback loops.) "Why multi-objective instead of optimizing CTR alone?" (Chasing CTR alone breeds clickbait and hurts long-term retention.)

③ Re-ranking — Business Rules and Diversity

Goal: the ranking score is not the final order. In ~10ms, re-ranking layers on the business and experience constraints a pure-probability model can't see.

Approach:

• Diversity / de-redundancy: avoid 10 items in a row from the same author/topic, balancing relevance and diversity with MMR or DPP-style methods.
• Business rules: pinning, ad insertion, filtering already-seen/already-purchased, compliance and safety takedowns.
• Freshness and exploration: give new content some exposure (exploration) to avoid the "popular gets more popular" feedback loop.

🎤 Interviewer follow-up: "Why can't the ranking model just learn diversity directly?" (Ranking scores items independently, blind to the combination on a page; diversity is a list-wise property, handled in a stage that sees the whole list.)

Cross-Cutting: Feature Store and Online/Offline Consistency

After the funnel, an interviewer almost always probes the engineering infrastructure. The crux is one diagram:

• Offline: batch-compute item vectors, long-term user features, and stats; write them to the feature store and ANN index.
• Online: low-latency feature reads (Redis / a dedicated feature store), with real-time features (current-session behavior) computed on the fly.
• Consistency: the same feature-computation logic must be shared between training and serving, or training-serving skew tanks online performance — the single most common cause of online regressions.

🎤 Interviewer follow-up: "How do real-time features (what was just clicked) get into ranking?" (Online feature assembly: look up static features from the store + compute session features live, joined just before ranking.) "How do you version features?" (Versioned feature store + point-in-time correctness to prevent label leakage.)

Cold Start: User / Item / System

• New user: no history → lean on popularity retrieval, demographics, an onboarding survey, context (device/geo).
• New item: no interactions → use content features (text/image embeddings) into the two-tower, plus exploration exposure to accumulate signal.
• New system: no data → ship a popularity/rules version first, collect data while training, and gradually replace it.

🎤 Interviewer follow-up: "How does a new item gain exposure fast without hurting overall metrics?" (Exploration budget / bandits: a small traffic slice for exploration, balancing explore-exploit.)

Evaluation and Launch: Offline Metrics → A/B

• Offline: candidate generation by Recall@k; ranking by AUC / NDCG / LogLoss. Don't use accuracy — recommendation is ranking with extreme class imbalance.
• Online: CTR, average dwell/watch time, retention — all validated by A/B test, with guardrail metrics monitored (diversity, complaint rate, latency).
• Monitoring: feature distribution drift, model staleness, per-layer latency SLAs, feedback-loop health.

🎤 Interviewer follow-up: "If it wins offline, can you ship it?" (No — offline is only a filter; the online A/B is the verdict, and offline/online decoupling is itself something to monitor.)

One Table to Close the Funnel

The Interview Framework Cheat

When answering "design a recommender," follow this line and you'll never stall:

1. Clarify goal and scale (long-term retention? 1B items? latency?)
2. Explain why a funnel (one big model can't finish — prove it with the latency number)
3. Go layer by layer: candidate gen (recall) → ranking (precision) → re-ranking (list-wise)
4. Add the cross-cutting: feature store, online/offline parity, cold start
5. Close on evaluation: offline metrics filter, online A/B decides, monitor drift

Remember one line: designing a recommender isn't picking a model — it's designing a funnel from a billion to ten, where each layer solves the right problem at the right cost.