System Design: A Feed / Search Ranking System (Features, Online/Offline, A/B Testing)

"Design a feed / search ranking system" is a cousin of the recommender funnel, but the follow-ups land differently. The recommender question tests "why a funnel"; the ranking question tests three more production-flavored things: how features are computed, how online and offline stay consistent, and how lift is measured. Beginners stall after describing the model — and those three questions are exactly where seniority shows.

This post focuses on the ranking layer itself (assume candidates are already retrieved), running on feed/search ranking, emphasizing the feature stack, training/serving parity, the data loop, and A/B testing, with the follow-ups interviewers actually ask.

⏱️ Latency budget: ranking scores hundreds to thousands of candidates, usually < 100ms. Feature fetching dominates latency, so "where features live and how they're fetched" is the design core — not the model.

Step 1 — Clarify: How Feed and Search Ranking Goals Differ

• Search ranking: there's an explicit query; relevance comes first. The goal is often "query-doc relevance + user satisfaction (clicks, dwell, no reformulation)."
• Feed ranking: there's no query; you infer interest from the user profile and context. The goal is usually long-term engagement and retention — beware over-optimizing instant clicks.
• In common: both frame as learning to rank — score candidates and order them, with offline and online metrics.

🎤 Interviewer follow-up: "How do the label definitions differ for Search vs Feed?" (Search can use clicks + human relevance judgments; Feed leans on implicit feedback — completion, like, share, negative feedback.) "Why not just optimize CTR?" (Clickbait, echo chambers, hurts long-term retention.)

Step 2 — Frame It as Learning to Rank

• Input: a feature vector over (user, query?, candidate_doc, context).
• Output: a ranking score (or a fusion of multi-objective scores).
• Three classic training objectives:
  • Pointwise: predict an engagement probability (e.g. CTR) per doc — simplest, but ignores relative order among docs on a page.
  • Pairwise (e.g. LambdaMART / RankNet): learn "doc A should rank above doc B," optimizing relative order directly.
  • Listwise (e.g. LambdaRank on NDCG): optimize the whole list's ranking metric directly.
• In practice GBDTs (LambdaMART) remain a strong search-ranking baseline; deep models excel with large sparse features and sequences.

🎤 Interviewer follow-up: "Pointwise vs pairwise — how do you choose?" (Care about absolute probability/calibration → pointwise; care only about ranking quality → pairwise/listwise, usually better NDCG.)

Step 3 — The Feature Stack (the real crux) ⭐

A ranking system lives or dies 70% on features. In an interview, classify them clearly by source and timeliness:

The key split — offline vs real-time:

• Offline batch features: affordable heavy compute, broad coverage (embeddings, long-term stats), precomputed into a feature store for low-latency online reads.
• Real-time features: current-session behavior, the live query — must be computed on the fly.

🎤 Interviewer follow-up: "How do you stop a feature like historical CTR from creating a self-reinforcing feedback loop?" (Position-bias correction, exploration, counterfactual/IPS weighting.) "What about a cold-start doc with no historical CTR?" (Fall back to content features + exploration exposure.)

Step 4 — Online/Offline Architecture and Training-Serving Parity ⭐

This is both the gimme and the landmine of ranking interviews. The crux is a two-path diagram:

Offline (training)                    Online (serving)
┌──────────────┐                    ┌──────────────┐
│ behavior logs │                    │ request (user, │
│  → join labels│                    │   query, ctx)  │
│  → compute    │   same feature     │  → fetch       │
│    features   │     logic          │    features    │
│  → train model│ ◄───── shared ───► │  → model score │
│  → offline eval│                   │  → rank & serve│
└──────────────┘                    └──────────────┘
        │                                  │
        └──── ship model + feature versions together ──┘

• Training-serving skew: the same feature-computation logic must be shared between training and serving. Compute in Spark offline but rewrite it in Java online, and any mismatch tanks production. Fix: share one feature-transformation codebase, or unify outputs via a feature store.
• Point-in-time correctness: a training sample's feature values must be those as of that timestamp — no future info (label leakage).
• Online feature assembly: look up static features from the store, compute session features live, and assemble the full vector before scoring.

🎤 Interviewer follow-up (classic): "Offline AUC/NDCG went up but online didn't move (or dropped) — how do you debug?" (In order: training-serving skew → online/offline feature mismatch → label definition/sampling bias → offline metric decoupled from business → A/B design or novelty effect.)

Step 5 — The Data Loop: Log-and-Train

A ranking system produces its own training data — but here lies the deepest trap:

• What to log: for every ranking, all candidates + a feature snapshot + display position + user feedback, not just the clicked one.
• Position bias: items shown higher get clicked more regardless of relevance; using raw clicks as labels makes the model learn "position," not relevance. Fixes: randomization experiments to estimate propensity, or include position as a feature but fix it at serving.
• Unexposed candidates: docs never shown get no feedback, causing selection bias — exploration traffic is needed to fill it.

🎤 Interviewer follow-up: "Your training data is chosen by the model itself — how do you avoid drifting more biased over time?" (Exploration + debiasing: position-debias, IPS weighting, a random holdout slice for unbiased data.)

Step 6 — Evaluation and A/B Testing (quantifying lift) ⭐

Offline is only a filter; the online A/B is the verdict.

• Offline metrics: Search uses NDCG@k, MAP, MRR; Feed uses AUC/LogLoss + ranking metrics. Not accuracy.
• A/B test design essentials:
  • Randomization unit: usually bucket by user (so one person isn't flipped between A and B), via a stable hash.
  • Primary metric + guardrails: a primary metric (session satisfaction, long-term retention) plus guardrails (latency, diversity, complaint rate, revenue) — avoid "won clicks, lost experience."
  • Statistical significance: size the experiment and run a full cycle (covering weekends/weekdays), use significance tests, and don't peek and stop early (peeking inflates false positives).
  • Novelty / primacy effects: a new feature may transiently look better/worse; run long enough to reach steady state.
• Post-launch monitoring: feature distribution drift, model staleness, per-percentile latency, A/B-vs-rollout consistency.

🎤 Interviewer follow-up: "The A/B primary metric is up 0.5% but latency rose too — do you ship?" (Discuss the trade-off and guardrails: latency's long-term retention damage vs the short-term gain; you may need to fix latency first.) "How do you rule out a novelty effect?" (Extend the experiment; check whether the metric curve decays over time.)

One Table to Close

The Answer Framework Cheat

When asked to "design a ranking system," follow this line:

1. Clarify Search vs Feed (query or not, what you optimize)
2. Frame as learning to rank (point/pair/list-wise + offline metrics)
3. Emphasize features (classify by source/timeliness, offline vs real-time, feature store)
4. Explain online/offline architecture (training-serving parity is the core, point-in-time)
5. Cover the data loop (log-and-train + position-bias debiasing)
6. Close on A/B testing (user bucketing, guardrails, significance, anti-novelty)

Remember one line: a ranking system's hard part is never the model — it's "feature consistency" and "measurable lift." Nail those two and you've won the question.