System Design: An ML Training + Serving Pipeline (Feature Store, Model Versioning)

"Design an ML training + serving pipeline" is the most MLOps-flavored system design question. It doesn't test model tuning — it tests whether you can string "data → features → training → deployment → monitoring" into a reproducible, versioned, safely-shippable, rollback-able production line. Beginners draw a single "train → deploy" arrow and stop; seniority shows in explaining three things clearly: the feature store's consistency, model version governance, and progressive rollout.

This post runs on an end-to-end pipeline, emphasizing the two cores you asked for — the feature store and model versioning — plus the training pipeline, deployment strategy, and monitoring, with the follow-ups interviewers ask.

⏱️ Design through-line: the whole pipeline must answer one question — "for any online prediction, can I fully reproduce which model, which features, and which data it used?" Answer that and the question is in the bag.

Step 1 — Clarify Requirements

• Task type: batch prediction (offline scoring, e.g. daily recs) or real-time online inference (< 100ms)? The serving architectures differ wildly.
• Scale / frequency: how many features, how many models, how often retrained (daily? hourly? triggered?).
• Constraints: latency, reproducibility, compliance (data lineage, auditability), team collaboration (many people editing features/models).

🎤 Interviewer follow-up: "Is your system offline batch or online real-time?" (Decides whether you need an online feature store and low-latency model serving.) "What triggers retraining?" (Schedule / data volume / metric decay.)

Step 2 — End-to-End Architecture at a Glance

        ┌─────────────────── Feature Store ───────────────────┐
        │   Offline store             Online store            │
        │   train-time fetch, backfill   low-latency reads     │
        └─────────▲───────────────────────────▲───────────────┘
                  │ one feature definition      │
   raw data ──► feature eng ──► training pipe ──► model registry ──► serving
   (logs/DB)   (batch/stream)   (reproducible)   (version+metadata)  (online/batch)
                                                     │
                            monitoring (drift/metrics/latency) ◄── pred logs ◄──┘

Two through-lines: offline (training) and online (serving), meeting at the feature store. We unpack each below.

Step 3 — Feature Store (one of the two cores) ⭐

A feature store exists to solve an old problem: the features used in training and serving don't match (training-serving skew). Its design points:

• Two stores, one definition:
  • Offline store (warehouse/Parquet): historical features for training and batch scoring — large, backfillable.
  • Online store (Redis/low-latency KV): the latest feature values for real-time inference — millisecond reads.
  • Key: both are produced from one feature definition / transformation logic, avoiding the "Spark computes one thing offline, Java computes another online" skew.
• Point-in-time correctness: when assembling training samples, take feature values as of the label event's timestamp — no future info (else label leakage inflates offline scores and craters online).
• Feature versioning and lineage: each feature has a definition version, source, and computation logic — auditable and traceable.
• Sharing and reuse: models share features instead of reinventing them; newcomers can discover "what features exist."

🎤 Interviewer follow-up: "How does the online store stay fresh?" (Streaming updates + batch backfill, a lambda architecture; depends on feature timeliness.) "Why does point-in-time join matter?" (Without it you use future features and fake the training score.) "What if the online store goes down?" (Degrade to defaults / last-cached version, with alerting.)

Step 4 — Training Pipeline (reproducibility is the floor)

Training shouldn't be "someone running a notebook on a laptop" — it's a reproducible, schedulable, traceable pipeline:

• Steps: fetch data → point-in-time feature assembly → split (train/val/test, with time-based splits to prevent leakage) → train → offline eval → emit a model artifact + metadata.
• Three reproducibility ingredients: pin the data version, feature version, and code/hyperparameter version. Same inputs must yield the same model.
• Orchestration: a workflow orchestrator (Airflow / Kubeflow / Metaflow) for DAGs, retries, scheduling, backfills.
• Triggers: scheduled, data-arrival, or auto-retrain when monitoring detects metric decay.

🎤 Interviewer follow-up: "How do you reproduce this model six months later?" (Version data + features + code + environment; store the artifact with metadata in the registry.) "Why time-based splits, not random, for time series?" (Random splits leak future data into the training set.)

Step 5 — Model Versioning and Registry (the second core) ⭐

A model isn't a file — it's a versioned artifact with metadata. The model registry governs it:

• What each model version records: the artifact, the data/feature versions used, hyperparameters, offline metrics, training time, author, and the corresponding feature schema.
• Lifecycle stages: staging (to be validated) → production (live) → archived (retired). One-click promotion/rollback.
• Model ↔ feature binding: a model version must bind to the feature versions and schema it depends on, or a new model paired with old features fails silently.
• Rollback-able: production can always one-click revert to the last known-good version — the safety net for shipping.

🎤 Interviewer follow-up: "How do you manage two model versions running A/B online?" (Registry tags + the serving layer routes traffic to versions.) "What if the model and feature schema mismatch?" (Feature order/types misalign → predictions silently wrong with no error, so validate the schema.)

Step 6 — Serving and Progressive Rollout (ship safely) ⭐

A new model is never shipped at 100% directly — ramp it up:

• Serving forms:
  • Online real-time: a model service (REST/gRPC), features from the online store, low latency; optimize with batching, GPUs, caching.
  • Batch: scheduled offline scoring written to a results table / online store.
• Progressive rollout strategies:
  • Shadow mode: the new model runs on live traffic but doesn't affect users — you only compare prediction differences to validate stability and consistency first.
  • Canary / gradual: 1% → 5% → 50% → 100%, watching metrics and guardrails at each step.
  • A/B test: scientifically quantify the new model's business impact (see the previous ranking-system post).
• Always rollback-able: at any stage, one-click revert if metrics go wrong.

🎤 Interviewer follow-up: "Shadow vs canary — what's the difference?" (Shadow doesn't affect users and only collects predictions; canary actually serves a small slice of traffic.) "How do you decide the canary ramp speed?" (Guardrail-metric stability + whether the sample is significant enough.)

Step 7 — Monitoring and the Data Loop

Shipping is the beginning — keep watching three signal classes:

• Data/feature layer: input distribution drift (data drift), feature missing rates, online/offline feature consistency.
• Model layer: prediction distribution drift, online metrics (if labels exist), model staleness.
• System layer: latency (per percentile), throughput, error rate, online-store hit rate.
• The loop: predictions + ground-truth feedback flow back to the offline store, becoming data for the next retrain; detected decay triggers retraining.

🎤 Interviewer follow-up: "How do you monitor model quality without real-time labels (e.g. risk control)?" (Proxy metrics + data drift + delayed-label backfill; set drift alerts.) "How do you catch training-serving skew after launch?" (Log the features actually fed to the model online and compare against an offline recomputation.)

One Table to Close

The Answer Framework Cheat

When asked to "design an ML pipeline," follow this line:

1. Clarify offline batch vs online real-time (decides the architecture)
2. Draw two end-to-end through-lines (offline training / online serving, meeting at the feature store)
3. Emphasize the Feature Store (two stores one definition, point-in-time, avoid skew)
4. Make training reproducible (version data/features/code)
5. Cover the Model Registry (versions, bind model to feature schema, rollback)
6. Ship safely (shadow → canary → A/B, always rollback-able)
7. Close on the monitoring loop (three-layer drift + retrain trigger)

Remember one line: an ML pipeline's hard part isn't training the model — it's "consistency" (features), "reproducibility" (training), "governance" (versions), and "safe rollout" (deployment). String those four into a loop and you've nailed the question.