"Design a content moderation / fraud detection system" is a variant of a classification problem, but its focus is totally different from recommendation or ranking. Its crux is one sentence:
Positives are extremely rare (maybe < 1%), and the costs of a false alarm vs a miss are severely asymmetric.
Those two traits make "handling imbalanced data" and "the precision/recall trade-off" the real test. Two traps beginners fall into: ① using accuracy as the metric (99% accuracy might catch zero fraud); ② only thinking "raise the model's F1" without discussing "which is costlier — wrongly blocking a legit user, or letting a fraudster through?"
This post runs on fraud detection / content moderation (same essence: finding a rare malicious few in a sea of normal), emphasizing metric choice, imbalanced data, the precision/recall threshold trade-off, and human-in-the-loop, with the follow-ups interviewers ask.
⏱️ Design through-line: this isn't about "how accurate" but "choosing the right operating point between precision and recall based on business cost." Nail that sentence and you've won half the question.
Step 1 — Clarify Requirements and the "Cost of Errors"
- Business goal: detect what (stolen cards? fake accounts? hate speech?)? Do what on a catch (block outright? send to human review? down-rank?).
- The cost of the two error types (most critical):
- False Positive (over-kill): flag a legit user/content as malicious → hurt experience, complaints, churn.
- False Negative (miss): let real fraud/violation through → monetary loss, platform risk, compliance issues.
- Which is costlier? Fraud often makes "misses expensive" (direct loss); moderation often makes "over-kills expensive" (harms speech/experience) — this directly decides whether you lean toward recall or precision.
- Scale / latency: must you block transactions in real time (< 100ms), or is near-real-time/batch acceptable?
🎤 Interviewer follow-up: "Which costs more, FP or FN, and why?" (The anchor question for the whole problem — miss it and the rest is hollow.) "Real-time interception or after-the-fact review?" (Decides the online/offline architecture.)
Step 2 — Why You Can't Use Accuracy (metric choice) ⭐
Suppose fraud rate is 0.5%. A model that "calls everything normal" has accuracy = 99.5%, but recall = 0 and is useless. Under extreme imbalance, accuracy lies.
The right metrics:
- Precision: of what you flagged, how much was truly bad → measures over-kill.
- Recall: of all the bad, how much you caught → measures misses.
- PR curve / PR-AUC: under imbalance, the PR curve is more sensitive than ROC-AUC (ROC is over-optimistic when negatives dominate).
- Pick the operating point by cost: use Fβ (β>1 leans recall, β<1 leans precision), or plot the precision-recall curve and choose a threshold by business cost.
🎤 Interviewer follow-up (guaranteed): "Why not accuracy?" "Under imbalance, ROC-AUC or PR-AUC — which is better and why?" (PR-AUC, because it focuses on the rare positive class and isn't diluted by the negative flood.)
Step 3 — The Precision / Recall Threshold Trade-off ⭐
The model outputs a probability score, not black-and-white. Turning it into "block / allow" requires a threshold — and the threshold is the precision↔recall slider:
- Raise the threshold → precision↑, recall↓ (more conservative: fewer over-kills, more misses).
- Lower the threshold → recall↑, precision↓ (more aggressive: fewer misses, more over-kills).
In practice it's not "one hard threshold" but tiered handling:
- High score (high confidence of fraud) → auto-block.
- Mid score (gray zone) → send to human review or require extra verification (2FA, captcha).
- Low score → allow.
This balances precision (the auto-block band must be very accurate) and recall (the gray zone catches more via humans).
🎤 Interviewer follow-up: "How would you pick this threshold?" (Plot the PR curve by FP/FN cost and choose an operating point, or set an "acceptable over-kill rate" and back out the threshold.) "Why tiers instead of one threshold?" (Auto-block needs high precision; hand the gray zone to humans for recall — balancing both.)
Step 4 — Handling Imbalanced Data ⭐
With too few positives, the model "cheats" by predicting all negative. Countermeasures at three layers:
- Data layer:
- Oversample positives (e.g. SMOTE synthesis) or undersample negatives to balance the training distribution.
- Careful: sample only on the training set; keep validation/test at the true distribution or metrics distort.
- Algorithm layer:
- Class weights / cost-sensitive learning: penalize positive-class errors more (e.g.
class_weight, focal loss). - Tree models'
scale_pos_weight(XGBoost) is a common lever too.
- Class weights / cost-sensitive learning: penalize positive-class errors more (e.g.
- Problem-framing layer:
- When extremely rare, frame as anomaly detection — learn "what normal looks like" and flag deviations, rather than supervised classification.
🎤 Interviewer follow-up: "Do you SMOTE before or after the train/test split?" (After, and only on the training set — doing it before leaks synthetic info into the test set.) "Oversampling vs class weights — how to choose?" (Enough data → weights are simple and stable; extremely rare → consider anomaly detection.)
Step 5 — Features and Architecture (real-time + offline)
- Features:
- Entity features: account age, history, credit score.
- Behavioral/sequence features: transaction frequency in a short window, geo jumps, device fingerprint.
- Graph features: fraud often comes in rings — use a user-device-IP graph to find suspicious communities (graph-based detection).
- Real-time vs offline: real-time interception needs on-the-fly velocity features (e.g. "transactions in the last 5 minutes").
- Architecture:
- Real-time path: a low-latency model + rules engine (hard rules block the obvious first), < 100ms decisions.
- Offline path: batch retraining, label backfill, mining new fraud patterns.
- Rules + model hybrid: rules block known patterns (explainable, fast to ship), the model catches the unknown.
🎤 Interviewer follow-up: "Why a rules engine if you have a model?" (Rules are explainable, ship instantly against new attacks, need no retraining; the model covers the rules' blind spots.) "How are real-time velocity features computed?" (Streaming aggregation / sliding-window counts in a low-latency store.)
Step 6 — Human-in-the-Loop, Adversaries, and Delayed Feedback
The "advanced points" here are almost all in these three:
- Human-in-the-loop: the gray zone goes to human review; the verdicts flow back as high-quality labels, forming a data loop.
- Adversarial: fraudsters actively evolve to evade the model — so monitor drift continuously, retrain fast, and patch holes with rules in real time. Very unlike a "static" recommender.
- Delayed labels: the "true label" for fraud may settle days later (chargebacks, appeals) — training must handle label delay, and evaluation must beware that "recent data lacks complete labels."
🎤 Interviewer follow-up: "Fraudsters adapt to your model — what do you do?" (Continuous monitoring + fast retraining + rule hot-fixes + exploration; treat it as an 'arms race,' not a one-shot model.) "Labels take 30 days to settle — how do you evaluate in real time?" (Early proxy metrics + delayed-label backfill correction.)
One Table to Close
| Stage | Focus | Interview gimme |
|---|---|---|
| Frame | asymmetric FP vs FN cost | anchor "which error is costlier" first |
| Metrics | can't use accuracy | PR-AUC, precision/recall |
| Threshold | precision↔recall slider | tiered handling (auto-block/human/allow) |
| Imbalance | sampling / class weights / anomaly | SMOTE only on the training set |
| Features/arch | real-time + rules + model + graph | velocity features, rules for blind spots |
| Adversarial loop | human review, adversaries, delayed labels | treat as arms race, handle delayed labels |
The Answer Framework Cheat
When asked to "design fraud detection / content moderation," follow this line:
- Anchor the cost of errors (FP vs FN — which is costlier, decides precision vs recall lean)
- Cover metrics (why not accuracy → PR-AUC, precision/recall)
- Cover the threshold trade-off (tiered handling: auto-block / human review / allow)
- Cover imbalance handling (sampling / class weights / anomaly detection, SMOTE only on training)
- Cover features and architecture (real-time velocity + rules engine + model + graph features)
- Close on the adversarial loop (human-in-the-loop, adversarial retraining, delayed labels)
Remember one line: the hard part of these systems isn't "how accurate the model is" — it's "under extreme imbalance, choosing the right precision/recall operating point by error cost, and continuously fighting an evolving adversary." Nail the cost-driven trade-off and you've answered the soul of this question.