A RAG (Retrieval-Augmented Generation) Q&A system is one of the most commonly asked AI system design questions today — because it's the first LLM project nearly every company ships. The question tests: how do you make an LLM answer from "your private documents," accurately, with citations, and without making things up?
This follows my system design style: problem → layered solution → interviewer follow-ups.
Companion reads: the ML system design 8-step framework and real-time multimodal LLM serving.
📋 The Problem
Design a RAG-based document Q&A system (e.g. an internal knowledge base / support assistant). Users ask in natural language, and the system answers from a private document corpus with citations, supporting multi-turn conversation.
1. Clarify Requirements
- Functional: natural-language Q&A, cited sources, multi-turn; which document formats (PDF, web, Confluence, Slack…)?
- Non-functional: scale (docs, users, QPS), latency (end-to-end < 2–3s), freshness (real-time / daily), access control (users only see what they're allowed to), and "say I don't know."
🎤 Follow-up: "Why RAG instead of fine-tuning the knowledge into the model?" Answer: knowledge changes often (RAG = update docs; fine-tune = retrain), you need citations, it's cheaper, and it enables access control.
2. Overall Architecture: Two Pipelines
A RAG system must split into two independent pipelines — offline indexing and online query:
[Offline Ingestion] docs → parse/clean → chunking → embedding → vector index
[Online Query] question → rewrite → embedding → retrieve → rerank → assemble prompt → LLM → cite
🎤 Follow-up: "Why separate offline and online?" Answer: indexing is heavy, batch, and precomputable; querying must be real-time and low-latency. Separating them lets each scale independently.
3. Offline: Ingestion & Indexing
- Parse & clean: extract text (PDF/HTML), remove noise, preserve structure (heading hierarchy).
- Chunking: split long docs into pieces. Common size is 200–500 tokens with overlap; advanced approaches use semantic chunking (by paragraph/meaning boundaries). Keep metadata per chunk (source, title, permission tags, timestamp).
- Embedding: turn each chunk into a vector with an embedding model; batch it.
- Vector DB: use an ANN index (HNSW / IVF) for approximate nearest neighbor; options like pgvector, Faiss, Milvus, Pinecone; support metadata filtering.
🎤 Follow-up: "How big should a chunk be? What goes wrong if it's too big or too small?" Answer: too big → retrieved content carries irrelevant text, dilutes relevance, eats context; too small → meaning gets fragmented and loses surrounding context. Overlap mitigates lost boundary info.
4. Online: Retrieval & Generation
- Query rewriting: in multi-turn chat, rewrite a pronoun-laden follow-up ("what about it?") into a standalone question; optionally HyDE (have the LLM draft a hypothetical answer first, then retrieve on it).
- Retrieval: embed the question and fetch top-k via vector search. In practice use hybrid search = dense (vector) + keyword (BM25/sparse), covering both semantics and exact terms.
- Rerank: use a cross-encoder to re-score the top-k candidates and keep the most relevant few. Retrieval is the coarse filter; rerank is the fine sort.
- Prompt assembly: system prompt (persona + rules) + retrieved content (with sources) + chat history + the current question + citation instructions.
- Generation: instruct the LLM to answer only from the provided content, cite sources, and say "I don't know" if absent; stream the output.
🎤 Follow-ups: "Retrieval already returns scores — why rerank?" Answer: vector retrieval uses bi-encoder embeddings (fast but coarse); a cross-encoder lets the query and chunk interact deeply for better ranking — but it's slow, so only on the top-k. "Why is hybrid search better than pure vector?" Answer: pure vector often misses exact keywords / proper nouns / code; BM25 covers that.
5. Evaluation (always evaluate RAG in two layers)
- Retrieval layer: Recall@k, MRR, NDCG — does the retrieved content contain the basis for the right answer?
- Generation layer:
- Faithfulness / Groundedness: does the answer stay true to the retrieved content (no fabrication)?
- Answer Relevance: is the answer on-topic?
- Are citations correct?
- Method: build a Golden QA Set (question + reference answer + expected source docs); score automatically with LLM-as-judge or RAGAS.
🎤 Follow-up: "When RAG gives a wrong answer, how do you know if retrieval or generation is at fault?" Answer: layered evaluation — check retrieval Recall@k first (nothing retrieved → a retrieval problem: chunking/embedding/strategy); retrieved but wrong answer → a generation problem (prompt/grounding).
6. Three Advanced Pain Points: Hallucination, Access Control, Freshness
- Hallucination control: enforce grounding ("answer only from the content below"), require per-sentence citations, refuse on low confidence, and return "no relevant info found" when nothing is retrieved.
- Access control (security-critical): filter by metadata at the retrieval layer — each user can only retrieve chunks they're permitted to see. Never retrieve everything and filter afterward (that leaks existence and content).
- Freshness: on document updates, do incremental indexing (only re-embed changed chunks); deletions must remove the vectors too; add versions/timestamps.
🎤 Follow-ups: "User A shouldn't see confidential HR docs — how do you guarantee that?" Answer: attach the user's permission tags as a metadata filter at retrieval time, so unauthorized content is never recalled. "The knowledge base updates daily — what now?" Answer: incremental re-indexing + sync on edits/deletes.
7. Latency / Cost Optimization + Monitoring
- Latency breakdown: embedding (fast) + retrieval (fast) + rerank (medium) + LLM generation (the main bottleneck). Use streaming to cut perceived latency.
- Semantic cache: serve cached answers for similar questions, saving LLM calls and latency.
- Monitoring: retrieval hit rate, citation rate / groundedness, end-to-end latency, cost per query, thumbs up/down, no-answer (refusal) rate, top-k relevance distribution.
🎤 Follow-up: "Which stage is slowest end-to-end, and how do you cut it?" Answer: LLM generation → streaming, a smaller model, semantic cache, and fewer chunks stuffed into the prompt.
One Diagram for the Whole Architecture
[Offline] docs → parse → chunking(+metadata) → embedding → vector DB (HNSW)
│
[Online] question →(multi-turn rewrite)→ embedding → retrieve (hybrid: vector+BM25) ┘
→ rerank (cross-encoder) → assemble prompt (content + cite instructions)
→ LLM generation (grounding/citations/refusal) → stream to user
Access: metadata filter at retrieval Freshness: incremental indexing
Eval: retrieval (Recall@k) + generation (faithfulness) Monitor: citation rate/latency/cost
The Answer Checklist
- Split into two pipelines: offline indexing + online query.
- Five online steps: rewrite → retrieve (hybrid) → rerank → assemble prompt → generate (grounded + cited).
- Evaluate in two layers: retrieval (Recall@k) vs generation (faithfulness), so you can attribute failures.
- Name the three pain points: hallucination (grounding/citations/refusal), access control (filter at retrieval), freshness (incremental indexing).
- Close with: latency (LLM is the bottleneck), semantic cache, monitoring citation rate.
Narrate "two pipelines → retrieve + rerank → two-layer evaluation → hallucination/access/freshness" and this question is in the bag.