Advanced System Design: A Real-Time Multimodal AI Assistant at 1M DAU (LLM Serving Deep Dive)

📺 The problem and architecture in this post are based on s09g's video AI System Design - MultiModel inference, reorganized in my own words with an added "interviewer follow-up" angle.

Classic system design asks you to "design a recommender / feed." But in 2026, ML/AI interviews increasingly ask something harder — how do you actually serve a large multimodal LLM in production? This tests not model tuning, but the systems engineering of GPU cost, inference scheduling, streaming, safety, and monitoring.

This is the advanced, applied companion to my earlier post on the ML system design 8-step framework: clarify first, estimate, then design layer by layer.

📋 The Problem

Design the backend for an AI assistant serving 1 million daily active users (1M DAU) of real-time conversation. It must support multimodal input (text, image, audio) and multimodal output (streamed text, speech, image generation), and maintain multi-turn context. Requirements: reliable, low-latency, scalable, observable.

This is too big to answer in one pass, so we go layer by layer, each time stating the "why" before the "how."

1. Requirements & Cost Estimation (figure out the GPU bill first)

Estimating scale right away is the biggest scoring move here — it directly drives the architecture.

Deriving QPS:

• Requests/day = 1,000,000 users × 3 sessions × 5 rounds = 15M requests
• Traffic concentrated in ~8 active hours → average 15,000,000 / (8 × 3600) ≈ 520 QPS
• Traffic is bursty; keep a 3× buffer → the system must handle ~1,500 QPS

Token throughput & GPU count (using H20):

• Per request ≈ 2000 input (history + RAG) + 500 output = 2500 tokens
• Total throughput = 520 × 2500 ≈ 1.3M tokens/s
• A single GPU's real online throughput ≈ 4000 tokens/s (limited by KV cache, prefill latency — far below theoretical)
• Average GPUs = 1.3M / 4000 ≈ 325; peak ×3 ≈ 1000 GPUs

Cost: 1000 GPUs around the clock = 1000 × $3/hr × 24 × 30 ≈ $2.16M/month. With auto-scaling (scale down off-peak) you cut it to $300K–400K/month — nearly 80% savings.

🎤 Follow-ups: "Why 8 hours, not 24?" "Where does 4000 tokens/s come from (why not the theoretical number)?" "Why a 3× peak buffer?"

2. Multimodal Request Orchestration (the pipeline)

Each modality is processed before the LLM, then assembled into one full query.

Audio

• Use Batch ASR for input, not Streaming ASR: after VAD segments the speech, transcribe the whole clip at once. Batch services are stateless, scale horizontally easily, and are more accurate; streaming requires maintaining complex long-lived state.
• Stream TTS for output: don't wait for all text — every 10–20 characters (by punctuation), hand off to TTS for async synthesis, streaming audio back.

Vision

• Vision Preprocessor: download, decode, resize, split into patches → produce a pixel tensor.

Prompt Assembly

Combine into one full query:

1. System prompt (persona)
2. User profile + RAG content
3. Chat history (multi-turn context)
4. Image placeholder <image_0>
5. The current user query

Embedding Injection: Native VLM vs Cascading

• Cascading: a separate machine runs the Vision Encoder (e.g. CLIP), then ships embeddings over the network to the LLM machine. Extra network hop, and without joint training, alignment is worse.
• Native VLM ✅: encoder and decoder live in one model, one process, trained end-to-end, with better image-text alignment and embedding injection done in-GPU for lower latency.

🎤 Follow-ups: "Why batch ASR over streaming?" "Why doesn't TTS wait for generation to finish?" "Native VLM beats cascading — so what's its cost?" (→ leads into section 3)

3. Resource Contention & Optimization (native VLM's hard pain point)

The cost of going native: encoder and decoder share one GPU. A single image can cost as much compute as hundreds of tokens; when it arrives it grabs the CUDA cores, stalling requests mid-stream (the decode phase), shrinking KV cache space, and dropping concurrency. Three fixes:

🎤 Follow-ups: "Can the encoder cache be DDoSed?" (→ quota + eviction) "How do you keep a 5-image request from blocking the whole GPU queue?" (→ compute budget)

4. Routing & Autoscaling

You can't round-robin an LLM — request weights vary wildly (prefill is heavy, decode is light) and there's KV cache state. You need two layers of routing:

• Semantic Router: a tiny classifier/rules engine up front decides intent. Simple chat → a small model (7B/8B); deep reasoning (coding, math) → a thinking model; and whether to trigger RAG.
• Inference-engine Router (LLM Router): aware of the backend vLLM/TensorRT-LLM cluster state — each pod's KV cache headroom, batch size, queue depth, whether it's mid-prefill — and scores/forwards precisely, often via an EPP (Endpoint Predictor) proxy.

Autoscaling: Kubernetes HPA with custom metrics (concurrency, KV cache utilization), or a dedicated LLM scheduler. Must handle uneven prefill/decode cost, KV cache state, and cold start.

🎤 Follow-ups: "Why not round-robin / a standard Nginx LB?" "On scale-out, how do you handle cold start (loading weights takes tens of seconds)?"

5. Streaming Output & Safety (guardrails)

For streamed output, safety uses a dual track:

• Incremental lightweight check: every ~50 tokens, run a few-hundred-MB fast classifier; on a violation, cut the stream immediately and show a canned reply ("Sorry, I can't answer that").
• Full async check: doesn't block the stream — the backend sends the complete answer to a large guard model (e.g. Llama-Guard) for full scoring, used for labeling, blacklisting, and alerts.

🎤 Follow-up: "If you catch a violation mid-stream, hasn't the user already seen it?" (→ the incremental check's truncation window vs the full audit afterward — they complement each other)

6. The 4D Monitoring System

Operating an LLM in production means monitoring four dimensions:

🔑 TTFT determines how "responsive" it feels; if ITL is too high, the AI seems to "stutter." These are the two core latency metrics in LLM serving.

🎤 Follow-up: "What's the difference between TTFT and ITL? Which is affected by prefill, which by decode?"

7. The Data Flywheel & Continuous Improvement

Launch is just the start. Build a loop so the system keeps improving — without retraining the base weights every time:

1. Collect: gather full traces + user feedback + online metrics; classify and attribute bad cases.
2. Five test sets:
   • Golden Set: core capabilities
   • Fix Set: freshly-surfaced online issues
   • Red Team Set: jailbreak / adversarial safety
   • Multimodal Set: interleaved image-text, noisy audio, etc.
   • Regression Set: every previously-fixed bug, tested on every release
3. Two improvement tracks:
   • Shallow (hours): tweak prompts, improve RAG reranking, adjust decoding params
   • Deep (weeks): mine data for fine-tuning or DPO/RLHF
4. Evaluate & ship: offline eval (replay + scoring) → quality gate → online canary A/B → new loop.

🎤 Follow-up: "Without retraining the model, how do you fix 80% of online issues fastest?" (→ shallow fixes: prompt / RAG / decoding params)

One Diagram for the Whole Architecture

Input (text/image/audio) → [Preprocess: ASR / Vision Preprocessor]
        → [Prompt Assembly + Embedding Injection (native VLM)]
        → [Semantic Router → Inference Router (KV-cache/queue aware)]
        → [GPU cluster: Encoder Cache / Compute Budget anti-contention]
        → [Streamed output + incremental safety check] → user
            ↘ [Full async safety audit / trace collection] → data flywheel → improve
   Throughout: 4D monitoring (infra / app / quality / business)

The Answer Checklist

1. Estimate first: QPS → token throughput → GPU count → cost. Show you know "an LLM burns GPUs."
2. Per-modality pipeline: batch ASR, streaming TTS, native VLM injection.
3. Name the resource contention: the key native-multimodal pain point + three fixes.
4. Two-layer routing: semantic + KV-cache-aware inference routing (not round-robin).
5. Close with dual-track streaming safety + 4D monitoring + the data flywheel.

Narrate the full line — "estimate → pipeline → contention → routing → safety → monitoring → flywheel" — and you've answered a top-tier AI infrastructure question.