ML System Design Interview
End-to-end ML pipeline design coaching for staff+ engineers. Covers the full arc from problem definition through production monitoring -- the scope expected at L6+ interviews at top-tier ML organizations.
This skill assumes 15+ years of ML/CV/AI/NLP experience. It does not teach fundamentals. It structures the knowledge you already have into the format interviewers reward.
When to Use
Use for:
- Practicing 45-minute ML system design rounds
- Structuring whiteboard presentations for recommendation, ranking, RAG, fraud, perception systems
- Analyzing serving architecture tradeoffs (batch vs online vs streaming)
- Identifying L6+ differentiation signals (problem ownership, org constraints, data flywheels)
- Reviewing and critiquing ML system design answers
NOT for:
- Coding interviews (use
senior-coding-interview) - Behavioral / leadership questions (use
interview-loop-strategist) - ML theory or math derivations
- Implementing models or writing training code
- Paper reading or research review
The 7-Stage Design Framework
Every ML system design answer follows this arc. The stages are sequential but you will loop back as constraints emerge. The Mermaid diagram below is your whiteboard skeleton.
flowchart TD
R[1. Requirements\n- Business goal\n- Users and scale\n- Latency/throughput SLA\n- Constraints] --> M[2. Metrics\n- Offline: precision, recall, NDCG\n- Online: CTR, conversion, revenue\n- Guardrails: latency p99, fairness]
M --> D[3. Data\n- Sources and collection\n- Labeling strategy\n- Pipeline: ETL, validation\n- Freshness and staleness]
D --> F[4. Features\n- Engineering and transforms\n- Feature store architecture\n- Online vs offline features\n- Freshness requirements]
F --> Mo[5. Model\n- Architecture selection\n- Training pipeline\n- Iteration strategy\n- Baseline and ablation]
Mo --> S[6. Serving\n- Batch vs online vs streaming\n- Caching and precomputation\n- Scaling and cost\n- Canary and shadow mode]
S --> Mon[7. Monitoring\n- Data drift detection\n- Model degradation alerts\n- A/B testing framework\n- Rollback strategy\n- Feedback loops]
Mon -.->|Feedback loop| D
Mon -.->|Retrain trigger| Mo
Stage Details
Stage 1 -- Requirements (5 minutes) Ask clarifying questions before designing anything. Establish: Who is the user? What is the business metric? What is the latency SLA? What scale (QPS, data volume)? What are hard constraints (cost, privacy, regulation)? An L6+ candidate owns the problem definition -- do not wait for the interviewer to hand you requirements.
Stage 2 -- Metrics (3 minutes) Define offline metrics that you can measure before deployment AND online metrics that matter to the business. Explain the gap: "NDCG improvement offline does not always translate to CTR lift online because of position bias and novelty effects." Define guardrail metrics: latency p99, fairness across user segments, cost per prediction.
Stage 3 -- Data (7 minutes) Where does training data come from? How is it labeled (human, weak supervision, implicit signals)? What is the class balance? How fresh does data need to be? What is the data pipeline (batch ETL vs streaming)? What data quality checks exist? This stage separates L6+ candidates from L5 -- junior candidates assume clean labeled data.
Stage 4 -- Features (5 minutes) What features does the model need? Which are precomputed (offline) vs computed at request time (online)? Feature store architecture: online store (low-latency lookups) vs offline store (batch training). Feature freshness: user features update daily, item features update hourly, contextual features are real-time.
Stage 5 -- Model (8 minutes) Start with a simple baseline (logistic regression, XGBoost) and explain why. Then propose the production architecture (two-tower, transformer, etc.) and justify the upgrade. Discuss training pipeline: how often, how much data, how to handle distribution shift. Iteration strategy: what experiments to run first.
Stage 6 -- Serving (8 minutes) This is where system design and ML intersect. Discuss: inference latency requirements, batch precomputation vs online inference, GPU/CPU tradeoffs, model serving framework, caching strategy, cost optimization (quantization, distillation, spot instances). Draw the serving architecture.
Stage 7 -- Monitoring (5 minutes) What happens after deployment? Data drift detection (PSI, KL divergence). Model degradation alerts (metric decay over time). A/B testing framework (sample size, duration, novelty effects). Rollback strategy (shadow mode, canary percentage). Feedback loops that improve the model over time.
45-Minute Time Budget
| Phase | Minutes | What to Cover |
|---|---|---|
| Requirements + Clarification | 5 | Business goal, users, scale, SLA, constraints |
| Metrics | 3 | Offline, online, guardrails, metric alignment |
| Data | 7 | Sources, labeling, pipeline, quality, freshness |
| Features | 5 | Engineering, store architecture, online/offline split |
| Model | 8 | Baseline, production arch, training, iteration |
| Serving | 8 | Latency, architecture, cost, deployment strategy |
| Monitoring | 5 | Drift, alerts, A/B testing, rollback, feedback |
| Q&A Buffer | 4 | Interviewer deep-dives, defend tradeoffs |
If the interviewer cuts in with questions, adapt -- but cover all 7 stages even briefly. Skipping monitoring is the most common L5 mistake.
Canonical Problem Set
| Problem | Key Challenges | Must-Discuss |
|---|---|---|
| Recommendation System | Cold start, position bias, multi-objective optimization | Two-tower retrieval + reranking, exploration-exploitation |
| Search Ranking | Query intent classification, relevance vs engagement, latency at scale | Inverted index + embedding retrieval, L1/L2 ranking cascade |
| Content Moderation | Multi-modal (text+image+video), adversarial evasion, precision-recall tradeoff | Human-in-the-loop, escalation tiers, appeal workflow |
| RAG Pipeline | Retrieval quality, chunk strategy, hallucination detection, evaluation | Embedding model selection, hybrid search, reranking, citation |
| Fraud Detection | Extreme class imbalance, adversarial adaptation, real-time requirement | Feature velocity, graph features, ensemble + rules, feedback delay |
| Autonomous Driving Perception | Sensor fusion, safety-critical latency, long-tail distribution | Multi-task architecture, simulation, OTA updates, regulatory |
Serving Architecture Comparison
| Pattern | Latency | Freshness | Cost | Best For |
|---|---|---|---|---|
| Batch prediction | N/A (precomputed) | Hours-stale | Low compute, high storage | Email recommendations, daily reports |
| Online inference | 10-500ms | Real-time | High compute (GPU) | Search ranking, fraud detection |
| Near-real-time | 1-60s | Minutes-fresh | Medium | Feed ranking, content moderation |
| Streaming | Sub-second | Continuous | High (always-on) | Fraud, anomaly detection, bidding |
Detailed serving tradeoffs, framework comparisons, and cost optimization strategies are in references/serving-tradeoffs.md.
L6+ Differentiation Signals
What separates a staff+ answer from a senior answer:
1. Own the Problem Definition Do not accept the problem as stated. Ask: "What business metric are we optimizing? Is this a revenue problem or an engagement problem? What is the current solution and why is it insufficient?" L5 candidates accept "build a recommendation system." L6+ candidates ask "what are we recommending, to whom, and what does success look like?"
2. Discuss Organizational Constraints Real systems live inside organizations. Address: team size (can we maintain a custom model or should we use a managed service?), on-call burden, cross-team data dependencies, compliance requirements, migration path from legacy system.
3. Data Flywheel Strategy Show that you think about the virtuous cycle: better model -> more engagement -> more data -> better model. Discuss how to accelerate it: active learning, implicit feedback loops, exploration strategies, cold-start bootstrapping.
4. Build vs Buy Decisions Not everything should be custom. Argue for managed services where appropriate (embedding APIs, feature stores, serving platforms) and custom solutions where competitive advantage demands it. Show you understand the total cost of ownership.
5. Multi-Objective Thinking Real systems optimize multiple objectives simultaneously: relevance AND diversity, accuracy AND fairness, quality AND latency. Discuss how to handle conflicts: Pareto optimization, constrained optimization, multi-task learning, business-rule post-processing.
Whiteboard Strategy
What to draw and when:
| Time | Draw This | Purpose |
|---|---|---|
| 0-5 min | Requirements box with bullet points | Anchor the discussion, show structured thinking |
| 5-8 min | Metric table (offline vs online) | Demonstrate you think beyond model accuracy |
| 8-15 min | Data pipeline diagram (sources -> ETL -> store) | Show you understand data engineering |
| 15-20 min | Feature architecture (offline store + online store) | Demonstrate feature store knowledge |
| 20-28 min | Model architecture + serving diagram | The core system design artifact |
| 28-36 min | Full system diagram with latency annotations | Connect everything, show you can ship |
| 36-41 min | Monitoring dashboard sketch + feedback arrows | Close the loop, show production thinking |
Use boxes for components, arrows for data flow, and annotate with latency/throughput numbers. The diagram should be readable by someone who walks in at minute 30.
Anti-Patterns
Model-First Thinking
Novice: Jumps to "I would use a transformer" or "Let me describe the attention mechanism" in the first 2 minutes, before understanding the problem, defining metrics, or discussing data. Spends 70% of time on model architecture and 0% on serving.
Expert: Spends the first 10 minutes on requirements, metrics, and data before mentioning any model. Names a simple baseline first (logistic regression on handcrafted features), then argues for complexity only when the baseline's limitations are clear. Allocates equal time to serving and monitoring.
Detection: Architecture diagram has a detailed model box but no data pipeline, no feature store, no serving layer, and no monitoring component. Mentions model architecture in the first sentence.
Ignoring the Data
Novice: Assumes clean, labeled data exists at scale. Says "we would train on millions of labeled examples" without discussing where labels come from, how much they cost, what the class distribution looks like, or how stale the data gets.
Expert: Asks about data sources, labeling strategy (human vs weak supervision vs implicit signals), class imbalance handling, data freshness SLA, and data quality monitoring. Discusses the cost of labeling and proposes strategies to reduce it (active learning, semi-supervised methods, synthetic data).
Detection: No discussion of data collection, labeling costs, class imbalance, data quality checks, or data freshness anywhere in the answer. The word "label" does not appear.
No Monitoring Story
Novice: Design ends at the serving layer. No mention of what happens after the model is deployed. Does not discuss how to detect degradation, how to roll back, or how to improve the model over time.
Expert: Discusses data drift detection (population stability index, feature distribution monitoring), model performance decay alerts, A/B testing framework with proper statistical rigor, canary deployment strategy, shadow mode for safe rollouts, and explicit feedback loops that flow data back into retraining.
Detection: Architecture diagram has no monitoring component. No feedback arrows from production back to training. No mention of A/B testing, canary deployment, or rollback.
Reference Files
Consult these for deep dives -- they are NOT loaded by default:
| File | Consult When |
|---|---|
references/ml-design-templates.md | Working through a specific problem (recommendation, search, RAG, fraud, content mod, perception). Contains 6 fully worked designs with Mermaid diagrams. |
references/serving-tradeoffs.md | Deep-diving on serving architecture, framework selection, caching, cost optimization, deployment strategies. Contains framework comparisons and latency targets by use case. |
references/evaluation-metrics-guide.md | Choosing metrics, understanding metric alignment, designing A/B tests, evaluating generative AI. Contains metric decision trees and formulas. |