Interview Bank

A curated bank of high-frequency interview questions, organized by module (Part 0, 2–8). Each question follows one schema: direct answer → deep dive → code (if applicable) → interviewer follow-up → linked concept.

Every question links back to the lesson it tests, and every lesson's "Interview links" section links here — a closed learn-and-practice loop.

By module

• Part 0 · Foundations
  • Prefill vs decode — which phase is compute- vs memory-bound, and why.
  • Attention variants: MHA/MQA/GQA — how KV heads set the KV cache and the throughput ceiling.
  • KV cache & throughput ceiling — why the KV cache, not compute, is usually the bottleneck.
  • GPU memory hierarchy & roofline — walk the memory tiers and use the roofline to explain why decode is memory-bound.
  • Latency vs throughput metrics — TTFT/TPOT/ITL/throughput/goodput, how to measure, and the batch-size trade.
  • Number formats & precision — FP16/BF16/FP8/INT8/INT4, range vs precision, and why low-bit speeds up decode.
• Part 2 · Single-GPU Inference Performance
  • Arithmetic intensity of GEMM & attention — derive an operator's intensity from its shapes, why decode attention is context-independent, and the batch that crosses the ridge.
  • VRAM budget & max concurrency — walk the full VRAM budget and size max concurrency; the knobs that hit a concurrency target.
  • FlashAttention & IO-aware attention — why it's faster at the same FLOPs, online softmax, and where it does/doesn't help.
  • CUDA graphs & kernel fusion — decode launch overhead, why it hits decode not prefill, and what enforce_eager trades.
• Part 3 · GPU Programming (Triton)
  • CUDA execution model: warps, SIMT & occupancy — what a warp is, the cost of SIMT divergence, and why maxing occupancy isn't always faster.
  • Memory coalescing, shared memory & bank conflicts — what makes an access coalesced, what uncoalesced costs, and what shared memory and bank conflicts are.
  • Triton programming model — what a Triton program maps to, program_id/offsets/masks, FP32 accumulation, and when to reach for Triton.
  • PagedAttention kernel & block tables — why KV lives in blocks, what a block table does, how the kernel gathers KV, and why it equals dense attention.
• Part 4 · Quantization
  • Quantization: why it speeds up inference — why quantization raises throughput (memory, not compute), the affine map, and what bounds the error.
  • Quantization schemes: granularity, symmetry, PTQ/QAT — per-tensor/channel/group, symmetric vs asymmetric, W4A16 vs W8A8, and why inference uses PTQ.
  • Quantization methods: GPTQ/AWQ/SmoothQuant/FP8 — place each method on the axes, its anti-outlier trick, and which to pick for a bottleneck.
  • Quantizing & serving in practice — quantize → serve → validate: the tool, the settings, and what to measure.
• Part 5 · Serving & Throughput (vLLM Core)
  • Static vs continuous batching — why static batching wastes the GPU, what iteration-level scheduling means, and what actually limits the batch size.
  • PagedAttention: block manager & fragmentation — why contiguous KV fragments, what the block manager does, how num_gpu_blocks is set, and how paging becomes throughput.
  • Preemption: recompute vs swap — what vLLM does when the KV pool is exhausted, why V1 defaults to recompute (and dropped swap), and the capacity knobs that stop it.
  • Chunked prefill & PD disaggregation — why a long prefill stalls decode, what chunked prefill trades, the max_num_batched_tokens dial, and when to disaggregate.
  • Prefix caching — how block hashing makes reuse safe, why only full blocks cache, when it helps, and why outputs are unchanged.
  • Speculative decoding — guess-and-verify, why it's free only because decode is memory-bound, what sets the speedup, and when it backfires.
  • Trace a request through vLLM's architecture — the V1 components (API server / engine core / worker), an end-to-end trace, and which optimization lives in which box.
  • Tuning knobs: which one for which SLO — which knob moves which end of the throughput/latency curve, its trade, and the sweep to run.
  • Sampling parameters: temperature, top-p/top-k & throughput — the sampling knobs and greedy decoding, why they barely move throughput, and why fixed sampling makes an eval reproducible.
• Part 6 · Advanced Inference Topics
  • Multi-LoRA serving: one base, many adapters — why a LoRA adapter is tiny, how vLLM batches heterogeneous adapters via grouped GEMM, and the knobs (max_lora_rank, max_loras, dynamic loading) that cap how many you can co-serve.
  • Guided / structured decoding — how a schema becomes a per-step logit mask, why the guarantee is hard rather than statistical, its cost, and why it fixes shape but never truth.
  • Long-context inference: positions, sinks & the KV wall — why models break past training length and how RoPE scaling (PI/NTK/YaRN) fixes it, what the attention sink is, and why the KV cache — not compute — is the long-context ceiling.
• Part 7 · Multi-GPU & Distributed
  • Parallelism: TP/PP/DP/EP & when to use each — the two reasons to parallelize, what each of TP/PP/DP/EP splits and costs to communicate, why TP stays within a node while PP crosses them, and how to pick a strategy from model size and topology.
  • MoE inference: active vs total params & expert routing — active-vs-total params and FLOPs, how the router picks experts per token, and why EP (not TP) is the multi-GPU answer for the experts' memory.
  • NCCL collectives & launching TP/PP — what all-reduce / all-gather / reduce-scatter each move, why ring all-reduce is ~2× the message independent of GPU count, which collective TP uses and how often, and how vLLM launches TP/PP single- vs multi-node (mp vs ray) — including debugging an init hang.
• Part 8 · Production & System Design
  • Serving over HTTP: the OpenAI-compatible server & its endpoints — what vllm serve exposes, /v1/chat/completions vs /v1/completions, what /health does and doesn't promise, how auth works, and interface vs capacity flags.
  • Load-testing & the concurrency knee (Little's Law) — what the knee is and why the curve bends there, open- vs closed-loop load, how Little's Law explains the runaway past it, and why you report goodput (not raw throughput).
  • Routing, autoscaling & KV-aware routing — why prefix-aware routing beats round-robin (per-instance caches), why you autoscale on num_requests_waiting rather than GPU utilization, and how cold-start and drain shape a safe policy.
  • Observability & profiling: metrics, traces & the kernel timeline — the three zoom levels (metrics → traces → profiles), which vLLM metrics you alert on, the prefill/decode split, and capturing a torch/Nsight profile without drowning in data.
  • SLO-driven tuning: goodput, the binding constraint & the loop — why you optimize goodput against an SLO, reading the binding constraint (queue/prefill/decode/KV) from metrics, which knob relieves which, and the one-knob-at-a-time loop.
  • The serving ecosystem: choosing vLLM vs TensorRT-LLM / TGI / SGLang / LMDeploy — the shared baseline vs the divergence axes, a defensible default with exceptions, and deciding by benchmarking OpenAI-compatibly on your own workload at your SLO.
  • System design: sizing & designing an inference service — long-form drills: the framework (clarify → napkin math → architecture → bottlenecks → trade-offs) and several complete worked designs (a chat API for X QPS at Y latency, a multi-tenant LoRA platform, long-context RAG).

How to use this bank

Each question is two-way-linked to the lesson it tests — read the lesson, then drill the question, or hit a question cold and follow the link back when you need the mechanism. This is a curated high-frequency set, not an exhaustive one; difficulty tiers, frequency tags, and weighting are intentionally out of scope.