Part 0 · Foundations & Motivation¶
Motivation-first. Before any optimization, we build the one mental model everything else hangs on: why LLM inference is memory-bound. Once that clicks, every later trick — KV cache tuning, quantization, PagedAttention, continuous batching — becomes a conclusion you can derive, not a fact to memorize.
What this part covers¶
- Why LLM inference is memory-bound (the throughput story starts here)
- The two phases: prefill vs decode, and which optimizations act on which
- KV cache — what it is, why it exists, how it grows, and why it is the core tension behind the throughput ceiling
- Inference metrics: TTFT, TPOT/ITL, throughput, goodput — and how to measure them
- Number formats: FP16 / BF16 / FP8 / INT8 / INT4 — enough to enter the quantization part unblocked
Lessons¶
- Inference Flow: Prefill & Decode — the two phases of autoregressive generation, and why prefill is compute-bound while decode is memory-bound.
- Transformer, the Infra View — read a decoder block as a cost model: which parts cost weights, prefill FLOPs, and KV cache.
- KV Cache — what it is, why it grows, and why it is the core tension behind the throughput ceiling.
- GPU Hardware Mental Model — the memory pyramid (HBM vs SRAM), the SM/warp execution model, and the roofline that proves decode is bandwidth-bound.
- Inference Performance Metrics — TTFT, TPOT/ITL, throughput, and goodput, plus how to measure each with
vllm bench serveand Prometheus. - Number Formats: FP16 · BF16 · FP8 · INT8 · INT4 — the range-vs-precision trade behind every dtype, paving the way into quantization.
Part 0 complete
Part 0A — inference flow, the Transformer infra view, KV cache — and Part 0B — GPU hardware, metrics, number formats — are written, each with a two-way-linked interview question. All vLLM flags/APIs are verified via Context7 (ADR-0004); the baseline is vLLM 0.26.0. See the Interview Bank for the linked question set.