Skip to content

Part 8 · Production & System Design

From a working engine to a production service — and to answering the system-design interview.

What this part covers

  • Serving over HTTP: the OpenAI-compatible server, load-testing to find the concurrency knee, and routing / autoscaling / KV-cache-aware routing across multiple instances
  • Observability & profiling and SLO tuning; framework trade-offs — TensorRT-LLM / TGI / SGLang / LMDeploy — for selection questions
  • Capacity planning: estimate VRAM / throughput given a model + hardware
  • System design drills: "design an inference service for X QPS at Y latency"

The Capstone pulls everything here together on a single 4090.

Lessons

  • Serving vLLM over HTTP: the OpenAI-Compatible Server — vllm serve wraps the engine core in a thin FastAPI frontend that speaks the OpenAI API, so any OpenAI client retargets with one base_url line. The endpoints (/v1/chat/completions applies the chat template, /v1/completions is raw, /v1/models lists the served id + LoRA adapters, /health is liveness 200/503, /metrics is the Prometheus feed), auth (--api-key / VLLM_API_KEY, repeatable for rotation), streaming (SSE), and why the interface knobs (--port / --served-model-name) are separate from the capacity knobs (--max-num-seqs / --gpu-memory-utilization) that set the ceiling — verified on vLLM 0.26.0.
  • Load-Testing to Find the Concurrency Knee — the knee is where a single instance's batch fills and vllm:num_requests_waiting climbs off zero; by Little's Law (\(L=\lambda W\)), pushing arrival rate past the max completion rate makes the queue and latency run away. You find it by sweeping vllm bench serve --request-rate upward (open-loop Poisson arrivals — not --request-rate inf and not closed-loop --max-concurrency), reading p99 TTFT/E2EL and goodput against your SLO, and reporting the last passing rate as the instance's honest capacity.
  • Routing, Autoscaling & KV-Aware Routing (Multi-Instance) — past one instance's knee you scale to N independent replicas behind a router; the two decisions that make it good are KV-cache-aware (prefix-aware) routing (caches are per-instance, so round-robin re-prefills shared prompts) and autoscaling on the queue (vllm:num_requests_waiting, not GPU utilization), with cold-start lag and drain-before-scale-down handled. vLLM ships this as the production stack (Helm: prefix-aware + model-aware router, engine pods, LMCache KV offload); SkyPilot autoscales on target_qps_per_replica.
  • Observability & Profiling: Metrics, Traces, and the Kernel Timeline — production debugging in three zoom levels: metrics (Prometheus /metrics — num_requests_waiting, gpu_cache_usage_perc, the TTFT / prefill-time / decode-time histograms, request_success_total{finished_reason}; Grafana dashboard included) to detect; OpenTelemetry traces (spans → Jaeger over OTLP) to localize a stage; and the PyTorch profiler (--profiler-config + /start_profile / /stop_profile) or Nsight Systems (nsys … --capture-range=cudaProfilerApi) to explain it at the kernel — cheapest-tier-first, profiling as an on-demand scalpel.
  • SLO-Driven Tuning: From Metrics to a Tuning Loop — tuning anchored on the SLO: score every config by goodput (throughput that meets the SLO), read the binding constraint from /metrics (queue → add replicas; prefill → --max-num-batched-tokens / prefix caching; decode → --max-num-seqs / quantization / speculative decoding; KV → --gpu-memory-utilization / --max-model-len), turn the one matching knob, re-measure, and stop at the plateau — one knob at a time, against a production-like workload.
  • The Serving Ecosystem: vLLM vs TensorRT-LLM / TGI / SGLang / LMDeploy — no global #1: a shared baseline (continuous batching, paged KV, prefix caching, OpenAI-compatible APIs) and a few divergence axes — vLLM for breadth/velocity/hardware-flexibility (the default), TensorRT-LLM for peak NVIDIA latency via an ahead-of-time engine, TGI for HuggingFace-native production, SGLang for shared-prefix/structured/agentic (RadixAttention), LMDeploy for TurboMind + weight-only quant. Answer selection as "default X, switch to Y when constraint Z," and settle it by benchmarking OpenAI-compatibly on your own workload at your SLO.
  • Capacity Planning: From One GPU's Throughput to a Fleet — the napkin math every system-design interview opens with: a single instance's capacity is min(what fits, how fast) — the VRAM gate (concurrency) and the speed gate (decode's bandwidth-set TPOT floor and its measured knee throughput). Turn the knee into requests/s (\(r_{\text{inst}} = T_{\text{out}}/\bar{o}\)), then size the fleet \(N_{\text{inst}}=\lceil \lambda_{\text{peak}}/(\rho\,r_{\text{inst}})\rceil\), \(N_{\text{GPU}}=N_{\text{inst}}\times\text{TP}\) — at peak with headroom, gated first by whether the SLO clears the TPOT floor. The bridge into the system-design long questions.

Part 8 complete

All seven lessons are in — the OpenAI-compatible server, load-testing the knee, and routing / autoscaling; observability & profiling, SLO-driven tuning, and the framework comparison; and capacity planning + system design — each two-way-linked to its interview question, and the last one to a set of system-design long questions. All vLLM flags/APIs are verified via Context7 (ADR-0004); the baseline is vLLM 0.26.0, and every performance number is an illustrative / order-of-magnitude reference. Next stop: the Capstone. See the Glossary and the Interview Bank.