How Things Work

Model Serving & Inference

Running models efficiently in production — quantization to fit memory, batching for throughput, the latency-vs-throughput trade-off, and the serving engines that optimize it all.

How It Works

Serving a model is the ongoing cost that scales with usage, governed by GPU memory and latency budgets. Quantization shrinks models (often onto a single GPU) for a small quality hit; batching maximizes GPU utilization and throughput. Modern serving engines (vLLM, TGI, TensorRT-LLM) combine continuous batching, KV-cache management, and speculative decoding to meet latency SLOs at the lowest cost per request.

Training vs Serving

Training a model is a one-time (expensive) job; serving it is the ongoing cost that scales with users. Inference has different constraints — tight latency budgets, high request volume, and GPU memory limits — so it's a distinct engineering discipline focused on doing more with less hardware.

Quantization

Models are stored as numbers; quantization reduces their precision (FP32 → FP16 → INT8 → INT4). This shrinks memory footprint and speeds up computation, usually for only a small drop in quality. It's frequently what lets a large model fit on a single GPU instead of several — a huge cost difference.

Batching

GPUs are massively parallel, so serving one request at a time wastes them. Batching processes many requests together in one pass, dramatically increasing throughput (requests/second) at the cost of a little added latency. Continuous (in-flight) batching, used by vLLM and TGI, merges new requests into running batches for near-optimal utilization.

Latency, Throughput & the Budget

Two metrics drive everything: latency (what a user feels — time-to-first-token plus per-token time) and throughput (what you pay for — requests served per second). Optimizations like KV-cache management, PagedAttention, and speculative decoding push both. The goal is to meet your latency SLO and quality bar at the lowest cost per request.

Key Concepts

📉Quantization

Lowering weight precision (FP16/INT8/INT4) to cut memory and latency for a small quality trade-off.

📦Batching

Serving many requests in one GPU pass to maximize throughput; continuous batching merges them in-flight.

🗃️KV Cache

Caching attention keys/values for generated tokens so each new token is cheap — but it grows with context length.

⚖️Latency vs Throughput

Per-request speed (user-facing) vs requests/sec (cost). Batching trades a little latency for lots of throughput.

⚙️Serving Engines

vLLM, TGI, TensorRT-LLM, ONNX Runtime — runtimes that implement these optimizations for you.

🎲Speculative Decoding

A small model drafts tokens that the large model verifies in bulk, cutting latency.

Inference optimization levers

tsx

1# Inference cost is dominated by GPU memory + how fully you use the GPU.
2
3# Quantization: shrink weights → less memory, faster, slight quality loss.
4#   FP32 (28GB) → FP16 (14GB) → INT8 (7GB) → INT4 (3.5GB)  for a 7B model
5# Often the difference between fitting on ONE GPU or needing several.
6
7# Serving stacks (vLLM, TGI, TensorRT-LLM) optimize throughput with:
8#   - Continuous batching: merge incoming requests into in-flight batches
9#   - PagedAttention / KV-cache management: pack more concurrent sequences
10#   - Speculative decoding: a small model drafts tokens a big model verifies
11
12# Mental model of the two metrics you tune:
13latency    = time_to_first_token + tokens * time_per_token   # what a user feels
14throughput = requests_served_per_second                       # what you pay for
15
16# Export to a portable runtime when deploying outside Python:
17#   torch.onnx.export(model, ...)  → run with ONNX Runtime / TensorRT

💡

Why This Matters

Inference, not training, is where most AI systems spend their money and where users feel performance. Understanding quantization, batching, the latency/throughput trade-off, and serving engines is core to deploying models cost-effectively — and a practical, increasingly common topic for ML-platform and AI-engineering interviews.

Common Pitfalls

⚠Optimizing total latency while ignoring time-to-first-token and streaming — UIs feel frozen.

⚠Over-quantizing (e.g. INT4) on quality-sensitive tasks, degrading outputs to save a little memory.

⚠Serving one request at a time, leaving the GPU massively underutilized.

⚠Forgetting the KV cache grows with context length — long prompts can blow GPU memory at high concurrency.

⚠Rolling your own serving loop instead of using vLLM/TGI/TensorRT-LLM, which already implement the hard optimizations.

Real-World Use Cases

1Fitting a Big Model on a Small Budget

Scenario

A team wants to self-host a 7B model but their FP32 footprint (~28GB) won't fit on the single 24GB GPU they can afford, and renting multi-GPU instances blows the budget.

Problem

Full-precision weights exceed available GPU memory, forcing expensive multi-GPU hardware. The naive serving setup also handles one request at a time, wasting the GPU and inflating cost-per-request.

Solution

Quantize the model to INT8 (~7GB) so it comfortably fits on the 24GB GPU with room for the KV cache, and serve it with vLLM's continuous batching to maximize utilization. Latency stays within budget, throughput jumps, and quality drops only slightly — all on one affordable GPU.

💡

Takeaway: Quantization plus batching are the two biggest levers for cheap inference. They often turn 'needs several GPUs, one request at a time' into 'one GPU, high throughput' with negligible quality loss.

2Slow First Token in a Chat UI

Scenario

A chat product feels sluggish: users wait seconds before any text appears, even though total generation time is fine. Engagement drops because the UI seems frozen.

Problem

The team optimized total latency but ignored time-to-first-token and didn't stream output. Users perceive the long initial wait as the app being broken, regardless of overall speed.

Solution

Stream tokens as they're generated so text appears immediately, and reduce time-to-first-token with a warmed-up model, prompt caching, and right-sized batching. Perceived latency drops sharply even though total compute is similar.

💡

Takeaway: For interactive UIs, perceived latency (time-to-first-token + streaming) matters more than total time. Stream output and optimize the first token — users forgive a steady stream far more than a frozen screen.

Join the discussion

Comments

Loading comments...

Cookie preferences

We use essential cookies for sign-in and preferences. With your permission, we also use basic analytics to improve lessons.