Serving LLMs: KV Cache, Continuous Batching, vLLM & TTFT

Think about a busy dosa counter. One tawa, many orders. The slow way is to take five orders, cook all five, hand all five over, then take the next five — the tawa sits half empty while the last dosa finishes. The fast way is to start a new one the moment a space frees up.

That is almost exactly the difference between naive and modern LLM serving, and it is one of two ideas that explain nearly everything about performance. The other is where the memory goes.

Two phases, two bottlenecks

Every request has two distinct stages, and confusing them is why people optimise the wrong thing.

Long prompt, slow start. Long answer, slow stream. If users say "it takes ages to start replying", that is a prefill and queueing problem — shrinking the input and caching the prefix will help, and a faster model won't.

The KV cache

Generating token 500 requires attending to the previous 499. Redoing that work every time would be hopeless, so the model caches the attention keys and values for tokens already processed. Each new token then only computes against the cache.

This is why generation is fast at all. It is also the reason concurrency is limited, because the cache lives in GPU memory and grows with sequence length × concurrent requests.

Weights set the floor on memory; the KV cache sets the ceiling on how many users fit. A 13GB model on a 24GB card leaves roughly 9–10GB for cache and overhead — and that remainder decides your concurrency, not the model size.

💡 Symptom worth recognising: GPU utilisation looks high but throughput is poor. That is usually cache pressure forcing small effective batches or preemption — the fix is capping sequence length, quantizing to free memory, or a serving stack with a paged cache, not necessarily a bigger GPU.

Continuous batching

Back to the dosa counter. With static batching, the server waits for N requests, runs them together, and the whole batch finishes at the pace of its longest generation — every finished slot idles until then. Since generation lengths vary wildly, the GPU spends much of its time doing nothing.

Continuous (in-flight) batching adds a new request to the running batch the instant a slot frees. Nothing waits for the slowest member.

This is the single largest throughput improvement in modern serving, and it is the main reason to use a purpose-built stack — vLLM, TGI, TensorRT-LLM — rather than a naive inference loop. They also implement paged attention, which allocates the KV cache in blocks rather than one contiguous reservation per request, drastically reducing wasted memory and raising concurrency further.

# vLLM: continuous batching + paged KV cache, OpenAI-compatible API
python -m vllm.entrypoints.openai.api_server \
  --model /models/support-13b-int8 \
  --quantization awq \
  --max-model-len 8192 \           # caps KV cache per request
  --gpu-memory-utilization 0.90 \  # leave headroom, don't take 100%
  --enable-lora \                  # serve adapters on one base
  --lora-modules support=/adapters/support legal=/adapters/legal

The latency numbers that matter

For an interactive product, TTFT is the number users judge. For a batch job with nobody waiting, TTFT is irrelevant and throughput per rupee is everything. Applying interactive instincts to batch work is a common and expensive mistake.

Serving many LoRA variants

The serving payoff from lesson 4: because the base is frozen, one copy of it in GPU memory can serve many adapters, swapped per request.

Ten fine-tuned variants — one per enterprise customer, or one per task — cost roughly one model's memory instead of ten. That is what makes per-customer customisation commercially viable at all.

Two operational notes: an adapter is bound to the exact base version it was trained on, so base upgrades mean re-training every adapter; and merged adapters lose the swapping benefit in exchange for zero runtime overhead.

Wait — should I self-host at all?

Most teams shouldn't, and the comparison usually gets done badly because people count GPU hours against API tokens and stop there.

The cost nobody puts in the spreadsheet: a rented GPU bills 24/7 whether traffic does or not, and someone has to own upgrades, incidents and capacity planning. At low utilisation, self-hosting is often more expensive and more work.

🎯 Selection-round radar: "How would you deploy an LLM in production?" wants four beats: a serving stack with continuous batching and a paged KV cache → quantization chosen against your own eval set, sized for weights plus cache headroom → TTFT and p95 as the latency targets, with streaming → and a hosted-versus-self decision driven by volume, residency and ops capacity rather than by GPU price alone. Naming the KV cache as the concurrency limit is the detail that marks you out.

A practical stack

If you do self-host, the shape most teams converge on:

• vLLM (or TGI / TensorRT-LLM) for continuous batching, paged attention and an OpenAI-compatible API — so client code doesn't care where the model runs.
• int8 by default, int4 if your eval set says quality holds and you need the concurrency.
• A cap on max sequence length, because unbounded contexts are how one request starves the rest.
• Autoscaling sized for p95, not peak, with a queue and a sensible timeout.
• An abstraction layer in your application over the model call, so switching between hosted and self-hosted is a config change. This costs a day and buys all your optionality.

Common mistakes

• Sizing a GPU from weights alone, ignoring the KV cache.
• A naive inference loop with no continuous batching.
• Optimising total latency while TTFT stays at three seconds.
• Self-hosting at low volume and paying for idle GPUs.
• Setting GPU memory utilisation to 100% and leaving no headroom.
• No cap on sequence length, so one long request starves the rest.
• Hard-coding a provider's SDK throughout the application.
• Forgetting that LoRA adapters are pinned to a base version.

Quick recap

Practice Zone — PYQs from real selection rounds

Six MCQs, then two tasks: a hosted-versus-self-host decision with a residency constraint, and halving time-to-first-token.

Q1Servingeasy

What is the KV cache and why does it matter for serving?

Asked in

aA cache of previous users' answersbStored attention key/value tensors for tokens already processed, so each new token doesn't recompute the whole sequence — it makes generation fast but consumes GPU memory that grows with sequence length and concurrencycThe model's weights in memorydA disk cache for the tokenizer

Q2Servingmedium

What does continuous (in-flight) batching do?

Asked in

aWaits for a fixed number of requests before startingbAdds new requests to the running batch as earlier ones finish, instead of waiting for the whole batch to complete — dramatically improving GPU utilisation and throughputcBatches the training datadSends requests one at a time

Q3Servingmedium

Which two latency numbers actually matter for a chat product?

Asked in

aTraining time and epoch countbTime-to-first-token (how long before text starts appearing) and inter-token latency (how fast it streams) — total time matters less when the response is streamedcDisk read speed onlydModel file size

Q4Servinghard

How does LoRA change serving economics?

Asked in

aIt doesn't affect serving at allbMany adapters can share one base model in GPU memory and be swapped per request, so ten fine-tuned variants cost roughly one model's memory instead of tencIt requires one GPU per adapterdIt makes inference slower by 10x

Q5Servingmedium

When is a hosted API the right choice over self-hosting?

Asked in

aAlwaysbMost of the time at low or moderate volume — no ops burden, automatic model upgrades, elastic scaling; self-hosting wins on very high steady volume, data residency, or serving an open fine-tuned modelcNever — self-hosting is always cheaperdOnly for prototypes

Q6Servinghard

Under heavy load your GPU utilisation is high but throughput is poor. A likely cause is:

Asked in

aThe model is too accuratebKV-cache pressure — long sequences and high concurrency are exhausting memory, forcing preemption or small effective batch sizescToo few usersdThe tokenizer is slow

Hands-on tasks:

FAQ

Next lesson: keeping it healthy once real traffic arrives — Lesson 9: LLMOps & Monitoring →