LLM Cost & Latency Optimisation: Caching, Routing and Batch Tiers

A demo costs nothing. Two hundred users cost something. Two million requests a day is a line item somebody in finance is now asking questions about — and the answer "AI is expensive" is not going to survive that meeting.

The good news is that most LLM bills contain a great deal of waste, and the largest reductions come from changes that take days and cost no quality at all. This last lesson is the ordered list.

Profile before optimising

Same rule as any performance work, ignored just as often. Before changing anything, get cost per request broken down by feature and by pipeline stage — and inside a request, break the input down by component.

Average request: 6,000 input tokens, 400 output tokens

  system prompt + tools     900   fixed, identical every request
  few-shot examples         600   fixed, identical every request
  retrieved passages      3,400   12 passages, all of them
  conversation history      980   full history, resent each turn
  user question             120
  ------------------------------
                          6,000 in  /  400 out

That profile answers the whole question. The bill is input, not output, so output-side tweaks are noise. 1,500 tokens are byte-identical on every request and should be cached. 3,400 tokens are twelve passages where five would do.

Teams routinely spend a sprint optimising a stage that is 5% of the bill. Ten minutes with a profile prevents it.

The free wins

In order of return per unit of effort. The first four cost no quality — several improve it.

1. Prompt caching. Reorder the prompt into a byte-identical stable prefix (system prompt, tool definitions, fixed examples) followed by everything variable. Cached input is billed at a large discount. The classic bug this fixes: a timestamp or username sitting above 900 lines of fixed rules, so nothing after it can ever be reused.

# ✗ nothing is cacheable — byte 1 differs every request
Request time: 2026-09-06T11:42:03Z
User: Priya S.
[900 lines of fixed rules]

# ✓ 1,500 tokens reused on every request, for free
[900 lines of fixed rules]
[600 tokens of fixed examples]
---
Session: Priya S., 2026-09-06T11:42Z
<docs>...</docs>
Question: ...

Nothing was added or removed. Only the order changed.

2. Rerank and truncate retrieval. Twelve passages to five saves roughly 2,000 tokens per request here — and quality usually improves, because the seven dropped passages were mostly distractors burying the useful one.

3. Compress conversation history. A rolling summary of older turns plus the last few verbatim. Saves most of that 980 and reduces drift in long chats.

4. Audit the system prompt. Long prompts accumulate duplicated and contradictory rules. Deleting them cuts tokens and usually improves consistency, since competing rules are why the model follows some and ignores others.

💡 Steps 1–4 commonly halve an input bill in under two weeks with no model change and no training. Do all of them before considering anything below.

Model routing

Most production traffic is much easier than the hardest case your system has to handle. Routing sends the easy majority to a small cheap model and escalates only the rest.

def route(request):
    # 1. Deterministic overrides FIRST — auditable, business-owned
    if any(k in request.text.lower() for k in ESCALATE_KEYWORDS):
        return BIG          # refunds, complaints, legal, fraud
    if request.account.is_high_value:
        return BIG

    # 2. Cheap classifier for everything else
    if classify_complexity(request.text) == "simple":
        return SMALL

    return BIG

# 3. And let the small model escalate itself
result = call(SMALL, request)
if result.low_confidence:
    result = call(BIG, request)

The design point that matters: the two routing errors are not equally bad. Sending a hard request to the cheap model is a customer-visible failure; sending an easy one to the expensive model just costs money. So bias the router toward escalation, and tune the threshold from data.

Roll it out by shadow-running the router first — log what it would have decided for a week while everything still goes to the big model, then compare quality on exactly the requests it would have downgraded. That measures the risk before you take it.

Batch work is a different problem

Nightly classification of two million documents has nobody waiting for it. Applying interactive instincts there is a common and expensive mistake.

Wait — isn't latency the same problem?

Mostly, and the overlap is why this works nicely: shorter inputs and prompt caching cut cost and time-to-first-token together. But they are not identical, and conflating them produces bad plans.

Two things worth remembering from that table. Streaming belongs in a latency plan and never in a cost plan — the same tokens are billed. And the pre-model pipeline is real latency: a 900ms reranker is 900ms of TTFT, so run independent stages in parallel.

🎯 Selection-round radar: "Your LLM feature is too expensive — what do you do?" The weak answer is "use a cheaper model". The strong one: profile first to find where the tokens are → prompt caching and input trimming, which cost nothing and often improve quality → then routing, with a shadow run to measure the risk → batch tier for anything asynchronous → and fine-tuning a smaller model only if all of that misses the target. Attaching a risk to each step is what makes it sound like a plan rather than a list.

What not to do first

• Don't start by switching to a cheaper model. It is the change with the largest quality risk and it is usually unnecessary once the free wins are taken.
• Don't fine-tune a smaller model as step one. Months of work; steps 1–5 usually reach the target first. It is a legitimate step six.
• Don't self-host to save money at low volume. A rented GPU bills 24/7 whether traffic does or not.
• Don't optimise without the eval set running after every step, or a quality regression will be attributable to the whole programme instead of one change.

Common mistakes

• Optimising before profiling, and fixing a 5% stage.
• Putting variable content above a long fixed prefix, killing caching.
• Passing everything retrieval returned with no rerank or truncation.
• Counting streaming as a cost saving.
• Routing without deterministic overrides for business-critical categories.
• Shipping a router with no shadow phase and no kill switch.
• Using the interactive tier for offline batch work.
• No cost-per-request alert, so regressions surface on the invoice.

Quick recap

Practice Zone — PYQs from real selection rounds

Six MCQs, then two design tasks: halve a real bill without hurting quality, and design a routing cascade.

Q1Cost optimisationeasy

Which usually gives the largest cost reduction for the least engineering effort?

Asked in

aFine-tuning a smaller modelbPrompt caching on a stable prefix, plus trimming input — reranking retrieved passages and summarising history — since you pay for every input token on every requestcRewriting the app in RustdBuying more GPUs

Q2Cost optimisationmedium

What is model routing (cascading)?

Asked in

aLoad-balancing across GPUsbSending easy requests to a small cheap model and escalating only hard ones to a large model, so you pay premium prices only where they are neededcRouting requests by geographydChoosing a model at random

Q3Cost optimisationmedium

Why is prompt caching layout-sensitive?

Asked in

aIt caches by request idbIt matches a prefix, so one changed character early invalidates everything after it — the stable content must come first, byte-identicalcIt only caches output tokensdIt requires a special model

Q4Cost optimisationhard

Which of these reduces latency but NOT cost?

Asked in

aStreaming the responsebTrimming input tokenscPrompt cachingdRouting to a smaller model

Q5Cost optimisationmedium

For a batch job with no user waiting — nightly classification of 2 million documents — the right optimisation is:

Asked in

aOptimise time-to-first-tokenbUse the provider's batch/asynchronous tier if offered (typically much cheaper for delayed results), maximise throughput, and pick the smallest model that passes your eval setcUse the largest model for safetydStream every response

Q6Cost optimisationhard

Before optimising cost, what should you establish?

Asked in

aWhich model is most popularbWhere the money actually goes — cost per request broken down by feature and pipeline stage — because teams routinely optimise a stage that is 5% of the billcThe GPU vendor's roadmapdNothing; just switch to a cheaper model

Hands-on tasks:

FAQ

That's the course. Next: the company pages — reported questions from each company's rounds, with full answers. Start with TCS Fine-Tuning Interview Questions →

Or continue the AI track: RAG, AI Agents, MCP and Prompt Engineering.