RAG in Production: Latency, Caching, Staleness, Cost & Multi-Tenancy

A RAG demo needs one thing: a good answer to a question you chose. A RAG product needs answers that stay correct when the policy changed this morning, arrive in under two seconds, cost less than they earn, never show one customer another customer's documents, and fail gracefully when the vector database is down. This lesson is that gap — and it's where interviews for real GenAI roles spend most of their time.

Latency — where the seconds actually go

Engineers instinctively optimize the vector search. Look at a real budget and you'll see why that's wasted effort:

So the real levers are: stream the response (free, and it changes perceived latency from three seconds to ~400 ms to first token), shorten the output (fewer generated tokens = less time), send fewer context tokens, and route easy questions to a smaller model. Optimize where the seconds are, not where the code looks clever.

Caching — the cheapest big win

Real support traffic obeys a power law: the same twenty questions arrive over and over. A semantic cache exploits that — embed the incoming question, and if it's very close to one you answered recently, return the stored answer instantly.

def cached_answer(question, threshold=0.97):
    q_vec = embed(question)
    hit = cache.nearest(q_vec)                 # tiny vector store of Q&As
    if hit and hit.similarity >= threshold:
        return hit.answer                      # ~5 ms instead of ~2 s
    ans = answer(question)                     # full RAG pipeline
    cache.put(q_vec, question, ans, ttl_hours=24)
    return ans

Two rules keep it honest: a high threshold (0.95+ — "leave for interns" and "leave for managers" are semantically close but must never share an answer), and invalidation on re-index, or your cache becomes the stale-answer machine described next. Prompt caching at the API level is a separate, complementary win for the fixed part of your system prompt.

Wait — the policy changed at 9 a.m. and the bot still quotes the old one

The most common production incident in RAG, and it has two causes worth separating.

Cause 1: slow indexing. If re-indexing is a nightly batch, you have up to a day of staleness by design. Fix: event-driven re-indexing — the document store emits a change event, the pipeline re-chunks and upserts that one document in minutes.

Cause 2: the delete nobody wrote. Far nastier. The pipeline upserts new chunks but never removes chunks for deleted sections or withdrawn documents — so the old policy stays retrievable forever, often outranking its replacement because its wording matches user questions better. Fix: deterministic chunk ids, per-document reconciliation on every run (delete orphans), a status/effective_date filter so withdrawn documents can never be retrieved, and a daily drift check comparing source documents with indexed chunks.

Staleness is an indexing-pipeline bug, never a model bug — and because it fails silently, prevention must include a monitor, not just a fix.

Cost — what actually shows up on the bill

People assume vector storage is the expense. It isn't: a few million vectors is a one-off, and cheap. The recurring cost is LLM tokens per query — retrieved chunks plus instructions, re-sent on every single request.

queries_per_day = 20_000
context_tokens, output_tokens = 2_500, 250     # 4 chunks + instructions
in_rate, out_rate = 0.010, 0.030               # Rs per 1K tokens

daily = queries_per_day * (context_tokens/1000*in_rate +
                           output_tokens/1000*out_rate)
print(f"Rs {daily:,.0f}/day   Rs {daily*30:,.0f}/month")

Rs 650/day   Rs 19,500/month

Levers in order of impact: fewer/tighter chunks in context (reranking pays for itself here), prompt caching, semantic caching, model routing, and only then output length.

Multi-tenancy and security

If your bot serves multiple customers or departments, cross-tenant leakage is the failure that ends the product. Enforce isolation where it cannot be argued around: separate namespaces or indexes per tenant, or a mandatory tenant filter injected server-side from the session — never taken from the client, never expressed as a prompt instruction.

Two details people miss: cache keys must include the permission scope (otherwise an admin's cached answer can be served to an intern), and citations must respect permissions too — a source filename can itself be sensitive ("Layoff_Plan_Q3.pdf").

Failure modes — decide the behaviour in advance

Selection-round radar: "What problems did you face running RAG in production?" is a favourite because it can't be answered from a blog post. Safe, real answers: stale chunks after document deletion, vocabulary gaps between user language and document language, latency dominated by generation, cost dominated by context tokens, and per-user access control. Pick two and explain how you detected them.

Logging that's useful and safe

You cannot debug retrieval without knowing which chunks were retrieved and with what scores — that single log line is the most valuable one in a RAG system. But questions contain PII (names, phone numbers, Aadhaar, account numbers).

The workable middle: log the PII-masked question, retrieved chunk ids and scores, model and prompt version, latency, token counts, and whether the bot refused — with retention limits. That gives you full debuggability without storing sensitive text, and it satisfies data-protection expectations (India's DPDP Act being the one to name).

Common mistakes

• Optimizing vector search while generation eats 80% of the latency.
• Upsert-without-delete, producing confidently-cited withdrawn policies.
• A semantic cache with a loose threshold or no invalidation.
• Cache keys that ignore the user's permission scope.
• No defined behaviour when the LLM or vector store fails.
• Logging raw conversations with PII intact — or logging nothing at all.

Quick recap

Practice Zone — PYQs from real selection rounds

Six MCQs and two tasks — cut a latency budget under 2.5 s, and write the postmortem for a three-week stale-answer incident.

Q1RAG in productionmedium

Which component usually dominates RAG response latency?

Asked in

aVector searchbLLM generation — retrieval is typically tens of milliseconds while generation takes secondscEmbedding the querydMetadata filtering

Q2RAG in productionmedium

What's the most effective cache in a production RAG system?

Asked in

aCaching the embedding model weightsbCaching answers for repeated/near-duplicate questions (semantic cache), since real traffic is dominated by a few common questionscCaching the entire vector index in the browserdCaching user sessions only

Q3RAG in productionhard

A policy document is updated at 9 am. Users get the old answer until 2 pm. What's the architectural fix?

Asked in

aRestart the LLMbAn event-driven re-index — trigger re-chunking/re-embedding of the changed document on save (plus cache invalidation) instead of relying on a slow batch jobcIncrease top-kdUse a bigger embedding model

Q4RAG in productionmedium

In a multi-tenant RAG SaaS, the safest isolation approach is:

Asked in

aOne shared index, with the tenant id in the prompt so the model filtersbHard isolation at the data layer — separate indexes/namespaces per tenant, or a mandatory tenant filter enforced in code, never in the promptcSeparate embedding models per tenantdRate limiting per tenant

Q5RAG in productionmedium

Which logging practice is both useful and safe for a production RAG bot?

Asked in

aLog raw questions, answers and user identifiers indefinitelybLog PII-masked questions, retrieved chunk ids, scores, latency and token counts — with retention limits — so failures can be diagnosed without storing sensitive textcLog nothing, for privacydLog only errors

Q6RAG in productionmedium

What is the biggest ongoing COST driver in most RAG systems?

Asked in

aVector storagebLLM tokens per query — retrieved chunks plus instructions are re-sent on every single requestcRe-embedding documentsdThe reranker

Hands-on tasks:

FAQ

Next lesson: the comparison every interviewer eventually asks for — Lesson 11: RAG vs Fine-Tuning →