RAG System Design: Design a Chatbot Over Your Docs

Forty minutes left in the interview. The interviewer says: "Design a chatbot that answers customer questions from our product documentation." Most candidates start drawing boxes within ten seconds — vector database, embeddings, LLM — and lose the round right there. This lesson is the alternative: a repeatable structure that turns everything in this course into a forty-minute answer that sounds like someone who has shipped one.

The structure that carries any RAG design round

Six steps, in this order. The order matters more than any individual component, because it demonstrates that you let requirements drive architecture rather than reciting a stack.

1 · Requirements — spend the first five minutes here

Ask, don't assume. Users: employees, customers, or both — and does everyone see the same documents? Corpus: how many documents, what formats, how often do they change, are there versions and withdrawals? Volume: queries per day, peak concurrency. Latency: chat UX or batch? Languages. And the one that reshapes everything: what does a wrong answer cost? A wrong cafeteria timing is a shrug; a wrong dosage, refund promise or safety step is a headline.

The cost of a wrong answer sets your refusal threshold, your grounding strictness, and whether a human sits in the loop. Stating that explicitly is the strongest opening move available to you.

2 · Ingestion & indexing

Narrate the pipeline with one decision per stage. Parsing — per format, with extraction sanity checks (a scanned PDF that yields no text must raise an alert, not disappear). Chunking — structure-aware where documents have headings or clauses, 200–500 tokens with overlap otherwise, tables and code kept whole. Enrichment — prefix document and section so fragments carry their context. Metadata — source, section, version, effective date, status, language, and permissions. Embedding — one model, chosen by testing recall on your own data. Maintenance — deterministic chunk ids, event-driven re-index on document change, delete orphans, alert on drift.

3 · Retrieval

Walk the funnel from lesson 6, and saywhy each stage exists for this product: query rewriting if it's conversational; filters first (permissions, product line, current version) so restricted or obsolete chunks never enter the pool; hybrid search because real queries mix natural language with model numbers and codes; fusion by rank; reranking down to 3–5 chunks; and a relevance threshold that allows retrieving nothing.

4 · Generation & guardrails

The prompt carries context, question, a grounding instruction, a refusal rule and a citation requirement — temperature 0. Around it: an input guardrail (PII masking, injection screening, scope filter) and an output guardrail (groundedness check, no-commitment rules, no leaked source names the user shouldn't see). Name the domain's catastrophic promise explicitly — the refund it must never offer, the seat it must never confirm, the medical step it must never invent — and put a check on it.

5 · Quality & operations

Golden set sampled from real questions, including must-refuse and paraphrased cases; recall@k and faithfulness measured separately; LLM-as-judge calibrated against human labels; the suite runs as a regression gate on every change. In production: thumbs-down and escalation rates, refusal-rate monitoring (a spike means the index broke), PII-masked logs of retrieved chunk ids and scores, and a weekly human audit that feeds failures back into the golden set.

6 · Scale & trade-offs — how to finish

Close with three sentences that make the whole answer sound senior. What I'd build first: the MVP — one language, the top documents, basic retrieval, grounding, refusal, golden set. What I deliberately deferred: reranking, semantic caching, agentic retrieval — because I'd add each only when measurement justified it. What breaks at 10×: generation latency and cost, ingestion throughput, hot-shard skew, cache hit rate — and how you'd respond (streaming, model routing, caching, sharding).

Selection-round radar: interviewers grade RAG design rounds on four things: did you gather requirements before designing, did you handle access control, did you say how you'd measure success, and did you name your trade-offs. Miss any one and a technically perfect diagram still scores mid.

A worked example, compressed

"Design a support bot over 10,000 product manuals, 6 languages." — Requirements: public customers, safety-critical instructions (so refusal beats guessing), manuals added weekly, chat latency. Ingestion:parse per product, structure-aware chunking that keeps procedures and tables whole, metadata = product id, model number, manual version, language. Multilingual: a multilingual embedding model so a Hindi question retrieves English manual text; answer in the user's language, cite the source. Retrieval: mandatory model-number filter (a Model X question must never retrieve Model Y — this is the safety-critical detail), hybrid for model codes, rerank to 4. Generation: grounded, cited, refuses when confidence is low, never invents a safety step. Quality: golden set per language from real support tickets; recall@k, faithfulness, refusal correctness. Rollout: top 3 products, 2 languages, measure, expand.

Common mistakes

• Drawing architecture before asking a single requirement question.
• Forgetting access control until the interviewer asks — it reshapes indexing, retrieval, caching and logging.
• No evaluation story: "how would you know it works?" must be answered before it's asked.
• Listing every advanced technique instead of choosing and justifying.
• Describing only the happy path — no failure behaviour, no degraded mode.
• No MVP-vs-later split, so the design sounds like a two-year project.

Quick recap

Practice Zone — PYQs from real selection rounds

Six MCQs and two full design tasks — the complete multilingual manual bot, then the same brief with a four-week deadline (what ships, what waits, and what must never be cut).

Q1RAG system designmedium

In a RAG design interview, what should you establish BEFORE drawing any architecture?

Asked in

aWhich vector database to usebRequirements: who asks what, corpus size and change rate, accuracy vs latency expectations, access rules, and the cost of a wrong answercThe embedding dimensiondThe programming language

Q2RAG system designmedium

Which requirement most changes the architecture of a document-QA system?

Asked in

aThe UI colour schemebWhether different users may see different documents — because access control must be enforced inside retrieval, not after itcThe choice of Python versiondWhether answers are shown in bold

Q3RAG system designhard

You're designing RAG over 50 million documents with 5,000 QPS. Which becomes the hardest problem?

Asked in

aWriting the promptbServing infrastructure: sharded/replicated ANN indexes, ingestion throughput, cost per query at that volume, and caching — plus LLM capacity and rate limitscChoosing the chunk sizedDeciding the temperature

Q4RAG system designmedium

How should a well-designed RAG chatbot handle a question that's clearly outside its corpus?

Asked in

aAnswer from the model's general knowledge without saying sobDetect low retrieval confidence, state that it doesn't cover that topic, and route to the right channel — with the behaviour tested like any other featurecReturn the closest chunk anywaydReturn an error code

Q5RAG system designmedium

Which failure mode should your design explicitly plan for?

Asked in

aThe LLM API being unavailable or rate-limitedbThe vector database being slow or downcIngestion silently failing for a document typedAll of the above — each needs a defined behaviour (retry/backoff, degraded mode, alerting)

Q6RAG system designmedium

What's the right way to close a RAG design answer in an interview?

Asked in

aList every advanced technique you knowbState the trade-offs you chose, how you'd measure success, and what you'd build first as an MVP versus latercSay the design is complete and has no weaknessesdAsk the interviewer to choose the components

Design tasks:

FAQ

Course complete. Now prove it — Company-wise RAG PYQs: start with TCS →