Your company hires a brilliant new employee — topper, sharp, knows something about everything. Day one, a customer asks: "what's your refund window for damaged goods?" She has never seen your policy. What do you do? You don't send her back to college for four more years. You hand her the policy document and say "answer from this." That's it. That's RAG. The rest of this lesson is just making that instinct precise enough to build — and to defend in an interview.
The problem: the model has never read your documents
An LLM knows what was in its training data — a huge slice of the public internet, frozen at a cutoff date. It has never seen your leave policy, your product manuals, your client contracts, or anything that happened last week. Ask anyway and you get one of two outcomes: an honest "I don't have access to that", or — far more often and far more dangerous — a fluent, confident, invented answer.
That's not the model malfunctioning. It generates the most plausible continuation of your prompt, and a specific-sounding refund policy is more plausible as text than "I don't know". (The full mechanism is in the GenAI course's hallucination lesson.) So: how do we make a general-purpose model answer from our knowledge?
Two obvious fixes — and why both break
Fix 1: train the model on our documents. Intuitive, and wrong for this job. Training bakes knowledge into weights as fuzzy patterns, not a lookup table — the model ends up "vaguely remembering" your policy and confidently misquoting the numbers. It can't cite anything. And every time a document changes, you retrain. (Lesson 11 settles this argument properly.)
Fix 2: paste all the documents into every prompt. Also intuitive, also broken. A 2,000-page corpus doesn't fit in any context window — and even when a subset does, you pay for every token on every single question, forever, while the model attends less sharply to one relevant line buried inside a haystack.
Notice what both failures have in common: they try to give the model everything. The winning idea gives it only what this question needs.
The open-book move
Same model, same question, wildly different reliability. The only difference is whether the right paragraph was in front of it. RAG converts an LLM's closed-book exam into an open-book exam: find the right page first, then answer from it.
What RAG actually is — three letters, three steps
R — Retrieve. Search your documents for the few passages most relevant to this question. (Not by keywords — by meaning, which is lessons 2 and 3.) A — Augment. Insert those passages into the prompt, with an instruction: answer only from this context; if it isn't here, say you don't know. G — Generate. The LLM writes the answer — and because the true text is right there, the most plausible continuation is now the truth.
Two phases hold this together, and mixing them up is the classic beginner error. Indexing happens once (and again when documents change): split documents into chunks, convert each to an embedding, store them. Query time happens per question: embed the question, find similar chunks, build the prompt, generate. Lesson 7 builds both in about thirty lines of Python.
Selection-round radar: "What is RAG?" is the opening question of nearly every GenAI interview. The answer that scores: the problem (model doesn't know your data, invents answers) → the open-book fix → indexing phase → query phase → the three wins below. Practise it to 45 seconds; you'll do it in the Practice Zone.
The three wins that made RAG the default
Fresh. Update a document, re-index that one file, done in minutes — no training run, no model version. A fine-tuned model is out of date the day after it's trained.
Citable. Because the answer came from a specific chunk, you can show the user the source: "12 casual leaves [Leave_Policy.pdf §4]". In regulated industries this isn't a nice-to-have — an unciteable answer is an unusable one.
Controllable. Retrieval respects permissions: an intern's query simply never retrieves the compensation sheet (lesson 4). Knowledge inside weights has no access control at all — once it's trained in, it's in for everybody.
Wait — when is RAG the wrong tool?
Every honest engineer needs this list, and interviewers love it. 1 · Behaviour problems. If the complaint is "wrong tone", "wrong format", "doesn't follow our template" — that's prompting or fine-tuning. Retrieval supplies facts, not manners. 2 · Aggregate questions. "What percentage of our 40,000 tickets mention delivery delays?" RAG retrieves a handful of chunks; it cannot count across a corpus. That's a database query. 3 · The model already knows. "Explain recursion" needs no retrieval — adding it costs latency and money for nothing. 4 · Tiny, static knowledge. If your entire knowledge base is two paragraphs that never change, put them in the system prompt and go home.
💡 Tip: the fastest way to answer any "RAG or fine-tuning or prompting?" question is to sort the problem first: knowledge → RAG, behaviour → tuning, neither → prompting. You'll use this sorting move throughout the course.
Common mistakes
- Calling RAG a kind of training — nothing about the model changes; only the prompt does.
- Forgetting the escape hatch. Without "say you don't know", an empty retrieval quietly becomes a hallucination.
- Blaming the LLM when answers are wrong — in RAG, suspect retrieval first, always.
- Expecting RAG to answer counting/aggregation questions over the whole corpus.
- Skipping citations, then wondering why nobody trusts the bot.
Quick recap
| Concept | One-liner |
|---|---|
| RAG | retrieve relevant chunks → augment the prompt → generate a grounded answer |
| Two phases | index once (chunk → embed → store); retrieve + generate per question |
| Why not fine-tune | facts in weights are fuzzy, unciteable and expensive to update |
| Why not paste everything | cost per call, attention dilution, and corpora that don't fit |
| Three wins | fresh · citable · permission-controllable |
| Not for | tone/format, corpus-wide aggregation, things the model already knows |
Practice Zone — PYQs from real selection rounds
Six MCQs and three tasks — including the 45-second "explain RAG" drill that opens most interviews. Attempt first, reveal second.
In the name Retrieval-Augmented Generation, what exactly is being *augmented*?
Asked in
Which problem is RAG not designed to solve?
Asked in
A RAG system returns a perfectly-worded answer that contradicts the company policy document. Where do you look FIRST?
Asked in
Which of these is a genuine advantage of RAG over fine-tuning for enterprise knowledge?
Asked in
What does 'grounding' mean in a RAG system?
Asked in
Which task would a RAG system handle WORST, even when perfectly built?
Asked in
Hands-on tasks:
For each requirement, say whether RAG is the right primary tool: 1) The bot must answer from a policy PDF updated monthly. 2) The bot must always reply in the company's formal Hindi register. 3) The bot must answer questions about news from this morning. 4) The bot must cite its sources. 5) The bot must count how many customers complained last quarter.
Asked in
"Explain RAG to me" is the opening question of most RAG interviews. Write your 45-second answer — definition, pipeline, and why it exists — then compare.
Asked in
A teammate proposes skipping RAG: "our whole handbook is 80 pages — just paste it into every prompt." Give three concrete reasons this breaks, with rough numbers where you can.
Asked in
FAQ
Does RAG make hallucinations impossible?
No — it's the strongest single reducer, not a cure. The model can still misread chunks, blend two policies, or answer from memory when retrieval returns nothing and no refusal rule stops it. That's why lesson 9 measures groundedness instead of assuming it.
Is RAG the same as giving the model a search engine?
Close in spirit, different in scope. Web search is one possible retrieval source; RAG is the general pattern — retrieve from whatever corpus (your PDFs, a database, tickets, code) and ground the answer in what comes back.
Next lesson: the piece of maths that makes retrieval possible — Lesson 2: Embeddings Explained →