RAG Explained: Retrieval-Augmented Generation Step by Step

Your company hires a brilliant new employee. IIT topper, knows everything about everything — except your company. Day one, a customer asks about the refund policy. What do you do? You don't send them back to college for retraining. You hand them the policy document and say "answer from this." That exact move — smart generalist + the right document at the right moment — is RAG, the most-used GenAI architecture in industry and the most-asked applied topic in GenAI interviews. This is the lesson to know cold.

The problem: the model has never read your documents

An LLM knows what its training data contained — public internet, frozen at a cutoff date. Your leave policy, your product manuals, your client contracts? Never seen them. Ask anyway and you get the worst case from lesson 7: a fluent, plausible, invented answer. So how do you make a model answer from your knowledge? The naive options both fail: retrain/fine-tune on the documents? Slow, costly, unreliable for facts, and repeats every time a document changes (lesson 10 has the full autopsy). Paste everything into every prompt? Ten thousand pages don't fit any context window — and even when they fit, you'd pay for all of them on every question.

The idea: make it an open-book exam

In a closed-book exam you answer from memory — you hallucinate under pressure (every student knows this feeling). In an open-book exam, you first find the right page, then answer from it. RAG is exactly the open-book exam: Retrieve the few passages relevant to the question, Augment the prompt with them, Generate the answer from that context. Don't teach the model your knowledge — hand it the right page at the right moment.

The full pipeline — one diagram to memorize

Phase 1 — Indexing (done once, and on every document update): collect the documents, split them into chunks, turn each chunk into an embedding vector, store vectors + text in a vector database. Phase 2 — Query time (every question, live): embed the user's question with the same embedding model, run a similarity search for the nearest chunk-vectors, take the top few chunks, build a prompt of instructions + chunks + question, and let the LLM generate a grounded answer — ideally citing which chunk it used. If you can draw this from memory and narrate each arrow, you can answer 80% of RAG interview questions. Let's zoom into the three steps that carry the design decisions.

Chunking — why and how to split

Why not embed each document whole? Because a 50-page PDF is about fifty topics — one vector for all of it is a blurry average that matches nothing well. And retrieval-wise, you want to hand the model the relevant paragraph, not 50 pages of mostly-noise. Chunking = splitting documents into focused pieces, typically a few hundred tokens each.

The trade-off interviewers probe: too big → blurry vectors, wasted context, diluted relevance; too small → fragments lose their meaning ("the notice period is 30 days" — notice period of what?). Sensible defaults: 200–500 tokens, split along natural boundaries (headings, paragraphs) rather than mid-sentence, with a little overlap between neighbours so a fact sitting on a boundary survives in at least one chunk. Smarter variants (semantic chunking, heading-aware splitting) exist, but defaults-plus-reasoning is the placement-level answer.

Embeddings and vector search — lesson 2 pays off

Remember embeddings: text → vector, where distance ≈ difference in meaning. That geometry is the whole search engine. The question "how many casual leaves do I get?" lands near the chunk "employees get 12 casual leaves per calendar year" — even though they share almost no words. That's the upgrade over keyword search: semantic search matches meaning, not spelling. "Sick leave" finds "medical leave". The vector database's job is doing this nearest-neighbour lookup fast over millions of chunks (names to drop: Pinecone, Chroma, FAISS, pgvector, Weaviate, Milvus).

💡 The silent killer: index with one embedding model, query with another, and nothing errors — the similarity scores are just meaningless, because each model defines its own coordinate system. Everything runs; results are garbage. Same embedding model on both sides, always.

Assembling the answer — the final prompt

The last step is just disciplined prompting (lesson 6). The retrieved chunks and the question become one grounded prompt:

prompt = f"""You are the HR assistant. Answer ONLY from the
context below. If the context doesn't contain the answer, say
"I don't have that information" - do not guess.
Cite the source document of your answer.

Context:
{retrieved_chunks}   # top 3-5 chunks from the vector DB

Question: {user_question}
"""

You are entitled to 12 casual leaves per calendar year, with
requests needing one day's notice. (Source: Leave_Policy.pdf)

All three hallucination defences from lesson 7 are visible right there: grounding ("only from the context"), the escape hatch ("say you don't have it"), and citations. RAG isn't a separate magic — it's lessons 2, 6 and 7 assembled into a system.

Wait — context windows are huge now. Why not paste everything?

The modern version of the question, and interviewers love it. Three answers. Economics: tokens cost money on every call — pasting 300 pages per question, thousands of questions a day, is a bonfire of budget; retrieval sends 3 relevant chunks instead. Quality: models attend less sharply when the one relevant needle sits in a giant haystack (the "lost in the middle" effect) — focused context beats bulk context. Scale: real corpora are gigabytes; no window fits them. Long context and RAG are teammates, not rivals: retrieve first, then be generous with what you retrieved.

And beyond the window debate, RAG's structural wins: freshness (update a document → re-index one file — no retraining), citability (answers point to sources), and access control (retrieval can respect per-user permissions — the intern's query never retrieves the salary sheet).

Beyond basic RAG — the terms you'll hear

Production systems layer upgrades onto this skeleton: hybrid search (combine vector similarity with keyword search — jargon and exact codes like "HRA-104" need keywords), reranking (retrieve 20 candidates cheaply, then a smarter model re-orders and keeps the best 3), metadata filtering (only chunks where department = "HR"), and agentic RAG (an agent decides whether, where and how often to retrieve — lesson 11 energy). At placement level you need the names and one-liners, not implementations — and the honest signal that basic RAG's failures are usually retrieval failures, which is exactly what the Practice Zone's debugging task drills.

Selection-round radar: "Explain RAG" / "design a chatbot over company documents" is reported from TCS, Infosys, Accenture, Cognizant AND product companies — it's the closest thing GenAI interviews have to "reverse a linked list." Practice narrating the two-phase pipeline in 90 seconds, with chunk-size reasoning and the same-embedding-model rule included.

Common mistakes

• Different embedding models for indexing and querying — runs fine, retrieves garbage.
• Chunking blindly at fixed character counts through mid-sentence — split on natural boundaries with overlap.
• No escape hatch in the generation prompt — the model answers from memory when retrieval comes back empty, and you're back to hallucinating.
• Forgetting to re-index when documents change — the bot confidently serves last year's policy.
• Blaming the LLM when answers are wrong — in RAG, the bug is usually in retrieval (wrong/missing chunks), so debug there first.
• Ignoring permissions — retrieval that can surface any document to any user is a data leak with extra steps.

Quick recap

Practice Zone — PYQs from real selection rounds

Six MCQs and three serious tasks: build a toy retriever in numpy, design the HR-bot pipeline out loud, and debug a failing RAG system — the three shapes RAG questions actually take.

Q1RAGeasy

RAG (Retrieval-Augmented Generation) primarily solves the problem of:

Asked in

aLLMs being too slowbLLMs not knowing your private/current data and hallucinating when asked about it — RAG fetches relevant documents and lets the model answer from themcLLMs using too much GPU memorydUsers writing bad prompts

Q2RAGmedium

The correct order of a basic RAG pipeline is:

Asked in

aGenerate answer → search documents → embed questionbIndex: chunk & embed documents into a vector DB. Query: embed the question → retrieve similar chunks → LLM answers using those chunkscFine-tune the model on documents → delete the documents → ask questionsdSend all documents with every question

Q3RAGmedium

Documents are split into chunks before embedding because:

Asked in

aVector databases refuse files larger than 1 MBbRetrieval precision — one embedding for a whole 50-page PDF averages away its topics; small chunks let search find the exact relevant passage, and only that passage goes into the promptcChunking makes the text grammatically correctdEmbeddings only work on single sentences

Q4RAGeasy

The vector database in a RAG system stores:

Asked in

aOnly the original PDF filesbChunk embeddings (vectors) alongside the chunk text/metadata, and answers 'find the k nearest vectors to this query' fastcThe LLM's weightsdUser passwords

Q5RAGhard

Why must the query be embedded with the same embedding model used for the documents?

Asked in

aLicensing rules require itbDifferent models produce vectors in different, incomparable spaces — similarity scores between vectors from different models are meaninglesscDifferent models produce different vector lengths, and that is the only issuedIt doesn't matter; any model works

Q6RAGhard

Context windows now fit hundreds of pages. Why still use RAG instead of pasting all documents into every prompt?

Asked in

aNo reason — huge contexts made RAG obsoletebCost and latency scale with tokens on EVERY call; relevance drops when the answer is buried in noise ('lost in the middle'); and corpora are often far bigger than any window. RAG sends only what's relevantcRAG is required by law for company datadLong contexts only accept English text

Hands-on tasks:

import numpy as np

chunks = {
    "Employees get 12 casual leaves per calendar year.":  np.array([0.9, 0.1, 0.1]),
    "The office cafeteria serves lunch from 12 to 2 pm.": np.array([0.1, 0.9, 0.2]),
    "Laptops must be returned on the last working day.":  np.array([0.2, 0.3, 0.9]),
    "Casual leave requests need one day's notice.":       np.array([0.8, 0.2, 0.2]),
}
query_vec = np.array([0.85, 0.15, 0.12])  # "how many casual leaves do I get?"

FAQ

Next lesson: the question every interviewer eventually asks — Lesson 10: Fine-tuning vs RAG vs Prompting →