RAG vs Fine-Tuning: The Interview Question Everyone Gets

A team once spent several lakh rupees fine-tuning a model on their product catalogue so the bot would "know the products". It learned to sound exactly like their catalogue — and then quoted prices that didn't exist, cited nothing, and needed retraining every time a price changed. They rebuilt it with RAG in two weeks. That story is the whole lesson, and some version of it is the most-asked question in GenAI interviews. By the end of this page you'll answer it with a framework, not a preference.

The real question hiding inside the question

"RAG or fine-tuning?" sounds like a comparison of two technologies. It isn't. It's a diagnostic question: what exactly is wrong with the model's output? Answer that, and the technology chooses itself.

There are only two kinds of wrong. Either the model doesn't know something (your policies, this week's prices, a client's contract), or it doesn't behave right (wrong tone, wrong format, wrong domain conventions). Those two failures have two different fixes, and mixing them up is the expensive mistake.

Knowledge vs behaviour — the whole framework

Fine-tuning teaches behaviour. RAG supplies knowledge. Prompting is the cheap first attempt at both. Say that sentence in an interview and you've answered the question before the follow-ups arrive.

Wait — why can't training just teach it the facts?

Because of what weights are. Training adjusts billions of numbers so the model's next-token predictions match patterns in the data. It is compression of patterns, not storage of records — there is no row in there saying "refund window = 30 days".

So after fine-tuning on your catalogue, the model has absorbed thestyle of your catalogue and a hazy sense of its content — rather like you remember a book you read last year. You'd recall the themes confidently and misquote the exact figures with equal confidence. That's exactly what the model does, and three consequences follow:

1 · Precision is unreliable — numbers, dates and names come out plausible rather than correct. 2 · Nothing is citable — the answer has no source, so nobody can verify it and no auditor will accept it. 3 · Updates cost a training run — one policy change, one retrain, one re-evaluation.

The comparison table interviewers want

When fine-tuning genuinely wins

This lesson isn't anti-tuning — a candidate who says "always RAG" is as wrong as one who says "always fine-tune". Fine-tuning is the right call when:

Behaviour must be reliable, not requested. A rigid regulator-approved template that prompting holds only 80% of the time becomes near-perfect after tuning on a few thousand real examples. The prompt has grown absurd. If every call carries 1,500 tokens of instructions and examples, tuning can compress that into learned behaviour — and then, the subtle part, a fine-tuned small model can often replace a large one, making tuning a cost optimization on top of RAG rather than a rival to it. Latency matters and prompts are long. Shorter prompts mean fewer input tokens and faster responses.

The hybrid answer (what production actually looks like)

Exam answers pick one. Real systems use all three, each doing what it does best: a carefully engineered system prompt for rules and format, RAG for the facts with citations, and — if scale justifies it — a LoRA fine-tune for the house voice or a strict output shape that prompting couldn't hold.

The build order never changes: prompt first, add RAG when facts are needed, fine-tune last — each step only after the previous one measurably falls short. (LoRA and PEFT at intuition level are covered in the GenAI course's fine-tuning lesson.)

Selection-round radar: the highest-scoring answer has four beats: (1) knowledge vs behaviour framework, (2) why facts in weights are fuzzy, unciteable and expensive to update, (3) when tuning genuinely wins — style, format, prompt compression, (4) production combines them. Add "and I'd try prompting first" and you've covered every follow-up they had planned.

The forgotten option: fine-tune the retriever

Everyone debates tuning the LLM. But if your problem is that recall@k is poor because your domain speaks a strange language — legal, medical, internal jargon — then the bottleneck is the embedding model, and that's what should be tuned. With real query→correct-document pairs from your logs, a tuned retriever can lift recall substantially, and every downstream metric improves with it.

Mentioning this unprompted is a genuine differentiator: it shows you diagnose which component is failing rather than reaching for the famous knob.

Common mistakes

• Fine-tuning to inject facts — the flagship mistake this whole lesson exists to prevent.
• Answering "always RAG" — behaviour problems don't respond to retrieval.
• Skipping prompting; a well-written system prompt kills most fine-tuning requests.
• Ignoring that the real fine-tuning bottleneck is quality training data, not GPUs.
• Forgetting that only RAG offers citations and per-user access control.
• Never considering that the retriever might be the component worth tuning.

Quick recap

Practice Zone — PYQs from real selection rounds

Six MCQs and two tasks — four client scenarios (one of which needs neither technique), and a cost comparison with the deciding factor that isn't money.

Q1RAG vs fine-tuningmedium

A team fine-tuned a model on 5,000 support tickets and it now writes perfect support-style replies — but invents order numbers. What does this demonstrate?

Asked in

aThe fine-tuning failed completelybFine-tuning transferred style and format successfully but cannot store precise, retrievable facts — those must come from retrieval or toolscThey needed more epochsdThe base model was too small

Q2RAG vs fine-tuningmedium

Which factor most strongly favours RAG over fine-tuning in an enterprise setting?

Asked in

aThe team knows PythonbKnowledge changes frequently and answers must be auditable against a sourcecThe company has spare GPUsdThe documents are written in English

Q3RAG vs fine-tuninghard

When might fine-tuning genuinely REDUCE your RAG system's cost?

Asked in

aNever — they're unrelatedbWhen a fine-tuned smaller model can follow your grounding/format rules reliably with a much shorter system prompt, cutting per-query tokens and letting you drop to a cheaper modelcBecause fine-tuning removes the need for retrievaldBecause fine-tuned models have larger context windows

Q4RAG vs fine-tuningmedium

"We'll fine-tune on our documents so we don't need a vector database" — the strongest single counter-argument is:

Asked in

aFine-tuning is expensivebFacts in weights are unciteable and fuzzy: the model may misquote numbers with total confidence and you can never show a user where an answer came from — plus every document change needs retrainingcVector databases are easy to rundFine-tuning requires GPUs

Q5RAG vs fine-tuningmedium

What does a 'RAG + fine-tuning' hybrid typically look like in production?

Asked in

aFine-tune on documents, then also retrieve the same documentsbRetrieval supplies the facts per request; fine-tuning shapes how the model uses them — house style, strict output format, domain phrasing, better refusal behaviourcFine-tune the embedding model instead of the LLMdAlternate between the two on different days

Q6RAG vs fine-tuninghard

Fine-tuning the embedding model (rather than the LLM) is worth considering when:

Asked in

aAnswers are badly formattedbRetrieval quality is the bottleneck in a domain with unusual vocabulary, and you have labelled query→correct-document pairs to train oncThe LLM is too slowdThe context window is too small

Hands-on tasks:

FAQ

Final lesson: putting everything together as an interview answer — Lesson 12: RAG System Design →