Fine-Tuning vs RAG vs Prompt Engineering: Which One When?

A new team member isn't performing. You have three options: write them a clearer brief (cheap, try today), give them the right documents whenever a task needs them, or send them for actual training that changes how they work permanently. LLMs have exactly these three options — prompting, RAG, fine-tuning — and "which one, when?" is the most repeated GenAI interview question in existence. After this lesson, you'll answer it with a framework, not a guess.

The three tools, honestly labelled

Prompt engineering (lesson 6): change the instructions. Costs minutes. Affects only the current request. RAG (lesson 9): change the information supplied per request. Costs days to set up. Knowledge stays outside the model, updated by re-indexing. Fine-tuning: change the model itself. Costs data + GPU time + evaluation. Permanent until you train again.

One line captures the divide: prompting and RAG change what goes INTO the model per request; fine-tuning changes the model. Everything else in this lesson is consequences of that line.

What fine-tuning really is

Take a pre-trained model and continue training it on your examples — typically hundreds to thousands of input→output pairs showing exactly the behaviour you want. The weights shift toward your patterns. Afterwards, the behaviour is built in: no examples needed in the prompt, shorter prompts, consistent style. That's the payoff. The bill: you need quality training data (the real bottleneck — garbage pairs in, garbage behaviour out), compute, evaluation before/after, and a hosting story for the resulting model. And two risks: overfitting to your examples, and catastrophic forgetting — push weights hard toward a narrow task and the model gets worse at everything else.

The core distinction: knowledge vs behaviour

Here's the framework that settles almost every scenario. Ask: is my problem about what the model KNOWS, or how it BEHAVES?

Knowledge problems — the model lacks facts, or the facts change: your policies, live prices, this week's documents. → RAG. Facts stay in an index you can update in minutes, answers can cite sources. Behaviour problems — it knows enough but responds in the wrong style, format, or domain habit: a legal-drafting tone, strict report templates, your codebase's conventions, a medical-transcription style. → fine-tuning (after prompting has genuinely been tried). And always start with prompting — it's an afternoon, not a project; the number of "we need fine-tuning" requests that die to a well-written system prompt is legendary.

LoRA and PEFT — fine-tuning without the price tag

Full fine-tuning updates billions of weights — serious GPU memory, serious money. LoRA (Low-Rank Adaptation) changed the economics: freeze the original model, add tiny trainable "adapter" matrices beside key layers, train only those — typically under 1% of the parameters. Think sticky notes on a textbook: the book stays untouched, your additions sit on top, and you can keep different sticky-note sets (one per task) for the same book. QLoRA adds quantization (lower-precision numbers) so even modest GPUs can fine-tune. The umbrella term is PEFT — Parameter-Efficient Fine-Tuning. Interview hierarchy, one line: PEFT is the family, LoRA its star member, QLoRA the budget edition.

Wait — can't we just fine-tune on our 500 PDFs so it "knows everything"?

The most expensive misunderstanding in applied GenAI, and a deliberately laid interview trap. It fails for three reasons. One: weights store patterns, not a lookup table — after training on your PDFs the model "vaguely remembers" them the way you vaguely remember a book from last year: it absorbs the style and gist, then confidently misquotes the exact numbers. Two: no citations — nobody can check where an answer came from. Three: the update problem — one policy changes, you retrain; RAG re-indexes one file in minutes. Fine-tuning teaches behaviour; RAG supplies knowledge. Deliver that sentence with the three reasons and you've passed the question senior engineers fumble.

The comparison table interviewers want

Real systems combine them

The exam answer is a choice; the production answer is a stack. A typical mature assistant uses all three: a carefully engineered system prompt (identity, rules, format), RAG for the knowledge (documents, tickets, prices), and — if scale justifies it — a LoRA fine-tune for the brand voice or strict output format that prompting couldn't hold consistently. The build order is always the same: prompt first, add RAG when facts are needed, fine-tune last — each step only when the previous one measurably falls short.

Selection-round radar: "RAG vs fine-tuning" appears in Infosys, Accenture, Amazon and Microsoft rounds with boring reliability. The full-marks structure: the knowledge-vs- behaviour framework → freshness and citation arguments → "facts via fine-tuning is unreliable" → close with "real systems combine: behaviour from tuning, knowledge from retrieval, and always try prompting first."

Common mistakes

• Fine-tuning to inject facts — the flagship mistake; facts live in retrieval.
• Jumping to fine-tuning before honestly exhausting prompting — an afternoon of prompt work kills most tuning requests.
• Forgetting the training-data bottleneck — no quality input→output pairs, no fine-tune, whatever the GPU budget.
• Ignoring catastrophic forgetting — narrow aggressive tuning degrades general ability; PEFT and gentle schedules mitigate.
• Treating them as rivals — mature systems use prompt + RAG + (sometimes) LoRA together.

Quick recap

Practice Zone — PYQs from real selection rounds

Six MCQs and three scenario tasks — including the budget-pushback conversation you may genuinely have at work one day.

Q1Fine-tuning vs RAGeasy

Your bot must answer from product docs that change every week. The right primary approach is:

Asked in

aFine-tune the model weekly on the new docsbRAG — update the document index; the model needs no retrainingcTrain a new model from scratch monthlydIncrease temperature so answers vary with the docs

Q2Fine-tuning vs RAGeasy

Fine-tuning a model means:

Asked in

aWriting a longer, more detailed promptbContinuing training on your own examples so the model's weights actually changecAttaching a vector databasedRaising the temperature parameter

Q3Fine-tuning vs RAGmedium

LoRA makes fine-tuning cheap by:

Asked in

aDeleting most of the model's layersbFreezing the original weights and training only small low-rank adapter matrices alongside them — a tiny fraction of the parameterscTraining on only 10 examplesdUsing CPUs instead of GPUs

Q4Fine-tuning vs RAGmedium

PEFT stands for / refers to:

Asked in

aParameter-Efficient Fine-Tuning — the family of methods (LoRA, QLoRA, adapters, prompt tuning) that adapt a model while training only a small parameter subsetbPre-Embedding Fast TransformercA vector database productdA prompt-writing technique

Q5Fine-tuning vs RAGhard

The classic mistake of fine-tuning on your company's documents to "teach the model the facts" fails because:

Asked in

aFine-tuning is good at teaching style, format and behaviour, but unreliable at storing precise retrievable facts — the model may still hallucinate them, and updating any fact means retrainingbFine-tuning always erases the base model's knowledge completelycDocuments cannot legally be used for trainingdIt works perfectly; there is no mistake

Q6Fine-tuning vs RAGhard

Catastrophic forgetting in fine-tuning refers to:

Asked in

aThe model deleting its training data after usebAggressive fine-tuning on a narrow dataset degrading the model's general abilities it had beforecUsers forgetting their API keysdThe context window being cleared between calls

Scenario tasks:

FAQ

Next lesson: 2026's headline topic — Lesson 11: AI Agents →