Your friend hires a brilliant assistant. Week one, two things go wrong. The assistant doesn't know the office wifi password — and the assistant writes emails in a chatty style the clients hate.
Those are not the same problem. One is fixed by telling them the password. The other needs a week of showing them how your firm writes. Confusing the two is the single most expensive mistake in AI projects — and it is exactly what "should we fine-tune?" usually means.
Three tools, three problems
Read that diagram once more, because everything else in this course assumes it:
| Symptom | Tool | Why |
|---|---|---|
| Wrong format, wrong scope, ignores a rule | Prompting | the task was never specified properly |
| Invents facts, can't answer about your data, out of date | RAG | knowledge it doesn't have must be supplied at question time |
| Right facts, right task, but consistently the wrong voice or shape — at volume | Fine-tuning | a behaviour worth baking into the weights |
Facts → retrieval. Behaviour → fine-tuning. Constraints → prompting. Say that sentence in an interview and you have already answered half the follow-ups.
A five-minute diagnostic
Architecture arguments can run for a week. This experiment usually ends them in an afternoon:
Take a case the system gets wrong. Paste the correct information into the prompt by hand. Run it again.
- The answer becomes right → the model was missing knowledge. That is a retrieval problem, and no amount of fine-tuning fixes it well.
- The answer is still wrong → the model has what it needs and still doesn't do the job. Now you are choosing between a better prompt and a fine-tune.
💡 Do this before any planning meeting. It converts "I think we need to fine-tune" into evidence, and it is free.
When fine-tuning genuinely wins
Four cases, and notice that three of them are about economics rather than capability.
1. Consistent behaviour at volume. Prompting already works, but you pay for the same eight few-shot examples on every one of two million daily requests. Bake the behaviour into the weights and the prompt shrinks — cheaper and faster, forever.
2. A voice or format you can show but not describe. Brand copy, a hospital's note format, a firm's house style for legal summaries. If you have thousands of approved examples and prose instructions keep falling short, examples in the weights beat examples in the prompt.
3. Making a small model do one job well. Often the strongest case commercially: a small model, fine-tuned on one narrow task, can match a much larger model on that task at a fraction of the cost and latency. You are not chasing capability — you are chasing the same capability cheaper.
4. Latency. A shorter prompt is a faster time-to-first-token, and a smaller model is faster still.
When it's the wrong answer
- To add knowledge. Facts blended into weights come back without a source, can be half-remembered, and go stale until the next training run. Retrieval keeps them outside the model where they can be updated, cited and deleted.
- For anything that changes. Prices, policies, inventory, staff. A weekly-changing fact in a quarterly-retrained model is a bug generator.
- For something a prompt rule fixes. "Too verbose" is a length constraint, not a training project.
- With a hundred examples. Almost always too few to beat a good prompt with a few of those hundred used as few-shot examples.
- Before measuring the prompted baseline. Without it you cannot tell whether the fine-tune helped, which makes the whole project unfalsifiable.
Wait — what does it cost after launch?
Teams cost the training run. The training run is often the cheapest line. What you are actually signing up for:
| Ongoing cost | What it means |
|---|---|
| A dataset you own | built and reviewed by people; weeks of human time, not GPU time |
| An evaluation suite | task quality and general-capability regression, maintained |
| A serving story | a hosted fine-tune at a higher per-token price, or your own inference stack |
| Version lock | you are pinned to one base model; a newer, better base means re-doing everything |
| Data-deletion complexity | training data is in the weights — you can't delete a record without retraining |
Fine-tuning turns a model you rent into a model you own — with everything ownership implies. That is fine when the case is strong. It is a bad trade for a format problem.
🎯 Selection-round radar: "When would you fine-tune instead of using RAG?" is one of the most-asked AI interview questions anywhere. Three beats: they solve different problems — RAG supplies knowledge, fine-tuning shapes behaviour → so I'd diagnose first, and the paste-the-facts-in test settles it in minutes → and I'd only fine-tune when a demonstrable behaviour is needed at a volume that makes prompt tokens expensive. Then add the closer about version lock and maintenance, which most candidates never mention.
Using them together
They are not competitors, and the strongest production systems use both. A fine-tuned model that writes clinical notes in one hospital's exact format still retrieves this patient's records. Behaviour comes from the weights; facts come from the context.
The order matters, though. Try prompting, then retrieval, then fine-tuning — each step costs more in data, money and maintenance than the one before, and each is harder to reverse. Teams that start at the expensive end usually discover, months later, that the cheap end would have worked.
Common mistakes
- Fine-tuning to inject facts, then wondering why they go stale.
- Skipping the prompted baseline, so success is unmeasurable.
- Costing the GPU hours and forgetting the weeks of data labelling.
- Ignoring version lock — you are pinned to today's base model.
- Choosing fine-tuning because it sounds more impressive than a prompt.
- Assuming a few hundred examples is enough to beat few-shot prompting.
Quick recap
| Concept | One-liner |
|---|---|
| The rule | facts → RAG, behaviour → fine-tuning, constraints → prompting |
| The diagnostic | paste the facts in by hand; right answer = retrieval problem |
| Best cases | behaviour at volume, a demonstrable house style, a small model doing one job |
| Worst cases | knowledge, anything that changes, anything a prompt rule fixes |
| Hidden cost | a dataset, an eval suite, a serving stack, and version lock to one base model |
| Order | prompting → RAG → fine-tuning; they compose, so it's not either/or |
Practice Zone — PYQs from real selection rounds
Six MCQs, then two decision tasks — four scenarios to route, and a cost argument to build.
A chatbot invents facts about your company's internal policies. What is the right fix?
Asked in
Which of these is the strongest genuine case for fine-tuning?
Asked in
What is the correct order to try approaches in?
Asked in
A quick diagnostic to decide between RAG and fine-tuning is:
Asked in
Can RAG and fine-tuning be used together?
Asked in
Which hidden cost of fine-tuning do teams most often underestimate?
Asked in
Hands-on tasks:
For each, say whether you would use prompting, RAG, fine-tuning, or a combination — and why. (1) A support bot must answer from a 4,000-page policy manual that changes monthly. (2) A medical-scribe product must output notes in one hospital's exact SOAP format, 50,000 times a day. (3) A model summarising legal contracts is too verbose. (4) A code assistant must follow a company's internal API conventions that appear nowhere public.
Asked in
A team wants to fine-tune to remove 6 few-shot examples (~1,200 tokens) from a prompt running 200,000 times a day. Sketch the cost argument they should make, including what they'd be signing up for.
Asked in
FAQ
How many examples do I need before fine-tuning is worth considering?
As a rough gate: if you can't reach several hundred genuinely good, consistently labelled examples, spend that effort on the prompt instead. Providers accept far fewer, but "the job runs" and "the result beats a good prompt" are different bars. Lesson 3 covers sizing properly.
Can I fine-tune a model I access only through an API?
Usually yes — most major providers offer hosted fine-tuning on selected models: you upload data, they run the job, you get a private model id. You don't get the weights, so you can't take it elsewhere. Open-weight models give you the artefact and the portability, at the cost of running the training and the serving yourself.
Is fine-tuning the same as training a model from scratch?
No, and the gap is enormous. Pretraining costs millions and needs trillions of tokens. Fine-tuning continues from that finished model with a few thousand of your examples, on one GPU, in hours. Essentially nobody at placement level is being asked about pretraining.
Next lesson: what actually changes inside the model, and the two failure modes that show up in every training run — Lesson 2: How Fine-Tuning Works →