Fine-Tuning in Practice: Project, Rollout, Versioning & Stop Conditions

Everything so far has been pieces. This lesson is the whole project: what actually happens between "we decided to fine-tune" and "it's serving traffic" — including the two things nobody puts in a plan, which are how you'd know it was a mistake and how you'd undo it.

Hosted or self-hosted?

Sensible default: start hosted. It removes every variable except the two that decide the project — is the dataset good, and does the result beat a prompt. Move to self-hosting when volume, residency or a specific open model demands it.

Baseline before anything

The single most important sentence in this lesson: build the evaluation set and score the prompted baseline before you train anything.

Not because it is tidy, but because without it the project is unfalsifiable. "The fine-tune is good" means nothing without "compared to what?" And the outcome teams least expect is the most common one: the prompted baseline is already good enough, and the project ends there. That is a success. You saved six weeks.

The baseline should be a genuine attempt — a proper system prompt, well-chosen few-shot examples, a sensible temperature — not a deliberately weak strawman that your fine-tune is guaranteed to beat.

What the timeline really looks like

Data is weeks; training is hours. If a plan you are shown reverses that proportion, the plan hasn't met a dataset yet — and saying so in an interview is a strong signal.

Versioning: what to record

A fine-tuned model without its provenance is unreproducible. Six months later someone asks why v3 behaves differently from v2, and the weights alone cannot answer.

model: support-reply-v3
base_model: <exact base version string>     # NOT a floating alias
dataset:
  version: support-2026-08
  examples: 2,847
  built_from: tickets 2026-01..2026-07, agent-reviewed
  split: grouped by account + time (70/15/15)
training:
  method: LoRA
  rank: 16
  alpha: 32
  epochs: 2 (early stop on val_loss)
  learning_rate: 1.5e-4
eval:
  task_win_rate_vs_prompted: 61%
  grounding_violations: 0.4% (baseline 0.6%)
  general_regression: none > 1pt
  cost_per_reply: -38%
  p95_latency: -12%
approved_by: <name>, 2026-08-29
rollback_to: prompted-v11

Note the last line. Every model version needs a named thing to roll back to, and it should be written down before launch rather than improvised during an incident.

Rolling it out

A model version is a deploy. Treat it like one.

1. Shadow run. Send a copy of live traffic to the new model, serve none of its responses, and diff against production. It costs only compute and it catches the failures your eval set missed — which, by definition, are the ones you didn't think of.
2. Ramp. 5% → 25% → 100%, watching quality and cost at each stage, not just errors.
3. Keep the old path warm. Rollback should be one config change, not a redeploy.
4. Watch behavioural signals, not just dashboards — edit rate, escalation rate, retry rate. They move before anyone files a complaint.

Wait — it scores better but users say it's worse

This happens often enough that it deserves its own section. Three causes, all common:

1. Catastrophic forgetting. The model got better at the task and worse at everything around it — ordinary conversation, refusals, a second language, follow-up questions. Your task eval set is blind to all of it. This is the most common answer.

2. Train-serve mismatch. The model was trained without the system prompt or context block that production sends. It excels on the eval harness (which mirrors training) and behaves oddly in the app.

3. An unrepresentative eval set. If it was authored rather than sampled from real traffic, a high score means "good at the cases we imagined".

A higher score on a narrow eval set is fully compatible with a worse product. The fix is structural: always run a general-capability regression set alongside the task set (lesson 7 gives you one), and always shadow-run before switching.

💡 And when it happens: roll back first, investigate second. A five-point eval gain is not worth an unquantified regression, and rolling back costs one config change.

Stop conditions

A project plan that can only end in shipping is not a plan. Write these down at the start, when nobody is emotionally invested:

• Before training: if the prompted baseline already meets the bar, stop. Ship the prompt.
• After the first 300 examples: if the measured gain from 300 is flat, more data probably won't save it — the task definition or the approach is wrong.
• After evaluation: if it doesn't clearly beat the baseline on task quality while holding general capability, safety and cost, don't ship it. The weeks already spent are not a reason to ship — that is the sunk-cost fallacy with a GPU bill attached.

🎯 Selection-round radar: "Walk me through a fine-tuning project" rewards structure over detail. Lead with eval set and prompted baseline first, be honest that data is weeks and training is hours, insist on a general-capability regression check, and finish on shadow run, gradual ramp, one-switch rollback. Naming a stop condition unprompted is the detail that lands.

Common mistakes

• Training before an eval set exists.
• Comparing against a deliberately weak prompted baseline.
• Planning days for data and weeks for training.
• Pinning to a floating base-model alias instead of an exact version.
• No recorded dataset version, so v2 can never be reproduced.
• Switching all traffic at once with no shadow phase.
• Shipping a marginal result because the project already cost six weeks.
• No named rollback target agreed before launch.

Quick recap

Practice Zone — PYQs from real selection rounds

Six MCQs, then two tasks: plan a project end to end with stop conditions, and debug a fine-tune that scores well and feels worse.

Q1Fine-tuning in practiceeasy

What is the practical difference between a hosted fine-tune and self-hosting your own?

Asked in

aThere is nonebA hosted fine-tune is a managed job and endpoint — you upload data and get an API — while self-hosting means you own the training environment, the weights and the serving stackcHosted fine-tuning cannot use your own datadSelf-hosting is always cheaper

Q2Fine-tuning in practicemedium

Before launching any training run, the single most valuable thing to have ready is:

Asked in

aA GPU clusterbThe evaluation set and the prompted baseline's score on it — without them you cannot tell whether the fine-tune helpedcA model namedA blog post to announce it

Q3Fine-tuning in practicemedium

Your fine-tuned model beats the prompted baseline on your eval set but users complain it is worse. Most likely cause?

Asked in

aThe model file is corruptedbThe eval set doesn't represent real traffic, or the model regressed on capabilities the eval set never tested — general conversation, refusals, other languagescUsers are imagining itdFine-tuning always makes models worse

Q4Fine-tuning in practicehard

What is the safest way to roll out a fine-tuned model?

Asked in

aSwitch all traffic at once and watch the dashboardsbShadow-run it on copied live traffic first, then a small percentage of real traffic, comparing quality and cost against the current version, with a fast rollback pathcDeploy on a Friday eveningdReplace the base model in place with no version change

Q5Fine-tuning in practicemedium

What should be version-controlled for a fine-tune to be reproducible?

Asked in

aJust the final weightsbThe base model version, the dataset version, the hyperparameters, the training code, the eval set and the resulting scores — the weights alone don't explain how they came to becOnly the hyperparametersdNothing; fine-tunes are disposable

Q6Fine-tuning in practicehard

A privacy consideration specific to fine-tuning that RAG avoids is:

Asked in

aFine-tuning requires sending data over HTTPbTraining data becomes part of the weights — it can potentially be surfaced in outputs, and it cannot be deleted without retraining, which complicates data-deletion obligationscFine-tuned models cannot be encrypteddThere is no difference

Hands-on tasks:

FAQ

Next lesson: how you actually know any of this is working — Lesson 7: Evaluating LLMs →