LoRA & PEFT Explained: Adapters, Rank and QLoRA

In 2021, fine-tuning a large model meant a cluster, a budget approval and a specialist. By 2023 students were doing it on a free cloud notebook. Nothing about the models got smaller. What changed was a technique with an unglamorous name — LoRA, Low-Rank Adaptation — and it is now the default way almost everyone fine-tunes.

The idea

Full fine-tuning updates every weight in the model. That means holding the weights, their gradients and optimizer state in GPU memory at once — several times the model's own size — and producing a complete new copy of the model at the end.

LoRA does something different. It freezes the original weights entirely and trains small matrices alongside them. At inference, the model computes as usual and the adapter contributes its learned adjustment on top.

The intuition behind why this works: the change you need for one specific task is far simpler than the model itself. You are not rebuilding general language ability — you are nudging behaviour. A small, low-capacity adjustment turns out to be enough to capture that nudge, and in practice well under 1% of parameters are trained, often a small fraction of a percent.

What it buys you

The serving trick

The last row of that table is the one with commercial weight, and it is worth understanding as its own point.

Because the base model is untouched, several adapters can share one copy of the base in GPU memory and be swapped per request. Ten fine-tuned variants — one per customer, or one per task — cost roughly one model's worth of memory instead of ten.

That turns per-customer customisation from a business nobody can afford into a feature. It is why "we fine-tune a variant for each enterprise client" is a sentence you now hear.

Rank — the one dial you'll be asked about

The adapter's rank (often written r) controls its size, and therefore how much it can learn.

A second parameter, lora_alpha, scales how strongly the adapter contributes; a common starting point is roughly twice the rank. And target_modules chooses which layers get adapters — starting with the attention projections and adding the MLP projections only if quality plateaus.

from peft import LoraConfig, get_peft_model

config = LoraConfig(
    r=16,                       # rank — capacity dial
    lora_alpha=32,              # ~2x rank is a reasonable start
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
    lora_dropout=0.05,          # cheap regularisation on small data
    task_type="CAUSAL_LM",
)

model = get_peft_model(base_model, config)
model.print_trainable_parameters()

trainable params: 8,388,608 || all params: 6,746,804,224
                  || trainable%: 0.1243

Eight million trainable parameters out of nearly seven billion. That number — roughly a tenth of a percent — is the whole story in one line.

💡 Don't copy a rank from a blog post. Start low (8–16), and only raise it if validation says the adapter is capacity-limited rather than data-limited. On a small dataset, a higher rank usually just buys faster overfitting.

QLoRA

Take LoRA one step further: quantize the frozen base model to 4-bit first, then train adapters on top of it. That is QLoRA, and it is the reason "I fine-tuned a large model on one consumer GPU" became an ordinary sentence.

The base is frozen anyway, so storing it at reduced precision costs much less than it would if those weights were being trained. The trade-off is that you are adapting a slightly degraded base, so the result can be a little weaker than LoRA on a full-precision base — which is a fine trade when the alternative is not being able to train at all. Lesson 5 covers quantization properly.

PEFT — the wider family

PEFT — Parameter-Efficient Fine-Tuning — is the umbrella term. LoRA is by far the most used member; you may also hear prefix tuning and prompt tuning, which learn small trainable vectors prepended to the input rather than modifying the model at all.

For placement purposes: know that PEFT is the category, LoRA is the technique that won, QLoRA is LoRA on a quantized base, and the shared idea is freeze most of the model and train a small addition.

🎯 Selection-round radar: "What is LoRA?" is now standard in GenAI rounds. Four beats: freeze the base and train small adapter matrices → well under 1% of parameters, so it trains on far smaller GPUs → the artefact is megabytes, so many adapters share one base in memory at serving time → and because the base is frozen, catastrophic forgetting is bounded. The serving point is the one most candidates miss and interviewers notice.

Wait — is full fine-tuning ever better?

Sometimes, and being able to say when is what separates a memorised answer from an understood one.

LoRA's constraint is also its ceiling. If you genuinely need the model's general behaviour substantially changed — adapting to a specialised domain with a very large dataset, not adjusting an output format — a low-rank adjustment may not have the capacity for it. With millions of examples and the compute to match, full fine-tuning can reach further.

But that is the exception. For the overwhelming majority of application work — style, format, one narrow task — LoRA matches full fine-tuning at a fraction of the cost, and it is the sensible default.

Common mistakes

• Copying a rank from a tutorial instead of tuning it on validation.
• Using a high rank on a small dataset, then blaming the technique for overfitting.
• Assuming LoRA eliminates catastrophic forgetting rather than bounding it.
• Forgetting that the adapter must be served with the exact base version it was trained on.
• Describing LoRA as "training a smaller model" — the model is the same size.
• Skipping evaluation because the run was cheap.

Quick recap

Practice Zone — PYQs from real selection rounds

Six MCQs, then two tasks: explain LoRA honestly with an analogy, and configure a real run.

Q1LoRA and PEFTeasy

What does LoRA do differently from full fine-tuning?

Asked in

aIt trains on more databIt freezes the original weights and trains small added adapter matrices instead, so only a tiny fraction of parameters are updatedcIt removes layers from the modeldIt changes the tokenizer

Q2LoRA and PEFTmedium

The biggest practical advantage of LoRA is:

Asked in

aIt always produces a better model than full fine-tuningbDramatically lower memory and storage — it can be trained on much smaller GPUs, and each adapter is megabytes rather than tens of gigabytescIt needs no training datadIt eliminates the need for evaluation

Q3LoRA and PEFTmedium

What is QLoRA?

Asked in

aA query language for modelsbLoRA applied on top of a quantized (typically 4-bit) base model, cutting memory further so large models can be tuned on a single consumer GPUcA quality-assurance frameworkdA way to train without any GPU

Q4LoRA and PEFThard

What does the LoRA 'rank' hyperparameter control?

Asked in

aThe number of GPUs usedbThe size of the adapter matrices — higher rank means more trainable capacity and more overfitting risk on small datasets; lower rank means fewer parameters and a stronger constraintcThe priority of the training jobdThe learning rate

Q5LoRA and PEFTmedium

Why does LoRA reduce catastrophic forgetting compared with full fine-tuning?

Asked in

aIt trains for longerbThe original weights are frozen, so the general capabilities encoded in them can't be overwritten — the learned change is an addition rather than an editcIt uses a different loss functiondIt doesn't — forgetting is worse with LoRA

Q6LoRA and PEFThard

When would full fine-tuning still be preferred over LoRA?

Asked in

aAlways — LoRA is a toy techniquebFor a very large domain shift with a lot of data, where you want the model's general behaviour substantially changed and you have the compute — LoRA's constrained capacity can be a ceiling therecWhen you have very little datadWhen you need a small file to distribute

Hands-on tasks:

FAQ

Next lesson: the other half of making big models fit on small hardware — Lesson 5: Quantization Explained →