Quantization Explained: 4-bit, 8-bit and GPU Memory Maths

You photograph a receipt to email it. The original is 12 megapixels; you send it at 2. Every word is still readable, the file is a sixth the size, and nobody complains. You would not do the same to a photo going on a billboard.

Quantization is that decision applied to model weights — and, like the photo, the right answer depends entirely on what the result is for.

What it is

A model's weights are numbers, and numbers need bits. Training typically uses 16 bits per weight. Quantization stores them with fewer — 8 bits, or 4 — so the same model occupies a fraction of the memory.

The information loss is real: a coarser scale can't represent every value the finer one could, so each weight is stored as the nearest available step. Modern methods are clever about where to spend precision, which is why the memory savings are large and proportional, while the quality loss is usually much smaller than the bit reduction suggests.

The arithmetic you'll be asked for

The calculation is parameters × bytes per parameter, and interviewers do ask you to do it out loud:

That table is the difference between "needs a data-centre GPU" and "runs on a laptop". A 7B model at 4-bit fits in a few gigabytes, which is why local model apps exist at all.

Wait — weights aren't the whole story

This is where most candidates stop, and where the interesting half begins. Weights are a fixed cost. Serving also needs memory that grows with your traffic:

• The KV cache — stored attention keys and values for tokens already processed, so each new token doesn't recompute the whole sequence. It grows with sequence length and with the number of concurrent requests.
• Activations and framework overhead — a gigabyte or two.

Weights set the floor on memory; the KV cache sets the ceiling on concurrency. A 13GB int8 model on a 24GB card leaves roughly 9–10GB for cache and overhead — and that remainder, not the model size, is what decides how many users you can serve at once.

💡 Practical rule: budget weights plus at least 25–40% headroom, and more if your prompts are long or your concurrency is high. Sizing a GPU by weights alone is how a deployment ends up refusing requests under load.

What it costs in quality

Honest summary: 8-bit is usually near-lossless for most tasks. 4-bit is noticeable but often acceptable. Below that, degradation becomes hard to ignore.

But the degradation is not uniform across tasks, and that is the part worth saying:

So the only correct answer to "is 4-bit good enough?" is run your own evaluation set on both and compare per-category. A published benchmark delta tells you about someone else's workload.

from transformers import AutoModelForCausalLM, BitsAndBytesConfig

quant = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_compute_dtype="bfloat16",   # compute in higher precision
    bnb_4bit_quant_type="nf4",           # a 4-bit format suited to weights
    bnb_4bit_use_double_quant=True,      # quantize the quantization constants
)

model = AutoModelForCausalLM.from_pretrained(
    "<model-id>", quantization_config=quant, device_map="auto"
)

Note compute_dtype: weights are stored at 4 bits but arithmetic happens at higher precision. That is a common misconception worth correcting if it comes up — quantization is primarily about storage.

PTQ vs QAT

When someone says "a quantized model", they almost always mean PTQ.

Where it matters

Self-hosted inference — this is the main event. Quantization decides which GPU you need, how many concurrent requests fit alongside the weights, and therefore your cost per request. It is one of the most direct cost levers available.

Fine-tuning — QLoRA quantizes the frozen base so adapters can be trained on modest hardware (lesson 4).

Local and on-device — running a model on a laptop or a phone is only possible at reduced precision.

Hosted APIs — irrelevant to you. The provider chose, and you pay per token regardless.

🎯 Selection-round radar: two questions recur. "What is quantization and what does it cost?" → fewer bits per weight, proportionally less memory, a usually-small and task-dependent quality loss. And the practical one: "how much GPU memory does a 7B model need?" → do the arithmetic out loud (7B × 2 bytes ≈ 14GB at fp16), then add the line that marks experience: "plus the KV cache, which grows with concurrency and is what actually limits how many users fit."

Common mistakes

• Sizing a GPU from weights alone and running out of memory under load.
• Assuming a benchmark's quantization delta applies to your task.
• Going straight to 4-bit when 8-bit would have fitted and cost less quality.
• Believing arithmetic happens at the storage precision.
• Switching precision in production without an eval comparison and a rollback.
• Quantizing when you use a hosted API, where it has no meaning.

Quick recap

Practice Zone — PYQs from real selection rounds

Six MCQs, then two tasks: size the hardware for a real deployment, and design the check that decides whether 4-bit is acceptable.

Q1Quantizationeasy

What is quantization in the context of LLMs?

Asked in

aSplitting the model across several GPUsbStoring the weights at lower numeric precision — 8-bit or 4-bit instead of 16-bit — so the model needs far less memorycReducing the number of layersdCompressing the training data

Q2Quantizationmedium

What is the trade-off?

Asked in

aThere is none — quantization is freebSome accuracy loss, usually small at 8-bit and more noticeable at 4-bit, in exchange for much lower memory and often faster inferencecThe model becomes slowerdIt only works on small models

Q3Quantizationmedium

A rough memory estimate for a 7-billion-parameter model at 16-bit precision is:

Asked in

aAbout 700MBbAbout 14GB for the weights — roughly 2 bytes per parameter — plus extra for activations and the KV cachecAbout 140GBdIt doesn't depend on parameter count

Q4Quantizationhard

What is the difference between post-training quantization and quantization-aware training?

Asked in

aThey are the samebPTQ converts an already-trained model to lower precision; QAT simulates the reduced precision during training so the model adapts to it, generally giving better quality at the cost of a training runcQAT is only for imagesdPTQ requires retraining

Q5Quantizationmedium

Where does quantization matter most in practice?

Asked in

aOnly during trainingbSelf-hosted inference — it decides which GPU you need, how many concurrent requests fit, and therefore your cost per requestcOnly for mobile appsdIt has no production relevance

Q6Quantizationhard

Which task type tends to suffer most from aggressive quantization?

Asked in

aCasual conversationbLong multi-step reasoning and precise code generation, where small errors compound over many tokenscShort summarisationdAll tasks degrade identically

Hands-on tasks:

FAQ

Next lesson: putting it all together — a fine-tuning project from decision to production — Lesson 6: Fine-Tuning in Practice →