LLM Settings Explained: Temperature, Top-p, Tokens & Context Window

Ask ChatGPT the same question twice and you might get two differently-worded answers. Is the model confused? No — it's a setting, and you can control it. LLMs come with a small set of knobs — temperature, top-p, max tokens, context window — and knowing what each one really does separates people who use AI from people who build with it. Interviewers test these because they're cheap to ask and instantly expose whether you've actually shipped anything.

Where the randomness comes from

Rewind to lesson 2: at every step the model produces a probability for every possible next token. Then one token must be picked. Always picking the single top token gives you the same answer every time — safe, but often stiff and repetitive. Sampling from the probabilities gives variety — sometimes brilliance, sometimes nonsense. The knobs in this lesson all answer one question: how adventurous should the picking be?

Temperature — the adventure dial

Temperature reshapes the probabilities before picking. Low temperature exaggerates the leader — the top token becomes near-certain. High temperature flattens the field — underdogs get real chances.

Practical ranges: 0–0.3 for anything that must be consistent and correct — extraction, classification, code, support answers, anything automated tests will check. 0.7–1.1 for creative work — marketing copy, brainstorming, story writing. And say this in interviews, because it's the part people get wrong: temperature changes randomness, not intelligence — and temperature 0 makes answers repeatable, not correct. If the model's best guess is wrong, temperature 0 just gives you that wrong answer reliably.

Top-p — trimming the tail

Even at moderate temperature, thousands of terrible tokens keep tiny probabilities — and occasionally one gets sampled, producing that one weird word in an otherwise fine sentence. Top-p (nucleus sampling) fixes this by cutting the tail: sort tokens by probability, keep the smallest set whose probabilities sum to p (say 0.9), sample only within that set.

The elegant part: the set size adapts. When the model is confident ("The capital of India is ___"), two or three tokens already cover 90% — tiny set, focused choice. When the context is open ("Once upon a time, ___"), dozens of tokens share the mass — bigger set, more freedom. Its blunt cousin top-k keeps a fixed count (say the top 50) regardless of confidence. Convention worth quoting: tune temperature OR top-p aggressively, not both — they fight over the same job.

Max tokens — the output guillotine

max_tokens caps how many tokens the model may generate. Hit the cap and generation just stops — mid-list, mid-sentence, mid-word. The API even tells you it happened via the finish reason:

response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Explain RAG in detail"}],
    max_tokens=40,  # deliberately tiny
)
print(response.choices[0].message.content)
print("finish_reason:", response.choices[0].finish_reason)

RAG (Retrieval-Augmented Generation) is a technique where a
language model first retrieves relevant documents from an
external knowledge source and then
finish_reason: length

finish_reason: length = truncated by the cap (a healthy completion says stop). Users report this bug as "the bot gives incomplete answers" — now you know the 30-second fix: raise max_tokens, or ask for a more concise format.

Wait — why did the bot forget the start of the conversation?

The context window is the model's working memory: the maximum tokens one request can hold — system prompt + conversation history + retrieved documents + the answer being generated, all together. It is a hard physical limit, not a suggestion. When a long chat outgrows it, something must give: apps silently drop or summarize the oldest messages. The user experiences it as amnesia — "it forgot my name from an hour ago" — but nothing malfunctioned. The model never remembers anything by itself; whatever isn't inside the current context window does not exist for it.

Selection-round radar: "What is a context window and what happens when it overflows?" — the expected answer names the components that share the window (prompt + history + documents + output) and one overflow strategy (sliding window, summarizing old turns, or storing facts externally and re-injecting them). That last one is literally RAG applied to chat memory — a connection worth saying out loud.

Tokens and money — the billing meter

APIs charge per token: one rate for input tokens (everything you send — including, on every single call, the entire conversation history), a higher rate for output tokens (what the model writes). Quick math you should be able to do on a napkin:

# 10,000 requests/day, 1,200 input + 300 output tokens each
# Rs 0.010 per 1K input, Rs 0.030 per 1K output

daily = 10_000 * (1200/1000 * 0.010 + 300/1000 * 0.030)
print(f"per day: Rs {daily:.0f},  per month: Rs {daily*30:,.0f}")

per day: Rs 210,  per month: Rs 6,300

Notice the input side dominates — 1,200 vs 300. That's typical, because history and instructions ride along on every call. The cost-cutting levers, in order of impact: trim the prompt and history, cache repeated prompt parts, route easy requests to a smaller model, and only then worry about shortening answers.

Common mistakes

• Using high temperature for extraction/automation tasks — then wondering why QA can't reproduce bugs.
• Believing temperature 0 guarantees correct answers — it guarantees repeatable ones.
• Blaming the model for truncated answers instead of checking finish_reason == length.
• Forgetting the answer also consumes context — a full window means no room left to respond.
• Ignoring that chat history is re-sent (and re-billed) on every call — long conversations get linearly more expensive.
• Cranking temperature AND top-p together — two hands fighting over one steering wheel.

Quick recap

Practice Zone — PYQs from real selection rounds

Six MCQs and three hands-on tasks — configure a real call, map bug reports to parameters, and estimate a monthly bill.

Q1LLM settingseasy

Setting temperature ≈ 0 makes the model:

Asked in

aAnswer faster because it thinks lessbAlmost always pick the highest-probability token — outputs become consistent and repeatablecRefuse creative tasksdUse fewer tokens per answer

Q2LLM settingseasy

You're building (a) a bot that extracts invoice numbers from emails, and (b) a bot that writes Instagram captions. Sensible temperatures are:

Asked in

a(a) high ~1.5, (b) low ~0b(a) low ~0–0.2, (b) higher ~0.8–1.0cBoth exactly 1.0 alwaysdTemperature doesn't matter for either

Q3LLM settingsmedium

Top-p (nucleus) sampling with p = 0.9 means:

Asked in

aOnly the single most likely token is ever pickedbThe model samples only from the smallest set of top tokens whose probabilities add up to 90%, cutting off the unlikely tailc90% of the answer is copied from training datadThe response length is capped at 90 tokens

Q4LLM settingseasy

If max_tokens (the output limit) is set too low, the visible symptom is:

Asked in

aThe model answers in a different languagebThe answer stops abruptly mid-sentencecThe model becomes more accuratedThe request is rejected entirely

Q5LLM settingsmedium

A chatbot conversation has grown longer than the model's context window. What actually happens / must happen?

Asked in

aThe model remembers everything anyway — the window is just a suggestionbOlder messages must be dropped, truncated or summarized to fit — the model genuinely cannot see what's outside the windowcThe API automatically fine-tunes the model on the old messagesdThe conversation crashes permanently

Q6LLM settingsmedium

LLM APIs are typically priced:

Asked in

aPer minute of compute timebPer token — with input tokens and output tokens billed at different ratescA flat fee per month regardless of usagedPer character of the final answer only

Hands-on tasks:

response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[
        {"role": "system", "content": "You are a support agent for TechMart."},
        {"role": "user", "content": "My earbuds won't pair. Help?"},
    ],
    temperature=___,
    max_tokens=___,
)

FAQ

Next lesson: the highest-leverage skill per minute spent — Lesson 6: Prompting Basics →