Ask a friend "what's 17 × 24?" and demand an instant answer, and you'll get a guess. Give them the same question and a piece of paper, and you'll get 408. The knowledge was identical. What changed was whether they had room to work.
Language models have exactly this problem, for a reason that is worth properly understanding — and the fix is a single sentence you add to a prompt.
See it happen
A word problem with a couple of steps in it. First, asked flat:
A shop sells 3 shirts at ₹850 each. There is a 15% discount on
the total, and then 5% GST is added on the discounted amount.
What does the customer pay?
Answer with just the number.Result
Now the same question with six extra words:
A shop sells 3 shirts at ₹850 each. There is a 15% discount on
the total, and then 5% GST is added on the discounted amount.
What does the customer pay?
Work through it step by step, then give the final amount.Result
This is chain-of-thought prompting: asking the model to produce intermediate reasoning steps before the final answer.
Why it works
A model generates one token at a time, each conditioned on everything written so far. When you demand only the final number, all the multi-step work has to collapse into a single leap. When the steps are written out, each one is small, and — crucially — each written step becomes part of the text the next step continues. The model gets to build on its own intermediate results instead of guessing the ending.
Say it that way in an interview and you'll stand out from "it makes the model think harder", which is both vague and slightly wrong.
How to trigger it
Three ways, in increasing order of control.
Zero-shot CoT — the famous one-liner. Add "Let's think step by step" or "Work through this step by step, then give the final answer." Costs nothing, works surprisingly well, and is the version most interview questions are about.
Few-shot CoT — show one worked example including its reasoning, and the model imitates the style of reasoning, not just the answer format. Useful when your domain has a particular way of working things out.
Structured CoT — name the steps yourself. This is the production version, because you get consistent, checkable stages:
Decide whether this refund request is eligible.
Work in this order:
1. State the order date and today's date.
2. Compute the number of days elapsed.
3. State the applicable policy clause.
4. Apply it.
5. Give the decision as "ELIGIBLE" or "NOT ELIGIBLE" and one
sentence of reason.
Request: {{request}}💡 Structured CoT has a quiet second benefit: because step 2 is always on its own line, you can extract it and check the arithmetic in code. Free-form reasoning is much harder to audit.
When it helps — and when it doesn't
| Helps a lot | Adds nothing |
|---|---|
| Arithmetic and multi-step word problems | Fact lookup ("capital of Odisha") |
| Logic puzzles, constraint checking | Translation |
| Multi-clause policy or eligibility decisions | Simple classification |
| Debugging: trace, then diagnose | Reformatting or extraction |
| Anything where you must audit why | Creative writing (it can flatten the result) |
The cost is real: more output tokens, more money, slower responses. For a single-step task you are paying for a paragraph of reasoning that changes nothing.
Wait — is that printed reasoning really what it did?
No, and this is the most important honest caveat in the lesson. The chain of thought is generated text, produced the same way as everything else. It is not a transcript of an internal process, and research has repeatedly found cases where the stated reasoning does not match the actual influences on the answer.
Practically, three consequences:
- A confident, well-formatted, completely wrong chain is entirely possible — and it's more persuasive than a bare wrong answer, which makes it more dangerous.
- An error in step 2 propagates cheerfully through steps 3, 4 and 5.
- "It explained its reasoning" is not verification. If the decision matters, check the steps — ideally in code.
🎯 Selection-round radar: "What is chain-of-thought prompting?" is asked in almost every GenAI round. Three beats: ask for intermediate steps → each step conditions the next, so multi-step problems get multiple steps to happen in → but the printed reasoning is generated text, not a real trace, so it can be confidently wrong. That third beat is what separates candidates.
Self-consistency — when the answer really matters
A natural extension. Run the same chain-of-thought prompt several times at a non-zero temperature, so the model takes slightly different routes, and take the answer that appears most often.
answers = []
for _ in range(5):
reply = client.messages.create(
model=MODEL,
temperature=0.7, # variety is the point here
messages=[{"role": "user", "content": cot_prompt}],
)
answers.append(extract_final_answer(reply))
final = Counter(answers).most_common(1)[0][0]
agreement = Counter(answers).most_common(1)[0][1] / len(answers)Different reasoning paths landing on the same answer is meaningful evidence; five paths giving four different answers tells you the model is guessing. That agreement number is genuinely useful — route the low-agreement cases to a human.
The catch is arithmetic: five calls cost five times as much and take five times as long. Reserve it for decisions where being wrong is expensive.
Hiding the working from users
Your customer does not want to read five steps of policy reasoning. The standard production pattern is to ask for both and show one:
Work through the eligibility check step by step inside
<reasoning> tags. Then give the customer-facing reply inside
<reply> tags, in 2-3 sentences, with no reference to the
internal steps.Then display <reply> and log <reasoning>. You get the accuracy benefit, a clean user experience, and an audit trail for the day someone asks why a refund was denied.
What about reasoning models?
Newer "reasoning" models do this internally by default — they generate extended thinking before answering, without being asked. Two implications for you.
First, telling such a model to "think step by step" is redundant and occasionally counterproductive. Second, the skill doesn't become obsolete: you still choose when a task justifies the extra latency and cost, you still design the steps when you need auditable stages, and you still can't treat the produced reasoning as proof.
Common mistakes
- Applying CoT to lookups and classification, paying for reasoning that changes nothing.
- Treating the printed steps as a genuine trace of the model's internals.
- Trusting a well-formatted chain without checking any of the numbers.
- Showing raw reasoning to end users instead of separating it.
- Running self-consistency at temperature 0 — identical runs, no information gained.
- Adding "think step by step" to a reasoning model that already does.
Quick recap
| Concept | One-liner |
|---|---|
| Chain-of-thought | ask for intermediate steps before the answer |
| Why it works | each written step conditions the next; a multi-step problem gets multiple steps |
| Zero-shot CoT | "let's think step by step" — free, effective, most-asked |
| Structured CoT | you name the steps; consistent and checkable — the production form |
| The caveat | the reasoning is generated text, not a trace; it can be confidently wrong |
| Self-consistency | sample several paths, take the majority; agreement is a usable confidence signal |
Practice Zone — PYQs from real selection rounds
Six MCQs, then two tasks — one designing structured steps, one deciding where CoT is worth its cost.
Why does asking a model to 'work step by step' improve accuracy on multi-step problems?
Asked in
When is chain-of-thought a poor choice?
Asked in
What is self-consistency?
Asked in
You need step-by-step accuracy but users must not see the reasoning. What do you do?
Asked in
How do reasoning models change chain-of-thought prompting?
Asked in
A model's written reasoning looks convincing but the final answer is wrong. What does this tell you?
Asked in
Hands-on tasks:
Write a prompt for an EMI calculator explainer that reasons carefully but shows the customer only a clean answer. Then say what your code must do.
Asked in
For each, say whether to use CoT: 1) classify support tickets into 5 categories; 2) decide eligibility from 4 policy conditions; 3) translate a paragraph; 4) reconcile two invoices and explain the difference; 5) extract a date from an email.
Asked in
FAQ
Does chain-of-thought reduce hallucination?
For reasoning errors, often yes — the model has room to catch itself. For missing knowledge, no: a model that doesn't know a fact will produce a tidy chain of steps leading to an invented one. Fabrication from ignorance is a retrieval problem, not a prompting one.
What temperature should I use with CoT?
Low — 0 to 0.3 — for a single run, since you want the most likely reasoning path. The exception is self-consistency, which needs variety, so 0.6–0.8 there.
Is CoT the same as prompt chaining?
No. Chain-of-thought is steps inside one response. Prompt chaining is several separate API calls, where each output feeds the next — which lets you validate between stages. Lesson 7 covers chaining as the decomposition pattern.
Next lesson: getting output your code can actually parse, every single time — Lesson 5: Structured Output & JSON →