Context Engineering: Budgeting Everything the Model Sees

Picture briefing a colleague before they walk into a client meeting. You have five minutes. You could read them the entire 200-page account history — technically complete, practically useless. What you actually do is choose: the three things that matter, the one number they'll be asked, and what to say if the client raises last quarter's outage.

That choosing is the job. In 2023 we called it prompt engineering because the prompt was the only thing there. Now an application's input is assembled from retrieved documents, chat history, tool results and memory — and the field renamed itself accordingly. This lesson covers the term interviewers have started using: context engineering.

What changed

The instruction is now a small fraction of what the model reads. Everything else in that window got there because a piece of your code decided to put it there — and that decision is the higher-leverage one.

The window is a budget, not a container

Every line item there is a choice with a price. Tokens cost money on every request, they cost latency, and past a point they cost accuracy. Treating the window as a budget with named line items — rather than an append-only log — is the mental shift the whole lesson turns on.

Wait — I have 200k tokens, why not use them?

Because bigger windows removed a hard limit, not the need to choose. Three costs, and the third surprises people.

Money. You pay per input token, per request. Forty passages instead of five is eight times the input bill, forever.

Latency. More input means a slower first token. Users feel it.

Accuracy. This is the counter-intuitive one. Irrelevant passages are distractors: they contain plausible-looking text near the topic, and the model has to sort signal from noise. The relevant passage also ends up buried in the middle, where attention is weakest. Stuffing the window frequently scores worse than a careful five.

Precision beats volume. A big window buys room for a good selection, not permission to skip selecting.

Selecting what goes in

The standard pipeline, and where each stage earns its place:

1. Retrieve wide — fetch 20–50 candidates, because recall at this stage is cheap and a missed document can never be recovered later.
2. Rerank — score candidates against the actual question with a cross-encoder or a small model, which is far more accurate than the first-pass similarity score (RAG lesson 6).
3. Truncate — keep the top 3–8. This is the step teams skip, and skipping it is what makes retrieval feel unreliable.
4. Compress — for long passages, extract only the question-relevant portion rather than passing the whole chunk.
5. Attribute — label each passage with its source so the answer can cite it and a reviewer can check it.

Managing conversation history

A long chat drifts and starts contradicting itself for two compounding reasons: early turns fall out of the window entirely, and what remains is diluted by chatter. The standard fix is hybrid:

def build_history(turns, keep_verbatim=4):
    old, recent = turns[:-keep_verbatim], turns[-keep_verbatim:]

    summary = summarise(old)        # decisions and facts, not dialogue
    return [
        {"role": "system", "content": f"Conversation so far:\n{summary}"},
        *recent,                    # last few turns, exact wording
    ]

The recent turns stay verbatim because pronouns and follow-ups ("do that for the other one too") need exact wording. The older turns become a compact statement of what was decided — "chosen stack: Next.js + Postgres; deadline 30 Sep; user prefers short replies" — rather than a transcript of how the decision was reached.

💡 Summarise the outcomes, not the conversation. "The user asked about databases and the assistant explained three options" is useless later. "Decided: Postgres, for JSONB support" is what the next turn actually needs.

Ordering and caching

Two rules, and the second is a cost bug waiting to happen in most first drafts.

Order for attention. Instructions first, bulk data in the middle, the user's question last, key instruction restated after long data. Same "lost in the middle" effect as lesson 7.

Order for caching. Prompt caching matches a prefix: identical leading bytes are reused at a large discount, and one changed character invalidates everything after it. So the stable material — system prompt, tool definitions, fixed few-shot examples — goes first and byte-identical, and everything variable goes after.

# ✗ hostile — the timestamp changes every request, so the 500 lines
#   of fixed rules behind it can never be reused
Request time: 2026-09-06T11:42:03Z
User: Priya S.
You are ShopKart's support assistant. [500 lines of rules]

# ✓ friendly — stable prefix first, variable suffix after
You are ShopKart's support assistant. [500 lines of rules]
[8 fixed few-shot examples]
---
Session: Priya S., 2026-09-06T11:42Z
<docs>...</docs>
Question: ...

Nothing was added or removed there. Only the order changed, and the bill drops sharply.

🎯 Selection-round radar: "What is context engineering?" is a 2026-flavoured question that separates candidates who read current material from those who don't. Answer: deciding what enters the context window at all — retrieval, history, tool results, memory — and how it's selected, ordered, compressed and budgeted. Prompt engineering is one part of it. Then add the line they're waiting for: "a bigger window doesn't remove the need to choose; irrelevant context lowers accuracy as well as raising cost."

Context rot in agents

Agents make all of this worse, because they append to their own context as they work. Twenty tool calls in, the window is mostly old results — and the original task is a distant memory near the top.

The symptoms are recognisable: the agent repeats a step it already did, drifts from the goal, or answers about something it read three steps ago. The structural responses:

• Pin the goal — restate the task each iteration rather than relying on it surviving at the top.
• Summarise old steps — replace step-by-step logs with "established so far: …".
• Externalise bulk — write large tool outputs to a file or store and pass an id, not the payload.
• Cap tool output — truncate or paginate; one chatty tool can flood a window by itself.
• Sub-agents with fresh context — hand a self-contained subtask to a clean window and take back only the result (agents lesson 7).

Common mistakes

• Treating a large window as a reason to stop selecting.
• Passing everything retrieval returned, with no rerank or truncation.
• Appending full chat history until it silently overflows.
• Summarising the dialogue instead of the decisions.
• Putting a timestamp or user id at the top of a fixed prefix.
• Letting an agent's tool results accumulate unbounded.
• Leaving no headroom, so a long input fails at the limit instead of degrading.

Quick recap

Practice Zone — PYQs from real selection rounds

Six MCQs, then two design tasks: budget a 32k window line by line, and fix a cache-hostile prompt layout.

Q1Context engineeringeasy

What does "context engineering" cover that "prompt engineering" does not?

Asked in

aNothing — they are two names for the same thingbDeciding what goes into the context window at all: retrieved documents, conversation history, tool results, memory — how it is selected, ordered, compressed and budgetedcOnly the choice of modeldOnly the temperature setting

Q2Context engineeringmedium

Your model has a 200k-token window. Why not simply stuff all 40 retrieved documents in?

Asked in

aThe API rejects requests over 10 documentsbCost and latency rise with every token, and accuracy often falls — irrelevant passages distract the model and the relevant one gets buried in the middlecDocuments must be under 100 words eachdLarge windows disable few-shot learning

Q3Context engineeringmedium

A long chat starts contradicting itself around turn 40. The standard fix is:

Asked in

aRestart the conversation each timebManage history deliberately — keep a rolling summary of older turns plus the last few verbatim, and carry key decisions forward as explicit factscIncrease max_tokensdSet temperature to 0

Q4Context engineeringhard

Which ordering usually works best for a prompt with instructions, retrieved documents and the user's question?

Asked in

aDocuments, then instructions, then question, in any order — position doesn't matterbInstructions first, then the documents clearly delimited, then the user's question last — with the key instruction restated at the endcQuestion first, instructions buried in the middledAlternate instructions and documents line by line

Q5Context engineeringmedium

Which part of a prompt should stay byte-identical across requests to benefit from prompt caching?

Asked in

aThe user's questionbThe stable prefix — system prompt, tool definitions, fixed few-shot examples — placed first, with variable content after itcThe retrieved documentsdNothing; caching is automatic regardless of layout

Q6Context engineeringhard

An agent's context fills with tool outputs and it starts losing the original task. Best structural response?

Asked in

aGive the agent more toolsbBudget the context: keep the goal pinned, summarise or drop old tool results, store bulky output externally and pass a reference, and cap how much any one tool may returncRaise the temperaturedAsk the model to remember harder

Hands-on tasks:

Request time: 2026-09-06T11:42:03Z
User: Priya S. (account 88213)
You are ShopKart's support assistant. [500 more lines of rules]
[8 fixed few-shot examples]
Retrieved docs: ...
Question: ...

FAQ

Next lesson: what happens when some of that context was written by someone who wants your assistant to misbehave — Lesson 9: Prompt Injection & Safety →