Structured Output & JSON Prompting: Getting Parseable Answers

There is a moment every developer hits on their first LLM feature. The prompt works. The answer is right. And then your code does json.loads(reply) and everything explodes, because the model helpfully wrapped the JSON in "Sure! Here's the data you asked for:" and a markdown fence.

Free-flowing text is lovely for humans and useless for programs. This lesson is about getting output your code can consume — reliably, on the thousandth request, including when the input is nonsense.

The problem

prompt = "Extract the name, email and phone from this text as JSON: " + text
reply = call_model(prompt)
data = json.loads(reply)     # 💥 sometimes

The three failure shapes you will actually see:

• Conversational wrapper — "Here is the extracted data:" then the JSON.
• Markdown fence — ```json … ```.
• Shape drift — phone_number today, phone tomorrow; a string where you expected a list.

All three come from the same root cause: you asked for JSON in the middle of a sentence and left everything else unspecified. So the model did what it does with unspecified things — it picked a plausible default, and picked a different one next Tuesday.

Asking for JSON properly

Extract contact details from the text below.

Return a single JSON object and nothing else. No explanation,
no markdown code fences, no preamble.

Schema:
{
  "name":  string | null,
  "email": string | null,
  "phone": string | null
}

Use null for any field not present in the text. Do not guess or
complete partial values.

Text:
{{text}}

Four things changed. "Nothing else" kills the wrapper. "No markdown code fences" kills the fence — models add them by default because that's how JSON appears in most of their training data. The literal schema fixes the key names. And the null rule tells it what to do when a field is missing, which is the case that causes most production surprises.

Give the schema, not a description

Compare these two ways of asking for the same thing:

The prose version leaves the model to invent key names, decide whether "urgent" is a boolean or the string "yes", and guess whether one issue should be a list or a bare string. Show the exact shape you want — literally, as a code block — rather than describing it.

And constrain the values wherever you can. "positive" | "negative" | "mixed" is far stronger than "the sentiment", because it converts an open question into a choice from three. Same trick as few-shot, in schema form.

Native JSON and schema modes

Most current APIs offer help beyond the prompt itself, in two levels.

JSON mode guarantees the output is syntactically valid JSON. No fences, no preamble, parses every time. It does not guarantee your keys or your types.

Structured outputs / schema mode takes a JSON Schema and constrains generation so the result conforms — right keys, right types, required fields present. Where available, this is the correct tool, and it converts a whole class of production bug into an impossibility.

schema = {
    "type": "object",
    "properties": {
        "sentiment": {"type": "string", "enum": ["positive", "negative", "mixed"]},
        "issues":    {"type": "array", "items": {"type": "string"}},
        "urgent":    {"type": "boolean"},
    },
    "required": ["sentiment", "issues", "urgent"],
    "additionalProperties": False,
}

reply = client.responses.create(
    model=MODEL,
    input=prompt,
    response_format={"type": "json_schema", "json_schema": schema},
)

💡 A tool call is structured output wearing a different hat. When you define a function with a parameter schema, the model must produce arguments matching it — which is why "define a fake tool just to get structured data" was the standard trick before schema modes existed (GenAI lesson 8).

Wait — with schema mode, can I skip validation?

You can skip shape validation. You absolutely cannot skip semantic validation, and this distinction is a favourite interview probe.

Schema mode guarantees "order_id" is a string. It does not guarantee the string is a real order id — the model can return a beautifully-typed hallucination. It guarantees "refund_amount" is a number, not that the number is right or even positive.

data = json.loads(reply)          # shape: guaranteed by schema mode

# semantics: still yours
assert ORDER_RE.match(data["order_id"]), "not an order id format"
assert data["order_id"] in known_orders, "order does not exist"
assert 0 <= data["refund_amount"] <= order.total, "amount out of range"

Structured output makes parsing safe. It never makes the content true.

Designing for the awkward cases

Extraction prompts break in production on inputs the author never pictured. Decide these four in advance and write them into the prompt:

🎯 Selection-round radar: "How do you make sure an LLM returns valid JSON?" is one of the most common practical questions. Full answer in four layers: specify the exact schema in the prompt and forbid prose and fences → use the API's JSON or schema mode where it exists → validate against the schema in code → and on failure, retry once with the parse error included in the prompt. Then add the closer: "shape validation isn't content validation — I'd still check the values are real."

When JSON isn't the answer

JSON is the default, not the law.

• Markdown tables — when a human reads the output. Easier to scan, and models produce them cleanly.
• XML-style tags — when you want a long free-text answer plus a machine-readable part. <reply>…</reply><category>billing</category> is easier for the model than embedding a paragraph inside a JSON string, where every quote and newline needs escaping.
• Plain enums — for classification, ask for one word from a list. Wrapping {"label": "billing"} around a single value buys you nothing.

Common mistakes

• Asking for JSON without forbidding prose and markdown fences.
• Describing the fields in a sentence instead of showing the schema.
• Leaving value sets open when they could be enums.
• No rule for missing fields, so the model invents plausible ones.
• Treating schema conformance as proof the content is correct.
• Forcing a long prose answer inside a JSON string field.
• Retrying blindly on a parse failure instead of feeding the error back.

Quick recap

Practice Zone — PYQs from real selection rounds

Six MCQs, then two tasks: design a schema with its edge-case rules, and write the retry logic for a parse failure.

Q1Structured outputeasy

Why force JSON output rather than parsing prose?

Asked in

aJSON uses fewer tokensbBecause code consumes the result — structured output is parseable and validatable, while prose ('Sure! Here's your answer…') breaks parsers unpredictablycJSON makes the model more accuratedIt's required for all API calls

Q2Structured outputmedium

With JSON mode enabled, should your code still validate the output?

Asked in

aNo — JSON mode guarantees correctnessbYes — JSON mode guarantees valid syntax, not correct fields, sensible values or your business rulescOnly in developmentdOnly for large responses

Q3Structured outputmedium

What most improves the reliability of structured extraction?

Asked in

aAsking politely for correct JSONbProviding the exact schema with field types and allowed values, plus an explicit rule for missing data (e.g. null), and low temperaturecHigher temperature for flexibilitydLonger field descriptions in prose

Q4Structured outputmedium

The model returns valid JSON but invents an enum value not in your list. Best fix?

Asked in

aAccept it and map it laterbState allowed values explicitly in the schema, validate against them in code, and either retry with the error fed back or route the case to human reviewcIncrease max_tokensdSwitch to prose output

Q5Structured outputhard

You need both a structured result and an explanation. What's the clean design?

Asked in

aAsk for prose and parse it with regexbPut both in the schema — e.g. {"decision": ..., "confidence": ..., "reasoning": "..."} — so the explanation is a field rather than free text around the datacMake two separate API callsdAsk for JSON followed by a paragraph

Q6Structured outputmedium

What should happen when structured output fails to parse in production?

Asked in

aCrash the requestbCatch it, retry once (optionally feeding the parse error back), and on repeated failure route to human review while logging the raw outputcReturn an empty object silentlydIgnore the failure and continue

Hands-on tasks:

FAQ

Next lesson: rules that apply to every conversation, not just this one — Lesson 6: System Prompts & Personas →