Using LLM APIs: Chat Completions, Function Calling & JSON Output

Everything so far, you could learn by chatting. This lesson is where you become a builder — because in every GenAI job, the model is reached the same way: an API call from code. The good news: it's one of the friendliest APIs in software. The interesting news: three ideas around it — roles, statelessness, and function calling — produce most of the practical interview questions, and one of them causes a bug that literally everyone writes once.

Anatomy of a call

Strip away the SDKs and every LLM API is the same shape: send a list of messages plus settings, receive one new message back.

from openai import OpenAI

client = OpenAI()  # reads OPENAI_API_KEY from environment

response = client.chat.completions.create(
    model="gpt-4o-mini",              # which model
    temperature=0.2,                  # lesson 5's knobs
    messages=[                        # the conversation so far
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "What is a vector database?"},
    ],
)

print(response.choices[0].message.content)
print(response.usage)  # token counts = your bill

A vector database stores embeddings and finds the most similar…
CompletionUsage(prompt_tokens=27, completion_tokens=96, ...)

Three things to notice, because interviews ask about each: the answer lives at response.choices[0].message.content; the usage object is your token bill from lesson 5; and the API key comes from the environment — never hard-code keys (a classic follow-up: "how do you manage secrets?").

The three roles

Each message carries a role, and the division of power matters: system — the developer's standing orders (identity, rules, format — lesson 6's skeleton lives here); user — the human's turns; assistant — the model's own previous replies. Why send the model its own old answers? That's the next section — the most practical thing in this lesson.

Wait — the bot forgot the user's name from one message ago?

The bug every first-time builder writes. User: "Hi, I'm Priya." Bot: "Hello Priya!" User: "What's my name?" Bot: "I don't have access to that information." Nothing is broken. The API is stateless: every call is a complete stranger. The server keeps zero memory between requests — if your second request contains only the second question, then for the model, the first exchange never happened.

The fix defines how all chat apps work: you resend the whole conversation — every user message and every assistant reply — on every single call. "Chat memory" is an illusion the client builds:

history = [{"role": "system", "content": SYSTEM_PROMPT}]

def chat(user_message):
    history.append({"role": "user", "content": user_message})
    response = client.chat.completions.create(
        model="gpt-4o-mini", messages=history,   # FULL history each time
    )
    reply = response.choices[0].message.content
    history.append({"role": "assistant", "content": reply})
    return reply

Two consequences you can now derive yourself: long chats get linearly more expensive (history is re-billed every call — lesson 5), and eventually the history outgrows the context window, so production apps trim or summarize old turns. Statelessness → resend history → trim/summarize: that chain of reasoning is a complete interview answer.

Structured output — when code reads the answer

A human reads prose; your program can't. If the model's output feeds code, force it into a machine-readable shape — JSON mode / structured output makes the model emit valid JSON (with the better APIs validating against your schema):

response = client.chat.completions.create(
    model="gpt-4o-mini",
    response_format={"type": "json_object"},
    messages=[
        {"role": "system",
         "content": 'Extract fields from the complaint. Reply as JSON: '
                    '{"product": str, "issue": str, "urgency": "low|medium|high"}'},
        {"role": "user",
         "content": "My new washing machine is leaking water everywhere!!"},
    ],
)
print(response.choices[0].message.content)

{"product": "washing machine",
 "issue": "leaking water",
 "urgency": "high"}

This little pattern — unstructured text in, clean JSON out — quietly powers document processing, ticket routing and form-filling projects across the industry. Still validate the JSON in code before trusting it; belt and braces.

Function calling — giving the model hands

The model can't check today's weather, your database, or anyone's order status — it only makes text. Function calling (tool calling) bridges that: you describe your functions to the API, and when the model needs one, it replies not with prose but with a structured request to call it.

The description you send is a schema — name, purpose, parameters:

tools = [{
    "type": "function",
    "function": {
        "name": "get_order_status",
        "description": "Get the live delivery status of an order by its id.",
        "parameters": {
            "type": "object",
            "properties": {
                "order_id": {"type": "string", "description": "e.g. '8123'"},
            },
            "required": ["order_id"],
        },
    },
}]

response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Where is my order 8123?"}],
    tools=tools,
)

call = response.choices[0].message.tool_calls[0]
print(call.function.name, call.function.arguments)

get_order_status {"order_id": "8123"}

Now your code runs the real function, appends the result as a tool message, and calls the API once more — the model turns the raw result into a human answer. Say the key sentence precisely in interviews: the LLM never executes anything — it only emits a JSON request; your code executes and returns the result. That separation is the security boundary, and it's the foundation agents are built on in lesson 11.

Selection-round radar: "Explain function calling" is now standard at product companies and GenAI-project service rounds. Full-marks answer: schema → model decides and emits JSON → your code executes → result goes back → model phrases the final answer. Bonus: "the tool description is a prompt — vague descriptions cause wrong tool choices."

Two production notes: streaming and failure

Streaming: with stream=True, tokens arrive as they're generated — the ChatGPT typing effect. Total time is unchanged; perceived latency collapses, and time-to-first-token becomes your UX metric. Failure handling: real systems see rate limits (HTTP 429), timeouts and overloaded-server errors — production code retries with exponential backoff, sets timeouts, and degrades gracefully. Mentioning retries-with-backoff unprompted is a small sentence that sounds like experience.

Common mistakes

• Sending only the latest message and expecting the model to remember — the amnesia bug. Resend history.
• Hard-coding API keys in source code — environment variables or a secrets manager, always.
• Parsing free-text answers with regex when JSON mode exists — and not validating the JSON even then.
• Writing lazy tool descriptions ("gets data") — the model chooses tools by reading them.
• Believing the model executed the function — it only requested; if your code didn't run it, nothing happened.
• No retry/backoff around calls — the first rate-limit spike takes your feature down.

Quick recap

Practice Zone — PYQs from real selection rounds

Six MCQs and three hands-on tasks: write a call from memory, design a tool schema, and debug the amnesia bug in real code.

Q1LLM APIseasy

In a chat-completion API, the three standard message roles are:

Asked in

aadmin, client, serverbsystem, user, assistantcprompt, response, errordinput, output, hidden

Q2LLM APIsmedium

A user asks a follow-up question and the bot has no idea what they're talking about. The most likely cause:

Asked in

aThe model's weights were reset between requestsbLLM APIs are stateless — the developer must resend the conversation history with every request, and this code didn'tcThe user's internet connection droppeddThe API bans follow-up questions

Q3LLM APIsmedium

In function calling (tool calling), the LLM:

Asked in

aDirectly executes your Python function on its serversbReturns a structured JSON request naming the function and arguments — YOUR code executes it and sends the result backcDownloads your codebase to inspect itdCan only call functions written in JavaScript

Q4LLM APIseasy

Structured output / JSON mode is used when:

Asked in

aYou want the answer to be more creativebYour code must parse the model's reply — forcing valid JSON (often against a schema) makes the output machine-readable and reliablecYou want to reduce API cost to zerodThe user speaks multiple languages

Q5LLM APIsmedium

A tool schema you pass to the API contains, at minimum:

Asked in

aThe function's compiled machine codebThe function's name, a natural-language description of when to use it, and a JSON Schema of its parameterscYour server's IP address and credentialsdThe full source code of your application

Q6LLM APIseasy

APIs offer streaming responses mainly so that:

Asked in

aThe total cost is lowerbUsers see tokens as they're generated instead of staring at a spinner — perceived latency drops dramaticallycThe model can change its answer midwaydAnswers become more accurate

Hands-on tasks:

def ask_bot(user_message):
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": user_message},  # only this turn!
        ],
    )
    return response.choices[0].message.content

FAQ

Next lesson: the most-asked applied topic in GenAI interviews — Lesson 9: RAG, Step by Step →