LLMOps & Monitoring: Logging, Tracing, Drift and Alerts

Monday morning. Thumbs-down doubled over the weekend. Nobody deployed anything. The prompt is unchanged, the code is unchanged, and the model id in your config is the same string it was on Friday.

In ordinary software this would be baffling. In an LLM application it is a Tuesday — and if you have the right five fields in your logs, you can name the cause in about ten minutes. This lesson is about having them.

What makes LLMOps different

The shift is from operating a model to operating a system whose most-changed component is text. A prompt edited in a dashboard is a global production change with no review, no version and no rollback — unless you build those.

The loop

Note the three orange triggers. Each re-enters the loop at evaluate without any change of yours. That is the structural reason LLMOps is continuous rather than a launch checklist.

What to log

log.info({
    "request_id":      request_id,
    "user_id":         hashed(user_id),
    "prompt_version":  PROMPT_VERSION,        # "support-reply@v7"
    "model_version":   resolved_model_id,     # the RESOLVED id, not the alias
    "retrieval_config": RETRIEVAL_VERSION,

    "input_tokens":    usage.input_tokens,
    "output_tokens":   usage.output_tokens,
    "cost":            estimated_cost,

    "latency_ms":      total_ms,
    "ttft_ms":         ttft_ms,
    "stage_ms":        {"retrieval": 210, "rerank": 90, "model": 1840},

    "retrieved_ids":   [d.id for d in docs],
    "tool_calls":      [{"name": t.name, "ok": t.ok} for t in tools],
    "flags":           {"refused": False, "guardrail_block": False},
    # prompt + output subject to your data policy: redact, and set retention
})

The three fields that earn their place hardest: prompt_version, model_version and cost. Without the first two, "why did it change?" is unanswerable. Without the third, you find out about a cost incident from the invoice.

💡 Log the resolved model id, not the alias you requested. An alias like "latest" can point somewhere new without your config changing — and that is precisely the incident you are logging for.

Tracing a pipeline

A modern LLM feature is not one call. It is retrieval, then reranking, then prompt assembly, then a model call, maybe tool calls, maybe an output check. When the answer is wrong, there are six candidate culprits.

A trace records all of them for one request, so "the answer was bad" becomes "retrieval returned the wrong chunk" or "the context block was empty" or "the tool timed out and we generated anyway".

Without tracing you debug LLM systems by guessing. With it, most investigations are ten minutes of reading one request.

What to alert on

And the quality signals worth tracking even if they don't page: edit rate, escalation rate, retry rate, abandonment. Behavioural signals come from everyone for free and measure usefulness; thumbs are sparse and biased toward extremes.

Wait — three things drift, and one is invisible

In a classical ML system, drift means your input distribution moved. In an LLM application, three independent things drift:

1. Your inputs. New user segment, new document format, longer questions, another language. Monitor input length and category distributions, not just volume.

2. Your knowledge base. Documents added, removed, reindexed, re-chunked. The prompt is innocent; its inputs changed. Retrieval metrics catch this.

3. The model. The provider updated it. This is the invisible one — nothing in your repository changed, and if you point at a floating alias, nothing in your config changed either.

Pin explicit model versions, log the resolved id, and re-run your evaluation set on every provider release. Teams that only run evals on their own changes learn about model updates from customer complaints.

🎯 Selection-round radar: "How do you monitor an LLM in production?" is now common even in service-company rounds. Four beats: log prompt version, model version, tokens, cost and per-stage latency on every request → trace multi-stage pipelines so failures are attributable → alert on cost per request and on model_version changing, not just on errors → and track behavioural quality signals like edit and escalation rate, because thumbs are too sparse to rely on.

The weekly human read

Thirty minutes a week, someone reads twenty real conversations end to end. Not sampled by score, not filtered to failures — just twenty.

This sounds unscientific next to a dashboard and it is the practice experienced teams never drop, for a simple reason: every automated metric you eventually build starts as something a person noticed while reading. The dashboard can only measure failures you already thought of.

What comes out of that half hour: new failure modes, queries with no good retrieval hit (a content gap, not an ML problem), and fresh cases to append to the frozen eval set.

Common mistakes

• No prompt version or model version in the logs.
• Pointing at a floating model alias in production.
• Finding out about cost changes from the monthly invoice.
• Treating a dashboard prompt edit as not-a-deploy.
• Alerting only on errors, when the real failure is fluent and wrong.
• Relying on thumbs alone for quality.
• No tracing, so a bad answer can't be attributed to a stage.
• Deciding logging retention and redaction after the first incident.

Quick recap

Practice Zone — PYQs from real selection rounds

Six MCQs, then two tasks: design monitoring for a real feature, and run an incident investigation.

Q1LLMOpseasy

What makes LLMOps different from traditional MLOps?

Asked in

aNothing — the practices are identicalbMost teams don't train the model, outputs are open-ended text with no single correct answer, prompts are a deployable artefact, and cost per request is a first-class metriccLLMOps needs no monitoringdLLMOps only applies to open-source models

Q2LLMOpsmedium

What is the minimum you should log for every LLM request in production?

Asked in

aJust the final responsebRequest id, prompt version, model version, input and output token counts, latency, and (subject to privacy rules) the assembled input and the outputcOnly errorsdNothing, for privacy reasons

Q3LLMOpsmedium

Why is drift monitoring different for LLM applications?

Asked in

aThere is no drift in LLM systemsbThree things drift independently — your input distribution, the retrieved knowledge base, and the model itself when the provider updates it — and the third is invisible unless you version-pin and monitorcOnly the training data driftsdDrift only affects images

Q4LLMOpshard

Which production signal usually reflects real quality best?

Asked in

aAverage response lengthbBehavioural signals — how often users edit, retry, escalate to a human, or abandon — because they measure whether the output did the jobcThumbs-up rate alonedGPU utilisation

Q5LLMOpsmedium

How should cost be monitored in an LLM application?

Asked in

aCheck the monthly invoicebPer request, broken down by feature and by pipeline stage, with alerts on cost per request — a prompt change can multiply spend overnightcCost doesn't need monitoringdOnly track output tokens

Q6LLMOpshard

What is the value of tracing in an LLM pipeline?

Asked in

aIt makes the model more accuratebIt records every stage of one request — retrieval, reranking, prompt assembly, model call, tool calls — so a bad answer can be attributed to the stage that caused itcIt reduces token costdIt replaces evaluation

Hands-on tasks:

FAQ

Last lesson: making it cheaper and faster without making it worse — Lesson 10: Cost & Latency →